EVE Technology Lab

I've been getting a 502 error for the past day.

Yes, the server was getting timeouts, and now mysql is angry I think.

@Scrapyard: That is what should be happening right now. It drops any items above 50, which is why the queue remains at 50. The problem is more that if it is slow enough to grow to 50, it will probably always remain there, as items are added faster than they are uploaded.

As uploads are handled in different threads, the only thing I can see having any effect on upload speed for other endpoints is the outbound bandwidth of your connection. Have you actually seen the uploader get a lot slower if one of the endpoints (AKA EVE Marketeer for the past week) is in trouble?

@Everyone: I'm sorry I am unable to fix the problems. It just grew a bit above my head, and I have little ideas on how to fix it except to do a total reset of the data involved, that might be painful, but at least the website would be working again.

Does it mean that you are going to lose all the history from 2009? It's one of the highest value your website provides.

If you can't save it, could you backup prices history in some text format and make it available to download? I'll gladly go dig it by hand if the alternative is to lose all.

Getting this error if i try to use the api: "http://www.evemarketeer.com/api/region/xml/10000043"

"Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 81 bytes) in /var/www/www.evemarketeer.com/core/mod/content/yapeal.php on line 115 "

Holy ******* hell. This was yapeal... I should've checked when I fixed the database, but the gathering of character, corporation and eve data is turned off by default when you generate the yapeal database (undoubtedly to protect us from too much data). In this case however, the only thing it was doing was preventing any useful data from being gathered at all.

The uploads seem not being processed anymore? Latest date i get is "2012-06-09" even if i upload data and wait couple of hours.

I may be able to point to EVE Addicts as somewhat of a recent "case study". They were running into the socket limit, seeing higher resource util on uploads, and generally struggling to keep up with the new EVEMon uploads. After disabling regular uploads and switching to EMDR, they found themselves dealing with a tiny fraction of the load they were before.

The reasons are numerous:

• Accepting many HTTP uploads in a short period of time involves dealing with that many HTTP connections.
• Rapidly opening and closing TCP sockets in those proportions can really kill a machine that is doing too many other things, or isn't configured just right (sysctl's ipv4 and tcp directives are huge here).
• EMDR's ZeroMQ underpinnings use one or two TCP connections at any time, max. The socket is left open, there is no constant opening/closing, and everything is compressed.
• 100% of EMDR's messages are compressed, unlike what you're dealing with in EMK.

I realize that you've got your own uploader and perhaps a slightly different set of people uploading, but the EVEMon flood sends us anywhere from 1-3 million uploads a day. A potential compromise may be to work with me to point your EMK format uploads at EMDR, then consume from EMDR. This would shift the burden, and potentially free your servers up somewhat. It would also be 100% revertible. You would then only need to take our sample consumer, add some concurrency, and parse from there.

Get in touch with me (GTalk, email, #emdr on irc.coldfront.net, whatever) if any of this appeals to you. If not, completely understand, let me know if we can help in any other ways.

First:
THANK YOU! EveMarketeer is pure awesomeness!

Second:
It looks like your site is being very slow.

Even uploading history I can't get prices newer than June 15.