These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information

Author
Andy DelGardo
#561 - 2012-04-23 12:25:32 UTC
Hi,

small bug report: I managed to break the group_summary lists for my account like "http://www.evemarketeer.com/item/group_summary/1119" by adding a 6th kinda unusual region, now my summary page is complete empty and i also cant remove/change the regions anymore since the button is missing. I tried deleting my cookies, but seems this is stored on the account? Is there a way to fix this view without recreating a new account?

thx Andy
Controller Vrelk
The Scope
Gallente Federation
#562 - 2012-04-25 15:56:57 UTC
I've been getting a 502 error for the past day.
Zifrian
Deep Space Innovations
#563 - 2012-04-28 15:48:31 UTC
My users are reporting a locking error. I'll put in a work around for timing out long requests but this was the response I got to the following query:

http://www.evemarketeer.com/api/averages/10000043/90/34_35_36/xml

Warning: mysql_select_db() expects parameter 2 to be resource, boolean given in /var/www/www.evemarketeer.com/core/lib/cordb/cordb.php on line 38 Warning: mysql_ping() expects parameter 1 to be resource, boolean given in /var/www/www.evemarketeer.com/core/lib/cordb/cordb.php on line 29 Database connection #0 failed

Hope that helps. Thanks

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder

Callean Drevus
Perkone
Caldari State
#564 - 2012-04-28 15:52:58 UTC
Yes, the server was getting timeouts, and now mysql is angry I think.

Developer/Creator of EVE Marketeer

Scrapyard Bob
EVE University
Ivy League
#565 - 2012-04-29 21:09:54 UTC
On the unified uploader - If a particular end-point is causing problems (the queue hits 50), then the algorithm should simply skip adding any more items to that end-point's queue until it drops below 50 again. That would probably take care of the issue that if a particular end point is having trouble, the rest of the end points suffer because the upload tool slows down.

Just drop any items which would overflow the queue for that endpoint. If the endpoint is having trouble, to the point that the queue is full, it's probably not going to catch up for a while so no point queuing up additional items.
Callean Drevus
Perkone
Caldari State
#566 - 2012-05-06 08:48:27 UTC  |  Edited by: Callean Drevus
@Scrapyard: That is what should be happening right now. It drops any items above 50, which is why the queue remains at 50. The problem is more that if it is slow enough to grow to 50, it will probably always remain there, as items are added faster than they are uploaded.

As uploads are handled in different threads, the only thing I can see having any effect on upload speed for other endpoints is the outbound bandwidth of your connection. Have you actually seen the uploader get a lot slower if one of the endpoints (AKA EVE Marketeer for the past week) is in trouble?

@Everyone: I'm sorry I am unable to fix the problems. It just grew a bit above my head, and I have little ideas on how to fix it except to do a total reset of the data involved, that might be painful, but at least the website would be working again.

Developer/Creator of EVE Marketeer

Andy DelGardo
#567 - 2012-05-06 11:11:25 UTC  |  Edited by: Andy DelGardo
Callean Drevus wrote:

@Everyone: I'm sorry I am unable to fix the problems. It just grew a bit above my head, and I have little ideas on how to fix it except to do a total reset of the data involved, that might be painful, but at least the website would be working again.


I have no problems with a reset, since i personally use the site for short-term price observations, so i'm only interested in data that is at least 7 days old. If mr. Bob continues his upload's the site will be back at normal in no time :p

Only the SVR data will be problematic for the first month.

bye Andy
Vaerah Vahrokha
Vahrokh Consulting
#568 - 2012-05-08 09:05:30 UTC
Callean Drevus wrote:

@Everyone: I'm sorry I am unable to fix the problems. It just grew a bit above my head, and I have little ideas on how to fix it except to do a total reset of the data involved, that might be painful, but at least the website would be working again.


Does it mean that you are going to lose all the history from 2009? It's one of the highest value your website provides.

If you can't save it, could you backup prices history in some text format and make it available to download? I'll gladly go dig it by hand if the alternative is to lose all.
Callean Drevus
Perkone
Caldari State
#569 - 2012-05-13 22:09:00 UTC  |  Edited by: Callean Drevus
Vaerah Vahrokha wrote:
Callean Drevus wrote:

@Everyone: I'm sorry I am unable to fix the problems. It just grew a bit above my head, and I have little ideas on how to fix it except to do a total reset of the data involved, that might be painful, but at least the website would be working again.


Does it mean that you are going to lose all the history from 2009? It's one of the highest value your website provides.

If you can't save it, could you backup prices history in some text format and make it available to download? I'll gladly go dig it by hand if the alternative is to lose all.


Price history was happily not lost, as I had just a week ago made a backup dump of all the statistical information. That was pretty good timing in hindsight.

You can reach this dump too, by going to http://www.evemarketeer.com/var/dump

I hope ot periodically update that with the new data, so it will never be suddenly lost.

On another note, the website is FUNCTIONAL again, accepting uploads, and updating. God (and you all) knows it took me long enough to fix it. Blink

For now, I'll try to keep it operational, and free of stupid bugs, but adding new features will be on the backburner for a bit, until I get more free time again at the beginning of September. Sorry.

PS: Though I hopefully will get around to implementing UUF messages (for EMUU) and retrieving messages from EMDR. After all, EVEMon will be the most terrific datasource we ever had.

Developer/Creator of EVE Marketeer

Andy DelGardo
#570 - 2012-05-25 09:32:23 UTC
Getting this error if i try to use the api: "http://www.evemarketeer.com/api/region/xml/10000043"

"Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 81 bytes) in /var/www/www.evemarketeer.com/core/mod/content/yapeal.php on line 115 "
Devai Starchild
Starchild Engineering
#571 - 2012-05-25 23:57:20 UTC
Currently will not register any of my character data,

When trying to view anything under the character tab, it shows this error:

MYSQL ERROR OCCURRED: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'ORDER BY lft ASC' at line 1


You have the best marketing site out there. Please don't let it die.
Callean Drevus
Perkone
Caldari State
#572 - 2012-06-03 14:26:01 UTC
Holy ******* hell. This was yapeal... I should've checked when I fixed the database, but the gathering of character, corporation and eve data is turned off by default when you generate the yapeal database (undoubtedly to protect us from too much data). In this case however, the only thing it was doing was preventing any useful data from being gathered at all.

Developer/Creator of EVE Marketeer

Dragonaire
Here there be Dragons
#573 - 2012-06-03 19:44:49 UTC
Callean Drevus wrote:
Holy ******* hell. This was yapeal... I should've checked when I fixed the database, but the gathering of character, corporation and eve data is turned off by default when you generate the yapeal database (undoubtedly to protect us from too much data). In this case however, the only thing it was doing was preventing any useful data from being gathered at all.
Just for future reference I thought I'd let everyone know I changed the default so all the sections are on now. One of those legacy things that I'd never really thought about so never looked if it was still really neededP Got a new post up about it on the Yapeal thread. Sorry for the trouble it causedOops

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Andy DelGardo
#574 - 2012-06-14 15:36:50 UTC
The uploads seem not being processed anymore? Latest date i get is "2012-06-09" even if i upload data and wait couple of hours.
Callean Drevus
Perkone
Caldari State
#575 - 2012-06-15 06:56:50 UTC
Yeah, the queue was filling up. It seems the processing threads had stopped, which I only noticed today because of another error. It seems there is a somewhat big backlog, but it will eventually all be processed now.

Developer/Creator of EVE Marketeer

Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#576 - 2012-06-15 14:09:34 UTC
I may be able to point to EVE Addicts as somewhat of a recent "case study". They were running into the socket limit, seeing higher resource util on uploads, and generally struggling to keep up with the new EVEMon uploads. After disabling regular uploads and switching to EMDR, they found themselves dealing with a tiny fraction of the load they were before.

The reasons are numerous:

  • Accepting many HTTP uploads in a short period of time involves dealing with that many HTTP connections.
  • Rapidly opening and closing TCP sockets in those proportions can really kill a machine that is doing too many other things, or isn't configured just right (sysctl's ipv4 and tcp directives are huge here).
  • EMDR's ZeroMQ underpinnings use one or two TCP connections at any time, max. The socket is left open, there is no constant opening/closing, and everything is compressed.
  • 100% of EMDR's messages are compressed, unlike what you're dealing with in EMK.


I realize that you've got your own uploader and perhaps a slightly different set of people uploading, but the EVEMon flood sends us anywhere from 1-3 million uploads a day. A potential compromise may be to work with me to point your EMK format uploads at EMDR, then consume from EMDR. This would shift the burden, and potentially free your servers up somewhat. It would also be 100% revertible. You would then only need to take our sample consumer, add some concurrency, and parse from there.

Get in touch with me (GTalk, email, #emdr on irc.coldfront.net, whatever) if any of this appeals to you. If not, completely understand, let me know if we can help in any other ways.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Shinya Shazih
Viziam
Amarr Empire
#577 - 2012-06-15 22:28:21 UTC
Site seems broken?
I get the same price for three regions

=ImportXML("http://api.eve-marketdata.com/api/importxml_prices2.xml?char_name=shinya shazih&buysell=b&region_id=10000002&type_ids="&JOIN(",",$D4:$D30); "/emd/price")

It worked before, anything changed?
Golden Gnu
Lobach Inc.
#578 - 2012-06-16 11:57:33 UTC
First:
THANK YOU! EveMarketeer is pure awesomeness! Cool

Second:
It looks like your site is being very slow.

Creator of jEveAssets - the asset manager

"Download is the meaning of life, upload is the meaning of intelligent life"

Andy DelGardo
#579 - 2012-06-17 12:02:08 UTC
"special: separate with _ to query multiple id's at once, max 25"

This seems not to work for the "History" api call? Is this a api or documentation error?


Thx Andy
Vaerah Vahrokha
Vahrokh Consulting
#580 - 2012-06-19 16:21:23 UTC
Even uploading history I can't get prices newer than June 15.