These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
123Next pageLast page
 

eve-marketdata.com
Author
Previous Topic Next Topic
Kallahar
The Scope
Gallente Federation
#1 - 2011-09-12 20:01:54 UTC
This thread can be used to discuss eve-marketdata.com, including any support or feature requests. I also created an in-game channel called eve-marketdata that I often monitor for anyone needing help.

Kallahar
Gary Hagon
W.A.R. Inc.
#2 - 2011-09-13 04:12:07 UTC
Quick question, and Im sure our audience is wondering the same...

In what ways is eve-marketdata.com different from eve central?
Please be detailed.
NephelimSeelenfeuer
circle of lost souls
#3 - 2011-09-13 06:40:44 UTC  |  Edited by: NephelimSeelenfeuer
Could it be that /api/item_orders_sell.txt is kind of buggy right now? Seems to me that it's ignoring the type_id parameter.

Compare http://eve-marketdata.com/price_check.php?step=Show&type_id=28583&type=&region_id=10000030
with
http://eve-marketdata.com/api/item_orders_sell.txt?type_id=28583&region_ids=10000030
and
http://eve-marketdata.com/api/item_orders_sell.txt?type_id=34&region_ids=10000030 (different type_id, but same result)

Thanks in Advance
Venko Trenulo
Zendian Solutions
#4 - 2011-09-14 00:52:05 UTC
I love how easy it was to follow the instructions to get your market data into my ship-building spreadsheet to see where I could make the most profit. I was using OpenOffice, but the availability of this feature to import the data to Google Spreadsheets led me to move it over - a trivial operation. I imagine there's a way to construct filters and stuff in OpenOffice to do the same thing, but this process was straightforward, and I appreciate instant gratification.

Very slick! Thanks!
Kallahar
The Scope
Gallente Federation
#5 - 2011-09-14 21:07:00 UTC
eve-central vs eve-marketdata:
eve-central is no longer being updated. The developer said that he no longer plays eve, and just leaves the site up because it runs itself. Also, as more people upload data to me and move away from eve-central, his data gets worse and mine gets better :)

api data:
check again, I found a few bugs and have fixed them. There's also a new version of the API coming out real soon that will be way easier.

Kallahar
Kallahar
The Scope
Gallente Federation
#6 - 2011-09-14 21:09:50 UTC
Over the last month and a half I'm sure a lot of you noticed that the site was slow, and would occasionally go down (usually only for a few minutes while it auto-rebooted). The problem was that there are so many requests coming in (about a million a day now) that apache became a problem.

Apache is designed to spawn a separate thread for each request, and each request starts a php.cgi process. So that meant that each concurrent request took up like 50 megs of memory. I'm on a VPS, so the amount of memory the server gets is configurable, but I have to pay for it. Right now I'm paying about $70/mo in hosting fees.

So, getting off Apache became a priority. I spent about 2 weeks trying to get lighttpd to work and just had continuous problems. I finally gave up and tried out nginx which configured easily and has worked great in testing. I just deployed it to production, so all of you should be reading this on the new server!

This should allow me to drop my hosting costs to about $50/mo, and also scale up to about 50x the traffic there currently is.

SO, if you find any problems, or notice weird data, or have any connection problems, please let me know ASAP in-game to Kallahar or channel eve-marketdata.

Thanks!
Kallahar
The Scope
Gallente Federation
#7 - 2011-09-19 18:21:33 UTC
New API released! You can now pass in multiple type_id's, marketgroup_id's, and region_ids into all of the API functions.

Read the full details in the blog post:

http://eve-marketdata.com/blog/?p=34

Let me know if you have any problems or find any bugs!

Kallahar
Arligor
Sebiestor Tribe
Minmatar Republic
#8 - 2011-09-20 16:07:54 UTC
How do I get Eve-marketdata to update the information, for example the asset list?
Philderbeast
Native Freshfood
Minmatar Republic
#9 - 2011-09-20 21:30:20 UTC
can i put in a request for the API.

i would love to be able to retrieve price data for all the market hubs, as a manufacturer its often in my best interest to get materials in systems other then jita.
Aineko Macx
#10 - 2011-09-21 06:00:55 UTC
There's a small problem with the static dump downloads: The http server is returning the wrong MIME type. If you save the file you have to gunzip twice to get the actual txt's.

iveeCore 3.0: The PHP engine for industrial activities and CREST library
Spoofeydoo
Eternity INC.
Goonswarm Federation
#11 - 2011-10-03 00:50:43 UTC  |  Edited by: Spoofeydoo
While was uploading regional market data I noticed an error message telling me that the data I was sending wasn't being sent. This mystified me, everything looked fine until I noticed the date of the client had changed from October 2, 2011 to December 11, 2001.

Anyone else have this happen to them?

The error has reappeared but my date and time are correct. This is the error message:

It doesn't appear that your uploads are successful, is the uploader running, and do you have key "****************" entered?

Maybe it's time to introduce my system. Slackware Linux 13.37/ Wine 1.3.29. I have the 2.6 version of the Eve-marketdata installer.

October 3, 2011 I ran the uploader for five hours while I was at work without a problem.

I have a fruit cup and I'm not afraid to use it.
Aineko Macx
#12 - 2011-10-03 20:16:43 UTC
Indeed the uploader IGB page is claiming the data isn't being sent successfully, although the uploader app is running successfully and also the market item update statistics improve by running the automatic uploader. The displayed error also asks if i'm using the key "" (empty string), so I'm not sure there is a genuine issue or if it's only a display/JS error due to IGB changes...

iveeCore 3.0: The PHP engine for industrial activities and CREST library
Fenris Nihilus
Gallente Federation
#13 - 2011-10-04 00:19:36 UTC
Did you check the download page where it gives you a key for the uploader? I noticed it will think your not uploading properly while you still are if your on an alt with your main's key. It goes key per character so reload it in your IGB and plug it in.
Fenris Nihilus
Gallente Federation
#14 - 2011-10-04 00:21:15 UTC
Kallahar wrote:
eve-central vs eve-marketdata:
eve-central is no longer being updated. The developer said that he no longer plays eve, and just leaves the site up because it runs itself. Also, as more people upload data to me and move away from eve-central, his data gets worse and mine gets better :)

api data:
check again, I found a few bugs and have fixed them. There's also a new version of the API coming out real soon that will be way easier.

Kallahar



Could you prove this? Links? Because according to the site's dev blog the guy has been making updates for new api and posted pretty recent....

No offense I like your site and have started using it, but you don't have to bash your competition to become popular.
Gheent
Starshadow Corporation
#15 - 2011-10-04 05:17:41 UTC
Kallahar wrote:
Over the last month and a half I'm sure a lot of you noticed that the site was slow, and would occasionally go down (usually only for a few minutes while it auto-rebooted). The problem was that there are so many requests coming in (about a million a day now) that apache became a problem.

Apache is designed to spawn a separate thread for each request, and each request starts a php.cgi process. So that meant that each concurrent request took up like 50 megs of memory. I'm on a VPS, so the amount of memory the server gets is configurable, but I have to pay for it. Right now I'm paying about $70/mo in hosting fees.

So, getting off Apache became a priority. I spent about 2 weeks trying to get lighttpd to work and just had continuous problems. I finally gave up and tried out nginx which configured easily and has worked great in testing. I just deployed it to production, so all of you should be reading this on the new server!

This should allow me to drop my hosting costs to about $50/mo, and also scale up to about 50x the traffic there currently is.

SO, if you find any problems, or notice weird data, or have any connection problems, please let me know ASAP in-game to Kallahar or channel eve-marketdata.

Thanks!


While I do love the site, as a professional linux admin I want to point out that this issue is not due to Apache, but a poorly configured and/or compiled Apache (and PHP).

I would suggest checking out FPM-PHP if you're not already using it, it should also be compatible with nginx. It should help increase your performance even more.
Trenker
#16 - 2011-10-06 15:08:41 UTC
Gheent wrote:
I would suggest checking out FPM-PHP if you're not already using it, it should also be compatible with nginx. It should help increase your performance even more.


My two cent: If I run into a problem like this, I try hiphop-php first.
Scrapyard Bob
EVE University
Ivy League
#17 - 2011-10-13 15:48:59 UTC
Under the Statistics-Region data, it seems to hit the server pretty hard. It might be good to simply calculate that page no more often then once per hour (serving up an existing calculation if it hasn't been more then an hour since the previous calculation).

Also, on the uploads statistics page, what does the "Upload backlog" numbers indicate? I notice that EVE MarketData number tends to bounce up and down. I thought the flow was that you sent straight from the upload client to eve-marketdata and that it bypassed the evemarketeer site entirely?
Jenn Makanen
Doomheim
#18 - 2011-10-13 16:43:27 UTC
memcache.

It can seriously help, when you've got speed issues and regularly hit data you can afford not to have 'live'.

method for use, when you have the server installed is:

check to see if it's cached. if it is, unserialize the structure and use it.
if it's not, generate it, serialize the structure, stick it in the cache with a expiry date, then use it.


Far faster than running db queries.
Callean Drevus
Perkone
Caldari State
#19 - 2011-10-13 17:14:46 UTC
Scrapyard Bob wrote:
Under the Statistics-Region data, it seems to hit the server pretty hard. It might be good to simply calculate that page no more often then once per hour (serving up an existing calculation if it hasn't been more then an hour since the previous calculation).

Also, on the uploads statistics page, what does the "Upload backlog" numbers indicate? I notice that EVE MarketData number tends to bounce up and down. I thought the flow was that you sent straight from the upload client to eve-marketdata and that it bypassed the evemarketeer site entirely?


I think you have the wrong thread ;)

I agree it would be a good idea to cache it. It would be a good idea to cache a great many things ;)

Regarding the EMD backlog. Those are the updates I downloaded from eve marketdata I still have to process. If the number is going up, it means I get data faster than I process it. Later, when things settle down, the number goes down again.

Developer/Creator of EVE Marketeer
Salakka
Drunk Butchers Corporation
#20 - 2011-10-14 21:04:48 UTC
Few performance tips.
- "ip" based client tracking, how many page loads in x amount of time ... and give 503 if too much spamming.
- set "Expires" response header.
- follow users session, "If-Modified-Since" and give 304 if data still valid.
- disable/remove not needed php modules.

Tough I can't help much else as you are trying use normal php and basic web servers .. php is monster of memory usage when starting to get some traffic.

Oh, by the way, Google Docs spreadsheet seems to obey http header Expire information (at least some extent) and automatically updates data :)
123Next pageLast page