These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

PHP/Crest/MarketData - Slow at 2+ seconds per item ?
Author
Previous Topic Next Topic
Eddard Moennan
Monte Inc
#1 - 2017-03-05 12:01:15 UTC
I'd like to fetch lowest sell price for 200+ items from a bunch of stations and dump the results into a database to then do more things with.

I have a process that can do this - Problem being that it much slower than I would like it to be.

Disabling the database writing doesn't improve the overall speed, so definitely on the fetching data side.
No individual item is slowing everything down; can see each market item taking 2 to 2.8 seconds.

So figure there is something just fundamentally wrong with my approach. My overall knowledge when it comes to programming is very limited - Hopefully an easy to identify flaw and fix:

Grab list of items from db, store as array
Loop through item list and build url for crest
Send url to crest via a curl function
Save the json response to an array
Loop through and find the lowest price
Write just the useful bits back to the db

Hoping someone smarter than I has a minute or five spare to provide some guidance

Pastebin with actual php being used, along with the output of curl_getinfo:
http://pastebin.com/KVbPJbDu

Thanks!
Althalus Stenory
Flying Blacksmiths
#2 - 2017-03-05 12:37:56 UTC
use the marketOrdersAll https://crest-tq.eveonline.com/market/10000002/orders/all/ and then filter on station / system and loop over the pages, it will be faster to get them this way.

Besides, you'd better use the ESI endpoint, since everything in CREST is in ESI (i believe, at least it's true for market) and as crest will be removed in a few month, you'd not have to change your code afterward :) https://esi.tech.ccp.is/latest/#!/Market/get_markets_region_id_orders

EsiPy - Python 2.7 / 3.3+ Swagger Client based on pyswagger for ESI
Eddard Moennan
Monte Inc
#3 - 2017-03-07 07:28:48 UTC
Yup - right you are. Never would have thought fetching 2000 orders would be faster than just 40 specific ones - but that seems to be the thing.

Thanks

Only done a tiny bit of reading about how/why/what ESI... not enough to actually make it go - a challenge for a future day.