These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Public CREST market data - How to import complete data set... periodic

First post
Author
Previous Topic Next Topic
Ethan02
Succurit Research and Development
#1 - 2014-05-12 04:02:39 UTC  |  Edited by: Ethan02
Good morning,

found out about the CREST endpoint for market data history this weekend and played around with it.
Goal would be to mirror/import the data into my own database and have them then analyzed for trends.

If I see this correctly, then there are 10,377 tradeable types and 97 regions. Thus in order to "get it all" I would expect to make 1,006,569 queries against CREST. That is a lot and may put some load on the CCP server.

To put this into perspective, if I would do this in serial and allow a second of delay between the requests, then the whole transfer would take more than 11 days.

Am I missing something? Is there a way to get more data (eg. all regions) within one request? Is it allowed to hammer the CREST API like that? If not, what is the recommended way to do this.

At best I'd like to have the data on a weekly basis (then obviously only the last week is of interest).

Thanks for the help.
Gordon Newman
Deep Core Mining Inc.
Caldari State
#2 - 2014-05-12 07:54:51 UTC
Yeah that's a problem. Although you could use multiple parallel connections (the CREST API should be able to handle that load), there might not be relevant history data for every region/type combination, so that you'd get a lot of 'typeNotOnMarket' errors. I'm currently working on feeding CREST data into EMDR, so you only get history updates for active region/type combinations once every 24h. All you would have to do is to filter the EMDR feed for data uploaded by our consumer. That would require you to listen to the EMDR feed 24/7 to catch all updates.

Alternatively we could add an API/daily data dump for that to Element43, as we already have all historic data in our DB.

Co-Founder of element43 - Join our team! | EVE Live | EMDRMon | accounts-eve
CCP FoxFour
C C P
C C P Alliance
#3 - 2014-05-13 17:37:26 UTC  |  Edited by: CCP FoxFour
Slowley... but really why do you need all of this data from all of the regions?

Generally speaking please keep public CREST requests down to 10 requests per seconds. Then yes, go through all of those requests.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Scarna Jenari
Tritanium Industries and Technology
Goonswarm Federation
#4 - 2014-05-13 18:14:36 UTC
30 regions are WH space so there's no real point in querying those.
brammator
SUB ZERO.
#5 - 2014-06-19 10:56:26 UTC
Scarna Jenari wrote:
30 regions are WH space so there's no real point in querying those.



That's a real relief, now we could download all daily data in about 20 hours! Yay!
brammator
SUB ZERO.
#6 - 2014-06-19 10:59:56 UTC
CCP FoxFour wrote:
Slowley... but really why do you need all of this data from all of the regions?



Well, for example I would like to search for best trade routes. That definitely need data about all regions.
CCP FoxFour
C C P
C C P Alliance
#7 - 2014-06-19 12:41:34 UTC
brammator wrote:
CCP FoxFour wrote:
Slowley... but really why do you need all of this data from all of the regions?



Well, for example I would like to search for best trade routes. That definitely need data about all regions.


OK, well at this time there are no plans to offer a get multiple option. So either slim the number of regions/types you are asking about or wait 17 hours for a full refresh.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#8 - 2014-06-19 13:32:28 UTC
Also please ensure you are checking your type has a marketGroupID before passing it to the endpoint. The SDE contains that information in the column marketGroupID.

You should also be able to only do items with a typeID < 350000 as everything above is DUST.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.