These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Querying History of all items of a Specific Region?

First post
Author
Previous Topic Next Topic
InsidiousMind Nardieu
The Scope
Gallente Federation
#1 - 2017-05-14 23:41:27 UTC  |  Edited by: InsidiousMind Nardieu
Hey all,

New on the forum here (and to using EveAPI's) but I've been developing quite a simple API tool for my own use in Rust, and I was wondering what the best API endpoint would be for querying the item history of all items of a specific region.

Just to test out my code, I tried iterating through all the typeID's and querying the /markets/{region_id}/orders/ endpoint on ESI, but this would take a painfully long time (there are alot of items in eve!).

Does anyone know of a tool/way to make this faster? I really don't want to wait 30min for my data to populate to the most up-to-date. I also don't want to put that much load at once on Eve's servers, though I'm quite certain they would be able to handle it.

If anyone would want to take a peak at what i'm trying to do, the repo with all the code is here: https://github.com/InsidiousMind/EveOnlineMarketFinder (under ESI-test branch)
Blacksmoke16
Imperial Academy
#2 - 2017-05-15 03:43:28 UTC
By history do mean like the low/high/volume per day as found in the history tab in the market? Or do you mean like a list of all active orders per region and the details for each?

If you mean the latter, then the one you are using would be the best option ESI wise doing it yourself. Another option would be using Steve's API:

https://market.fuzzwork.co.uk/api/

I use this to keep my market prices up to date, can do all about 12,000 items in about 45 seconds on my Pi (~25 on desktop).

If you mean the former then you could use the /market/{region_id}/history endpoint.
InsidiousMind Nardieu
The Scope
Gallente Federation
#3 - 2017-05-15 03:50:58 UTC  |  Edited by: InsidiousMind Nardieu
Blacksmoke16 wrote:
By history do mean like the low/high/volume per day as found in the history tab in the market? Or do you mean like a list of all active orders per region and the details for each?

If you mean the latter, then the one you are using would be the best option ESI wise doing it yourself. Another option would be using Steve's API:

https://market.fuzzwork.co.uk/api/

I use this to keep my market prices up to date, can do all about 12,000 items in about 45 seconds on my Pi (~25 on desktop).

If you mean the former then you could use the /market/{region_id}/history endpoint, (typeid is required)


Yeah I want the low price, high price, volume for buy/sell orders. Preferable to have some history (I really only need 30days, though) but if i can only get one day, so be it.

I am right now using /market/{region_id}/history endpoint, but if i want all the items then i have to query each item individually. There are alot of items in the eveverse (from my dump, I have ~32K?) and though ESI responds very quicky, sometimes the latency can be ~700ms. Multiply that by 32K and it's quite the time to wait for it to populate.


I will look into Steve's API, however, thanks for that.

as of now my app is actually half-functional so you can see my problem if you do a cargo run on it (assuming you have Rust build tools), you'll see each query coming out.
Blacksmoke16
Imperial Academy
#4 - 2017-05-15 04:02:56 UTC  |  Edited by: Blacksmoke16
In that case I'd either have a script to get every item's history data from every region at like 1am or something, this would give you the most accurate data as it is what you would see in game looking in the history tab for that item.

The other option is Steve's API, this would return each the current market data for regions or stations, so might not be as accurate. This would be faster since you can query like 900 items in one call.

Also do know that not all items can be sold on the market, so would be wasting calls getting data for those.

I use https://crest-tq.eveonline.com/market/types/ and iterate thru this and each page to get the items that can be sold on the market.
InsidiousMind Nardieu
The Scope
Gallente Federation
#5 - 2017-05-15 04:35:21 UTC
Blacksmoke16 wrote:
In that case I'd either have a script to get every item's history data from every region at like 1am or something, this would give you the most accurate data as it is what you would see in game looking in the history tab for that item.

The other option is Steve's API, this would return each the current market data for regions or stations, so might not be as accurate. This would be faster since you can query like 900 items in one call.

Also do know that not all items can be sold on the market, so would be wasting calls getting data for those.

I use https://crest-tq.eveonline.com/market/types/ and iterate thru this and each page to get the items that can be sold on the market.



Ahok makes sense. Thanks man!

I guess there's no quick way to query what items are on the market (w/o any other data?)
Blacksmoke16
Imperial Academy
#6 - 2017-05-15 04:41:57 UTC
There is a CREST endpoint for bulk market data, IE you give it a regionID and it returns ALL the market orders in that region. I am not sure if this includes citadels, most likely just stations.

https://developers.eveonline.com/blog/article/new-crest-resource-for-bulk-market-orders/
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#7 - 2017-05-15 12:05:03 UTC
Blacksmoke16 wrote:
There is a CREST endpoint for bulk market data, IE you give it a regionID and it returns ALL the market orders in that region. I am not sure if this includes citadels, most likely just stations.

https://developers.eveonline.com/blog/article/new-crest-resource-for-bulk-market-orders/



It's not including structures, other than ranged buy orders.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
InsidiousMind Nardieu
The Scope
Gallente Federation
#8 - 2017-05-15 15:22:44 UTC  |  Edited by: InsidiousMind Nardieu
Thanks guys!


I suppose I can just gather all the market orders and figure out the history myself. Though, that's a bit more work than I expected to do for this small project of mine. Though I might definitely use that endpoint just to figure out what types are on the market, and decrease the total # of request to query for market history.


I think I will setup a small API on a personal server of mine which will query the history everyday at ~1am, and keep a database of it's own like the first response suggested. Then i can just query my server whenever I need the latest data.

:-)

Cheers!