These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

ESI. Mass market orders.

First post
Author
Previous Topic Next Topic
Jone Sad
Doomheim
#1 - 2017-01-16 11:57:29 UTC
Everyone know sites like http://evepraisal.com/ or other that use eve price apis.

When I do call to api (ESI MarketOrder), this call spends time. For example for 10-50 items it spent about minute maybe more (for fetching from eve api prices). But other sites much less time. Why?

First thought were that there is a cron that update prices in db (i think there is mongoDB) and use them (latest).

Is there any other solution how get prices quicker?

PS: With 200+ items (count assets for example) script can hang out for a 5-15 minutes...
Jone Sad
Doomheim
#2 - 2017-01-16 12:03:53 UTC
Unfortunately we need call x2 queries for one item (buy + sell). So for 200 items it will be about 400 call executions. Shocked
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#3 - 2017-01-16 13:00:02 UTC  |  Edited by: Steve Ronuken
You only need one call per item. If you don't pass the order_type, or pass it as 'all', you'll get all the orders for it (in the region)

What I'd normally suggest, when you're getting more than 20 odd items, is that you get the entire region and just process it all. Little more code required, substantially fewer queries. Just leave out the type_id parameter. Then call it with an incrementing pages parameter until you get no data back.

It's how I do things on https://market.fuzzwork.co.uk/ (well, I'm using crest, but that's because I wrote it before ESI came out, and I've not done the basic rework to shift it over to ESI. Need to mark out an hour or so for that.)

_most_ sites only work with aggregate data on their APIs, which is substantially lower volume, and is often cached for longer periods than ESI/CREST

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Jone Sad
Doomheim
#4 - 2017-01-23 01:12:20 UTC
Some calculations....

ESI allows get one item, sell\buy or sell+buy. It's ok... But, let's try to use it...

When we have a lot of items (simple example assets) we need do huge api calls. Even if we will call without type_id marketOrders we will get ATM about 25 pages, 10k entries each (25 * 10 000 = 250 000) . So our number (minimal) is 250k entries in one "update" of DB prices.

Weak point is that we can use "type_id" param (in query) only as INT not STRING (for example array of params like "34,35,...").

In this case, when we can limit result by several type_id we will have MUCH less entries and we can use it easy.

At this moment we have x2 cases for prices updates:

1. We cannot use global market without type_id because it return a lot of unneccessary entries, but we still have to parse them to get lowest\biggest price. So script spents A LOT OF time for this...
2. Query single item - yes it's quick enough for api but if we need get prices for 250 items - script can be hanged out (execution for a minute or even 5 it not a deal at all).

Developers of ESI pls do something with it. Pls change type_id to STRING (comma delimeter) so we can use it like "34,35,36".
Blacksmoke16
Resilience.
#5 - 2017-01-23 01:26:56 UTC  |  Edited by: Blacksmoke16
Or just cache the marketable items to its own table, update those prices every 15-30min then query the db prices vs using the ESI endpoint.

This is what i do, there are around 11,000 marketable items. Takes about a minute or so to update prices for jita for buy/sell on my desktop, (PI is like 5min). I use Steve's aggregate data in groups of 900 so it only takes about 12 calls to get all price data.
Jone Sad
Doomheim
#6 - 2017-01-23 02:22:03 UTC
Hm, optimized script retrieve 30 pages (cached seems like) data for 124 seconds. First run gave me 227 seconds... Not much. maybe is not bad... Will see in a future... But anyway CCP Devs you should add more params for order queries...