These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
Previous page12
 

Devsite Blog: New CREST Resource for Bulk Market Orders

First post First post
Author
Previous Topic Next Topic
Pete Butcher
The Scope
Gallente Federation
#21 - 2016-06-01 17:06:25 UTC
Dread Griffin wrote:
I understand i can get these bulk orders from The Forge for example by using https://crest-tq.eveonline.com/market/10000002/orders/all/ but how do I walk to that path from the root instead of hard coding it?

I've tried walking down these paths found in the root, but they don't seem to provide a way to the bulk path:
https://crest-tq.eveonline.com/market/prices/
https://crest-tq.eveonline.com/market/groups/
https://crest-tq.eveonline.com/market/types/


It's in the individual region endpoints.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Dread Griffin
Pator Tech School
Minmatar Republic
#22 - 2016-06-01 17:40:20 UTC
Pete Butcher wrote:
Dread Griffin wrote:
I understand i can get these bulk orders from The Forge for example by using https://crest-tq.eveonline.com/market/10000002/orders/all/ but how do I walk to that path from the root instead of hard coding it?

I've tried walking down these paths found in the root, but they don't seem to provide a way to the bulk path:
https://crest-tq.eveonline.com/market/prices/
https://crest-tq.eveonline.com/market/groups/
https://crest-tq.eveonline.com/market/types/


It's in the individual region endpoints.


Thanks, I guess that does make sense. I'll try it out.
Nuke Beta
Tacere Servitium
#23 - 2016-06-01 18:14:19 UTC
Pete Butcher wrote:
Nuke Beta wrote:
Pete Butcher wrote:
The main thing killing the performance is the serial nature of the endpoint - we cannot make concurrent requests for different pages; we need to fetch page by page. With Eve having ~60 regions, it means we can make at most ~60 requests per few seconds it takes to get a reply and parse it.

I propose another endpoint which will return the page count upfront, so we can use our rate limit in full and fetch the pages concurrently.


After I get the first reply I check the page count and queue up each of the remaining pages so I'm typically asynchronously downloading some number of them in no particular order (Python asyncio/aiohttp). Isn't this = "making concurrent requests for different pages"? I don't do Python professionally so I could certainly have missed the big picture...

My OCD is struggling with not being able to be sure that the data set that comes back is coherent, I think the cache can update in the middle of the download? It seems like it would be handy to have an indication of which cache "version" the data came from so you could redo the ones from the previous cache. However, I have a vague memory that there may be two cache servers that aren't synced so there may be no way to guarantee that you can get a coherent set?


The problem is - you don't know the url structure to make requests basing only on page count. If you assert one, your app will stop working on any change. That's why an endpoint with links to each page should solve all issues.


Doesn't an endpoint with links to each page have the same problem with the cache as my concerns with getting a coherent set of data? You query to get links to each page of data, cache updates, page count changes +/- 1, your query for the last page may fail or you may not get all the data?
Pete Butcher
The Scope
Gallente Federation
#24 - 2016-06-01 18:23:44 UTC
Nuke Beta wrote:
Pete Butcher wrote:
Nuke Beta wrote:
Pete Butcher wrote:
The main thing killing the performance is the serial nature of the endpoint - we cannot make concurrent requests for different pages; we need to fetch page by page. With Eve having ~60 regions, it means we can make at most ~60 requests per few seconds it takes to get a reply and parse it.

I propose another endpoint which will return the page count upfront, so we can use our rate limit in full and fetch the pages concurrently.


After I get the first reply I check the page count and queue up each of the remaining pages so I'm typically asynchronously downloading some number of them in no particular order (Python asyncio/aiohttp). Isn't this = "making concurrent requests for different pages"? I don't do Python professionally so I could certainly have missed the big picture...

My OCD is struggling with not being able to be sure that the data set that comes back is coherent, I think the cache can update in the middle of the download? It seems like it would be handy to have an indication of which cache "version" the data came from so you could redo the ones from the previous cache. However, I have a vague memory that there may be two cache servers that aren't synced so there may be no way to guarantee that you can get a coherent set?


The problem is - you don't know the url structure to make requests basing only on page count. If you assert one, your app will stop working on any change. That's why an endpoint with links to each page should solve all issues.


Doesn't an endpoint with links to each page have the same problem with the cache as my concerns with getting a coherent set of data? You query to get links to each page of data, cache updates, page count changes +/- 1, your query for the last page may fail or you may not get all the data?


Yes, it does have the same problem with cache, but you can fetch each page concurrently, rather in series.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Nuke Beta
Tacere Servitium
#25 - 2016-06-01 20:34:12 UTC
Pete Butcher wrote:
Nuke Beta wrote:
Pete Butcher wrote:

The problem is - you don't know the url structure to make requests basing only on page count. If you assert one, your app will stop working on any change. That's why an endpoint with links to each page should solve all issues.


Doesn't an endpoint with links to each page have the same problem with the cache as my concerns with getting a coherent set of data? You query to get links to each page of data, cache updates, page count changes +/- 1, your query for the last page may fail or you may not get all the data?


Yes, it does have the same problem with cache, but you can fetch each page concurrently, rather in series.


At the risk of beating a dead horse, doesn't the Jita region have more than 20 pages of data (you may also have another region already queued up)? So you can't get them all concurrently and there will always be a risk of the cache updating before you finish resulting in a different page count. Just another thing to handle in code I guess... Is it not possible to return some indication of the cache age (or ??) so you have a direct indication of the problem?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#26 - 2016-06-01 20:39:27 UTC
Jita has, currently, 9 pages. (up to 30k records per page)

Yes, there's potential cache issues. (Ran into them sometimes) Though you can mitigate it by getting the types at the edges separately.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Pete Butcher
The Scope
Gallente Federation
#27 - 2016-06-02 18:59:52 UTC
CCP FoxFour wrote:
Pete Butcher wrote:
There should be some filtering available. I've done some testing and making ~750 requests for individual items is almost 3x faster than making whole market requests for 12 regions (those include The Forge and Sinq Laison).


Can you do that testing again after todays deployment.


Whole market fetching is about 30% slower than individual types, because of lack of concurrency. That endpoint with links to all pages would be a really good idea, especially since all you have to do is make a list of strings in a loop.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Shiri Xuri
War Exchange
#28 - 2016-06-09 15:35:56 UTC
It seems like this path returns active market orders as well as market orders that are no longer valid (as in they are no longer available on the market). Is this intentional? I was hoping to be able to get all the market orders that are currently active on the market.
foxjazz
Center for Advanced Studies
Gallente Federation
#29 - 2016-06-18 19:51:40 UTC
I would like to suggest some technology that may be helpful in servicing subscribed requests.

First I don't know what the underlying database or filebase the system uses to hold or transact orders, and I really don't care unless it's sql server.

If it's sql server, then utilizing service broker in conjunction with signalr would be a nice way of handling push notifications to clients.

Either way, I would like to suggest using SignalR in allowing your clients to subscribe to push notifications (or web-socket) but SignalR is more robust and very fast.

with signalr the system could just post the notification of a change for subscribed items, and the client's would receive the simple json message on the change. itemid, stationid, price, buy/sell etc...

I am sure the traffic would be active within the farm, but the action of the subscribed users traffic would range based on the requested data.

foxjazz
Aineko Macx
#30 - 2016-06-19 05:54:32 UTC
Pete Butcher wrote:
Whole market fetching is about 30% slower than individual types, because of lack of concurrency. That endpoint with links to all pages would be a really good idea, especially since all you have to do is make a list of strings in a loop.

I find your performance findings strange, because even without concurrency pulling the 9 pages for The Forge is still much faster then querying 11k items @ 100reqs/s.
But yes, I have asked for collection page indexes a long time ago. Due to the lack of it this is one of the situations where I disregard the CREST convention and construct the URLs to fetch collection pages in parallel.

iveeCore 3.0: The PHP engine for industrial activities and CREST library
Pete Butcher
The Scope
Gallente Federation
#31 - 2016-06-19 06:47:27 UTC
Aineko Macx wrote:
Pete Butcher wrote:
Whole market fetching is about 30% slower than individual types, because of lack of concurrency. That endpoint with links to all pages would be a really good idea, especially since all you have to do is make a list of strings in a loop.

I find your performance findings strange, because even without concurrency pulling the 9 pages for The Forge is still much faster then querying 11k items @ 100reqs/s.
But yes, I have asked for collection page indexes a long time ago. Due to the lack of it this is one of the situations where I disregard the CREST convention and construct the URLs to fetch collection pages in parallel.


I have already noticed CREST performance varies a lot, for different people. Different proxies? Nevertheless, I'm starting to consider building links manually too. Whole market import currently is a bit too slow for typical market usage.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Previous page12