EVE Technology Lab

To help with developers working with market data, we're adding a new Bulk Market Order endpoint to CREST. Find all of the details here.

This is excellent news, and a great idea.

Unsolicited advice: a thing that stuck out to me was that the size of the returned response was still very large. Perhaps you might look into alternate transport formats for this data? Something like protocol buffers work very well for serializing data in a very small way. It may or may not be practical; just throwing it out there as a way to reduce the payload's size further.

I would SOOOOO love to do this!

It would be awesome if it was possible to access bulk market, but with some sort of a sieve list. For example, instead of specifying individual type ids or fetching them all, why not allow for multiple types,

https://crest../market/xyz/orders/multiple?type=12345&type=234&type=345...

With maybe maximum number of type ids set to 100 so it doesn't overflow the 10k entries page response?

That way my application could be reduced to one query per region to sync all data, instead of 60+

With this new resource it should be about 30 requests at the high end for regions like The Forge and next week if I am able to get it up to 30,000 orders per page then it is only 10 requests at the most per region. The bigger issue is just the sheer size of it all. Whelp.

but if each typeID has been recently cached, couldn't the request including multiple IDs simply collect info on each item and send in bulk rather than triggering a cache for the specific request?

I send requests to eve-central all the time that include 28 itemIDs (I found that once I get into the 5 digit IDs, any more requests break the URL). this is an example of the first request I send eve-central for my spreadsheet.

http://api.eve-central.com/api/marketstat?typeid=34,35,36,37,38,39,40,377,380,393,394,399,400,434,438,439,440,442,443...etc

Couldn't crest handle it similarly?

Edit: I may not understand caching at all, I do think the data transfer from crest could be reduced a lot by letting us choose a station or something to filter the response. Is it a problem of processing vs bandwidth and they just prefer to heap it on bandwidth?

There should be some filtering available. I've done some testing and making ~750 requests for individual items is almost 3x faster than making whole market requests for 12 regions (those include The Forge and Sinq Laison).

How are the responses sorted? It doesn't seem to be sorted by time issued, time remaining or type id.

After I get the first reply I check the page count and queue up each of the remaining pages so I'm typically asynchronously downloading some number of them in no particular order (Python asyncio/aiohttp). Isn't this = "making concurrent requests for different pages"? I don't do Python professionally so I could certainly have missed the big picture...

My OCD is struggling with not being able to be sure that the data set that comes back is coherent, I think the cache can update in the middle of the download? It seems like it would be handy to have an indication of which cache "version" the data came from so you could redo the ones from the previous cache. However, I have a vague memory that there may be two cache servers that aren't synced so there may be no way to guarantee that you can get a coherent set?

The problem is - you don't know the url structure to make requests basing only on page count. If you assert one, your app will stop working on any change. That's why an endpoint with links to each page should solve all issues.