EVE Technology Lab

Hi
Looking at the way that the Market data is delivered, it's very much designed for a grab as needed approach. Knowing what EVE players are like however, there's undoubtedly a desire for being able to pull large amounts of data for aggregation, which requires hammering CREST for an extended period of time.

I was wondering how feasible it would be to instead have the servers generate a bulk file say once per day, with all active orders in it, one file for each region, which could then be compressed and then fetched by players. This would speed up how quickly most people would be able to get bulk data, level the playing field for people with reduced programming knowledge (the excel users basically) and I imagine would reduce the load on the server as it would have to deal with the same requests for each item in each region, and would send far less over the network.

I also just ran into this. Even with a "grab as needed" approach, some industry stuff I wanted quickly ran into needing several hundred sometimes even thousands of requests if I wanted more than what https://public-crest.eveonline.com/market/prices/ provides directly...

What about third party / player data sources? CCP set the cache timer to 5 minutes, and it would be nice to have access to fairly "live" data.

Given CCP's 20 connections / 150 requests limit, you can maybe keep 1 region fully updated from a single server box (buy and sell plus number of SDE items with market groups gave me 24738, so at 150 requests/sec about 3 minutes.

But scaling that out to all regions seems expensive. Cheapest I can think of using AWS is about $40 per year per ip address (on the assumption that CCP rate limit is per-ip), plus all the storage, io, bandwidth, etc. charges.

Well looks to me like the relevant public-crest API's are largely what the game client uses but with an update rate that is many times slower, so guess they didnt have time to make anything completely new.

Eve Central have an API (see http://dev.eve-central.com/evec-api/start ), including daily dumps, so for non-live datasets there is that. I believe they use CREST for at least some of it now (but could not find a clear "this is how we get data" page...), I guess they just run at the rate limit looping around all the types and regions

If you look at say http://api.eve-central.com/api/quicklook?typeid=34&regionlimit=10000002 , I generally see that they have the same "reported" time for all listed orders of a given type and region, which makes sense if either cache scraping or using CREST, that is how the data comes. It is only when you look at multiple types or regions that the times vary, but then the same happens using CREST directly right now, since it takes hours to crawl all the types/regions.

So just using quicklook with all the types and no region filter would get nearly the same thing?

Only way I see being clearly better is the "multiple IP" thing, and aggregating on a central server (such that say EVE-Centrals data set is always less than 30 minutes, rather than upto around say 4 hours)

Evernus uses eve-central as an alternative and they haven't complained (yet), so maybe it's not a problem.

Sure, but with this being EVE and all, there's no guarantee the data provided by all parties will be accurate. There's also the difference is collection and storage methods which make it more of a pain to combine and the risk that one would suddenly stop getting data creating a gap. Like I say, from my perspective I'll just be getting the data straight from CCP one way or the other, it just seems it would be better for us and them if they bulk fed it out too.

Evernus fetches data on demand from clients, so it will be burts of data from mutiple IPs which I imagine they are OK with. That's a bit different from one IP fetching their whole database through their API once/twice a day.

FOXFOURWHYDOYOUHATEME

Actually, it was designed this way for Dust. They then opened it up so we can get at it.


I'd suggest, if people want a firehose of market data, take a look at EMDR.


It doesn't let you request specific things, but it does pull most things, over time.


Long term, maybe we'll get a regions complete order book as a single download, though with the size it would be, it would have to be a slice in time, rather than 'live' data.

I take it as a (potential) good thing. Not that having a Dust-appropriate API means they can't also have an EVE-appropriate API, but at least there's some operational reason for things to be how they are. That's a superior alternative to "this is how we think an EVE-appropriate API should function".


Reasonably up-to-date snapshots would be sufficient for many uses and a bulk data endpoint could exist along side the existing resources. File size and the bandwidth they want to support is a legit issue, but bandwidth usage could be kept the same and service levels massively improved with a bulk data endpoint.

The last snapshot I was able to get covered the 23 empire regions and 10375 items (I took out stuff like Dust items from my requests, but that's most of the items in the game). That's 477,250 calls to the API. The response headers alone were 293MB and the payload (compressed on the wire) was 97.3MB. It took a little under 2 hours, so transferred at a ~.5Mbps overall. This was at ~70 rq/s, so roughly half the rate limit. I assume that's the bulk of the orders that exist in the entire game, but I've never requested null-sec markets so I don't know that for sure.

The same data gzipped as one file is 56.1MB. 10.5MB of that is The Forge. If I was adding a bulk data endpoint I'd pull out a lot of the redundant data (e.g. return no data at all for itemTypeIds that have no orders, remove most of the bulky urls), but even if the data was identical it would still be about 1/8th the size of what they are sending me now, most of which is wasted on headers. I could get the same market data at the same .5Mbps every ~15 minutes and it wouldn't take any more data transfer on their part. That's a worthwhile improvement IMO.

I disagree. This is just a bad excuse - given Dust market works like in Eve, the same issues still apply there. Bad design is still bad design; inconsistencies are still inconsistencies; unreliability is still unreliability. Telling it was metaphorically copy-pasted to eve is just another argument nobody thought CREST through.