These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
Previous page12
 

CREST Market Data

First post
Author
Previous Topic Next Topic
William Kugisa
Perkone
Caldari State
#21 - 2016-03-26 23:32:32 UTC  |  Edited by: William Kugisa
What about the idea of just having a simplified transaction/change log?The existing API's, or once 3 months of logs are available would give anyone the initial data to then update from, so a CCP provided snapshot is not needed. That would also give live / near live data, without crawling or creating snapshots of the entire market at a comparable frequency. Ideally the cache time for this could be in seconds, rather than minutes or hours.

I think "snapshots" alone are less interesting if they are only once a day / every few hours, the "liveness" of such data would be comparable to EVE-Central, or crawling the current endpoint anyway. Also third party developers can create there own snapshot API's easily then.
Cornbread Muffin
The Chosen - Holy Warriors of Bob the Unforgiving
#22 - 2016-03-27 04:39:21 UTC  |  Edited by: Cornbread Muffin
William Kugisa wrote:
What about the idea of just having a simplified transaction/change log?The existing API's, or once 3 months of logs are available would give anyone the initial data to then update from, so a CCP provided snapshot is not needed. That would also give live / near live data, without crawling or creating snapshots of the entire market at a comparable frequency. Ideally the cache time for this could be in seconds, rather than minutes or hours.

I think "snapshots" alone are less interesting if they are only once a day / every few hours, the "liveness" of such data would be comparable to EVE-Central, or crawling the current endpoint anyway. Also third party developers can create there own snapshot API's easily then.


Diff updates would be brilliant as well. I agree that slow snapshots are worthless since people will just use the "real time" (5 minute cache) endpoints instead. Snapshots will only be used if they can get you fresher bulk data than hitting all endpoints individually can. Fortunately, this should be easy.

Based on the findings I put up in my previous post, I would think they could provide regional snapshots every 15 minutes with no issues. Even faster (or the same speed but with less bandwidth requirements) if they prune the data down to a minimal format.

The current cache is 5 minutes, so I'll use that as the definition of "real time". For the data I retrieved it breaks down like this: 




Name                               Size (gzip+headers)         bandwidth needed @ 5min updates
----------------------------------------------------------------------------------------
Current API (current format)              390.3 MB                    10.4 Mbps
Bulk Endpoint (current format)             56.1 MB                     1.5 Mbps
Bulk Endpoint (slim format)                 7.7 MB                     0.2 Mbps


The slim format looks like this:


{
    "regions": [
        {
            "id": "10000002",
            "lastUpdate: "2016-01-28T14:56:39",
            "orderFields": ['orderId', 'itemTypeId', 'issued', 'stationId', 'volumeEntered', 'volume', 'minVolume', 'range', 'price', 'duration', 'buy'],
            "orders": [
                [911346113, 0, "2016-03-02 06:40:50", 60006484, 22778, 22778, 1, "region", 14.00, 365, true],
                ...
            ]
        },
        ...
    ]
}


They already transfer to me at 0.5Mbps, I just get updates every few hours instead of every 5 minutes. Sending the same dataset 60x more often while still using 60% less bandwidth seems like a no-brainer. If bandwidth is still a big deal you could have bulk endpoints use protocol buffers or some other binary format. I just went with the simplest possible update from the current API.
Previous page12