These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Crest Market Orders / how fast can you collect it all?

First post
Author
Previous Topic Next Topic
Parkeur
Science and Trade Institute
Caldari State
#1 - 2015-03-07 00:38:24 UTC
For those of you who have written a Market Orders downloader using the CREST market orders endpoint....

How long does it take your app to download all buy and sell orders for all market types for The Forge region and store the data somewhere? I'm asking because it seems it would take many hours to download all orders for all regions and I wonder if others are doing it radically quicker than I am.

I'll risk embarrassing myself first: Blink

20 minutes (9189 market types, 228,218 orders).

The main components are:

Flow is implemented in Spring Integration XML with some Java for messaging endpoints
Jackson for JSON handling
Google oauth2 library
Data stored in HSQLDB
4 threads pulling data from CREST, rate limited at 25 per second
Pete Butcher
The Scope
Gallente Federation
#2 - 2015-03-07 07:18:04 UTC
Parkeur wrote:
For those of you who have written a Market Orders downloader using the CREST market orders endpoint....

How long does it take your app to download all buy and sell orders for all market types for The Forge region and store the data somewhere? I'm asking because it seems it would take many hours to download all orders for all regions and I wonder if others are doing it radically quicker than I am.

I'll risk embarrassing myself first: Blink

20 minutes (9189 market types, 228,218 orders).

The main components are:

Flow is implemented in Spring Integration XML with some Java for messaging endpoints
Jackson for JSON handling
Google oauth2 library
Data stored in HSQLDB
4 threads pulling data from CREST, rate limited at 25 per second



To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed. CREST is really slow if you want to do mass import. Most of the time your threads will be IO bound, so you can make a lot of them, especially with HT. Benchmark and find your optimal number. Of course, after downloading it all, your bottleneck would shift to the storage, but if you want to do some in memory processing, you'll be fine.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Kali Izia
GoomWaffe
#3 - 2015-03-08 05:36:51 UTC
Pete Butcher wrote:
To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed.

Wait what

You don't need to use multiple clients IDs.
The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can.

CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about.
Pete Butcher
The Scope
Gallente Federation
#4 - 2015-03-08 07:43:32 UTC
Kali Izia wrote:
Pete Butcher wrote:
To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed.

Wait what

You don't need to use multiple clients IDs.
The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can.

CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about.


As far as I know, there is a limit on the number of requests an app can make. Can't remember what it is tho. Do you have any info on that?

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Kali Izia
GoomWaffe
#5 - 2015-03-08 08:02:26 UTC
There isn't.
There are rate limits for public CREST, which you should adhere to for all non-market endpoints on authed CREST, but otherwise there are no other hard limits.
Pete Butcher
The Scope
Gallente Federation
#6 - 2015-03-08 08:15:41 UTC
Kali Izia wrote:
There isn't.
There are rate limits for public CREST, which you should adhere to for all non-market endpoints on authed CREST, but otherwise there are no other hard limits.


Oh, if that's the case, you can indeed use one app. My bad.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
CCP FoxFour
C C P
C C P Alliance
#7 - 2015-03-09 10:26:23 UTC
Kali Izia wrote:
Pete Butcher wrote:
To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed.

Wait what

You don't need to use multiple clients IDs.
The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can.

CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about.


Very much this. :D

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Parkeur
Science and Trade Institute
Caldari State
#8 - 2015-03-10 00:40:47 UTC  |  Edited by: Parkeur
Kali Izia wrote:
Pete Butcher wrote:
To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed.

Wait what

You don't need to use multiple clients IDs.
The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can.

CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about.


Thanks for this Kali Izia, I've done some tweaking on the database, a bit of code optimization, removed the rate limiter and upped the threadpool size and it now takes 6 minutes for all buy/sell orders for The Forge. For anyone that's interested:



  • 10,513 market items
  • 9193 actually have orders in The Forge
  • 229,699 total orders


  • 25 Java threads
  • 58 HTTP requests per second
  • 638 database rows per second inserted
  • Commit for each market item: ~20 rows - but JDBC batching benefits from much higher sizes than this


My 5-year old laptop was indeed breaking a sweat at this point so I don't think it will go much higher on my hardware but nevertheless it's not bad for a
Pete Butcher
The Scope
Gallente Federation
#9 - 2015-03-10 08:13:45 UTC  |  Edited by: Pete Butcher
CCP FoxFour wrote:
Kali Izia wrote:
Pete Butcher wrote:
To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed.

Wait what

You don't need to use multiple clients IDs.
The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can.

CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about.


Very much this. :D


Are sure there's no limit? I just fired up fetching market history at full speed (hundreds, if not thousands, of requests per second) and got a crapton of "Bad gateway" errors.

EDIT: And actually, the CREST server stopped responding with 753 requests still pending.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool