These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
Previous page123Next page
 

Devsite Blog: A note on CREST rate limits

First post
Author
Previous Topic Next Topic
Pete Butcher
The Scope
Gallente Federation
#21 - 2015-11-19 10:37:18 UTC
Dulcet wrote:
Better yet, maybe give us a way to grab all the market data from the region in one area instead of having to traverse all the types. I'm sure that would be better than the 2 million+ req per app per day scenario for those of us that want to analyze all the market data.



That would the optimal way - an endpoint for bulk import (based on type id filter) per region. This will be in line with all applications which do not need to discover which regions/types exist (I bet almost all), would save a ton of load on the servers, and a ton of time for the users. I've been pushing for use-case centric API since CREST went online.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Kazuno Ozuwara
Science and Trade Institute
Caldari State
#22 - 2015-11-19 13:25:27 UTC
Pete Butcher wrote:
Dulcet wrote:
Better yet, maybe give us a way to grab all the market data from the region in one area instead of having to traverse all the types. I'm sure that would be better than the 2 million+ req per app per day scenario for those of us that want to analyze all the market data.



That would the optimal way - an endpoint for bulk import (based on type id filter) per region. This will be in line with all applications which do not need to discover which regions/types exist (I bet almost all), would save a ton of load on the servers, and a ton of time for the users. I've been pushing for use-case centric API since CREST went online.


But bigger responses tend to be very slow (looking at market types endpoint: requesting 13 huge pages concurrently consumes ~20 sec, and ~6 sec when cached). Also using filters is not good for server side caching.
Maybe market data per market group will be optimal.

!!SCIENCE!!
Pete Butcher
The Scope
Gallente Federation
#23 - 2015-11-19 13:44:30 UTC
Kazuno Ozuwara wrote:
But bigger responses tend to be very slow (looking at market types endpoint: requesting 13 huge pages concurrently consumes ~20 sec, and ~6 sec when cached). Also using filters is not good for server side caching.
Maybe market data per market group will be optimal.


It's not a problem when you do it right. Make a cache per type per region, which is most likely done anyway, and just make a union of the result set, and send it to the user. I don't imagine it being a big deal. Note that such approach will always be faster than making thousands upon thousands of requests to fetch the exact same data - both on server and client side. Somebody just has to make it. Right now, everybody looses:

  • users - CREST is both slow and unreliable, leading to both developer and end-user frustration
  • CCP - the servers are being hammered by thousands/millions of unnecessary requests

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
CCP FoxFour
C C P
C C P Alliance
#24 - 2015-11-19 14:45:50 UTC
Pete Butcher wrote:
Kazuno Ozuwara wrote:
But bigger responses tend to be very slow (looking at market types endpoint: requesting 13 huge pages concurrently consumes ~20 sec, and ~6 sec when cached). Also using filters is not good for server side caching.
Maybe market data per market group will be optimal.


It's not a problem when you do it right. Make a cache per type per region, which is most likely done anyway, and just make a union of the result set, and send it to the user. I don't imagine it being a big deal. Note that such approach will always be faster than making thousands upon thousands of requests to fetch the exact same data - both on server and client side. Somebody just has to make it. Right now, everybody looses:

  • users - CREST is both slow and unreliable, leading to both developer and end-user frustration
  • CCP - the servers are being hammered by thousands/millions of unnecessary requests


The performance of our backend servers seems to be... well fine. Trying to figure out why things take so long as we speak. Assuming things were fast, which we admit they are not, we wouldn't mind the large number of requests.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Iam Widdershins
Project Nemesis
#25 - 2015-11-21 05:37:42 UTC
Dulcet wrote:
Today I was able to pull in ALL of the market data for all applicable market type, buy/sell, in all regions, in just about 4 hours. I have to imagine this is probably fast enough for most applications. This is using rate limits of 100 req/sec.

Unfortunately, I am still receiving a lot of errors (dropped connections, socket hang ups, and the random missing data). Over the course of the 2 million requests I had about 150k errors of which I simply did a re request. I should note that if I go above 100 req/sec I see ALOT more disconnects, so I'm staying away from the suggested max of 150 req/sec.

Do you know what your maximum concurrent connections is? This seems to be a key factor in the failure rates we've been seeing.

Lobbying for your right to delete your signature
CCP FoxFour
C C P
C C P Alliance
#26 - 2015-11-21 12:28:02 UTC
Out of curiosity how many connections are most people opening?

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Pete Butcher
The Scope
Gallente Federation
#27 - 2015-11-21 12:33:54 UTC
CCP FoxFour wrote:
Out of curiosity how many connections are most people opening?


In my case - 6 at the moment. Before the switch to public CREST it could be as high as 60 and it usually worked.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Iam Widdershins
Project Nemesis
#28 - 2015-11-21 19:05:28 UTC
CCP FoxFour wrote:
Out of curiosity how many connections are most people opening?

I WAS opening about as many as I needed to get a response rate up near the limit, up to 150 actually. I had 150 threads running through two timed semaphores to give a 3ms minimum spacing (not sure if this is necessary but it seems polite) and to keep the rate at 150 requests per any given second. If everything were to go spectacularly well this could open one connection per thread.

Since I am accessing several endpoints with varying ratelimits I've added a third semaphore that introduces a hard limit on the number of concurrent requests in process; even with 60 connections opened at once, or indeed anything more than about 20, the error rate begins to skyrocket, as at the old limit of 150 I would be receiving JSON failure messages on around 15-30% of my calls.

Lobbying for your right to delete your signature
CCP FoxFour
C C P
C C P Alliance
#29 - 2015-11-21 21:22:08 UTC
Iam Widdershins wrote:
CCP FoxFour wrote:
Out of curiosity how many connections are most people opening?

I WAS opening about as many as I needed to get a response rate up near the limit, up to 150 actually. I had 150 threads running through two timed semaphores to give a 3ms minimum spacing (not sure if this is necessary but it seems polite) and to keep the rate at 150 requests per any given second. If everything were to go spectacularly well this could open one connection per thread.

Since I am accessing several endpoints with varying ratelimits I've added a third semaphore that introduces a hard limit on the number of concurrent requests in process; even with 60 connections opened at once, or indeed anything more than about 20, the error rate begins to skyrocket, as at the old limit of 150 I would be receiving JSON failure messages on around 15-30% of my calls.


https://eveonline-third-party-documentation.readthedocs.org/en/latest/crest/intro/

Check down the bottom, notice section on rate limiting, see max concurrent connections is 20. :) Anything more than 20 and our NGINX proxy is just going to start saying **** you. I assume it tosses 503's but not 100%.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Iam Widdershins
Project Nemesis
#30 - 2015-11-22 09:36:09 UTC
CCP FoxFour wrote:
Check down the bottom, notice section on rate limiting, see max concurrent connections is 20. :) Anything more than 20 and our NGINX proxy is just going to start saying **** you. I assume it tosses 503's but not 100%.

Thanks! I hadn't found that in my googling forays; I'll keep it down to 20 connections then.

I have to mention, though, this is fairly restrictive/difficult for those farther away from the servers: I have around 150 ping, which means that even if the server fetches my pages instantaneously, I'm not going to get more than around 133 pages per second unless I start utilizing HTTP/1.1 pipelining. And let me tell you, libraries that support pipelining are few, far between, and hard to use. For any language. If your language has a readily available tool that supports pipelined GET requests on a single connection, you are lucky indeed.

Lobbying for your right to delete your signature
Kazuno Ozuwara
Science and Trade Institute
Caldari State
#31 - 2015-11-22 19:39:11 UTC
Iam Widdershins wrote:
And let me tell you, libraries that support pipelining are few, far between, and hard to use. For any language..

What?
It's hard like calling one function.
Which language\environment has no reliable lib with HTTP pipelining?

!!SCIENCE!!
Iam Widdershins
Project Nemesis
#32 - 2015-11-23 09:22:00 UTC
Kazuno Ozuwara wrote:
Iam Widdershins wrote:
And let me tell you, libraries that support pipelining are few, far between, and hard to use. For any language..

What?
It's hard like calling one function.
Which language\environment has no reliable lib with HTTP pipelining?

Python? C#? Java? Why, what are you using?

Lobbying for your right to delete your signature
Kazuno Ozuwara
Science and Trade Institute
Caldari State
#33 - 2015-11-23 15:13:49 UTC
Iam Widdershins wrote:
Kazuno Ozuwara wrote:
Iam Widdershins wrote:
And let me tell you, libraries that support pipelining are few, far between, and hard to use. For any language..

What?
It's hard like calling one function.
Which language\environment has no reliable lib with HTTP pipelining?

Python? C#? Java? Why, what are you using?


.NET supports pipelining by default (using it right now).
5 min of googling for for Java and Py: http://www.innovation.ch/java/HTTPClient/advanced_info.html#pipelining, https://gist.github.com/coady/f9e1be438ba8551dabad.

!!SCIENCE!!
Pete Butcher
The Scope
Gallente Federation
#34 - 2015-11-23 18:31:26 UTC
In C++ you have a crapton of HTTP libs with transparent pipelining support. Nothing hard there.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Iam Widdershins
Project Nemesis
#35 - 2015-11-25 08:10:41 UTC
Kazuno Ozuwara wrote:
.NET supports pipelining by default (using it right now).
5 min of googling for for Java and Py: http://www.innovation.ch/java/HTTPClient/advanced_info.html#pipelining, https://gist.github.com/coady/f9e1be438ba8551dabad.

That's good, I'm glad to hear .NET has it.

The python recipe you linked, however, is pretty close to the only thing available short of learning how to use cURL, and it seems pretty hacky; it does, however, look like what I'll end up using. It might not be obvious, but the set of _HTTPConnection__state to idle is a sketchy, un-pythonic hijack of the connection object's private internal state to trick it into sending additional requests without waiting, counter to its designed behavior.

Pete Butcher wrote:
In C++ you have a crapton of HTTP libs with transparent pipelining support. Nothing hard there.

Haha, aside from C++, yeah. I should really learn that language sometime.

Lobbying for your right to delete your signature
Silvia Sotken
HC - Redemption
#36 - 2015-11-26 23:34:40 UTC
Lol it just clicked. I get a lot of ConnectionErrors, now I know why and not much I can do about it.
Never thought of doing a tracert, while I'm not as good at programming as you guys, I'm learning what I can.
While it was a little problem under Authed, under Public it is a lot more time consuming to run my data feeds.

Tracing route to public-crest.eveonline.com [87.237.38.221] over a maximum of 30 hops:

1 7 ms 1 ms 1 ms mygateway1.ar7 [10.1.1.1]
2 * * * Request timed out.
3 10 ms 10 ms 12 ms cpcak3 [101.98.0.1]
4 9 ms 8 ms 10 ms pts [101.98.5.20]
5 11 ms 8 ms 8 ms pts [101.98.5.21]
6 10 ms 12 ms 9 ms bundle-100.bdr02.akl05.akl.vocus.net.nz [175.45.102.65]
7 134 ms 134 ms 132 ms bundle-10.cor01.akl05.akl.VOCUS.net.nz [114.31.202.100]
8 134 ms 136 ms 133 ms bundle-200.cor02.lax01.ca.VOCUS.net [114.31.202.47]
9 135 ms 134 ms 134 ms bundle-101.bdr02.lax01.ca.VOCUS.net [114.31.199.51]
10 134 ms 177 ms 168 ms v301.core1.lax2.he.net [64.62.151.125]
11 131 ms 137 ms 154 ms 100ge2-1.core1.lax1.he.net [72.52.92.121]
12 194 ms 198 ms 199 ms 100ge15-2.core1.ash1.he.net [184.105.80.201]
13 203 ms 201 ms 199 ms 100ge5-1.core1.nyc4.he.net [184.105.223.166]
14 272 ms 266 ms 274 ms 100ge7-2.core1.lon2.he.net [72.52.92.165]
Zifrian
The Frog Pond
Ribbit.
#37 - 2015-12-28 17:39:59 UTC  |  Edited by: Zifrian
I've started looking at my market history calls and I'm noticing some errors from time to time but it's pretty reliable. I started getting 'forcibly disconnected' responses or the file wouldn't download completely. The second part I can handle (and should) but as I go higher in threads I'm noticing more issues. Most of this seems to have gone away though now with some settings changes to httpwebrequest in .net but they were coming up quite a bit. I'm using 20 threads and I get to about 1.5 to 2 minutes to do 2800 items. It's not too bad but I wish it were faster.

As mentioned above, it would be awesome to just download the entire market of data in one file per region. Individually is ok but I think most of us want to get it all and just cache it for use throughout the day. Certainly less server calls.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Soltys
Brutor Tribe
Minmatar Republic
#38 - 2016-02-04 14:26:37 UTC  |  Edited by: Soltys
From Evernus user's perspective - crest pulls for market analysis are now hours-long frustrating repetitions of connection closed errors. I mean hours literally, as it can take that much time to get any data from say Jita without errors.

It's near unusable now - and it worked just fine before switch to public endpoint, so I don't really suspect Evernus being at fault here. On its own it makes 6 connections and I've further limited it in options to 20 requests/s (and it constantly keeps failing).

Jita Flipping Inc.: Kovl & Kuvl
Cornbread Muffin
The Chosen - Holy Warriors of Bob the Unforgiving
#39 - 2016-02-26 23:47:46 UTC  |  Edited by: Cornbread Muffin
I just started using CREST to write apps, so figured I would chime in here as well. Bulk requests to CREST are glacial and error prone. Starts great and ends in tens of thousands of errors. =/

FWIW my company had the exact same issue with API access for one of our products. Especially when we started adding users in other continents and mobile usage increased (mobile = crap latency). We switched from many small requests to few large requests and it fixed the performance issues immediately. We tossed pagination as well because our users wanted all of the pages anyway. As a bonus, it's far easier to troubleshoot. Connection flooding is a hassle every step of the way. It's just not handled well at any layer along the pipe. Pushing bulk data only has a couple of failure points and you catch them once instead of needing to handle millions of errors for little chunks of data.

Highly compartmentalized APIs are best if the use case is a few, essentially random nodes from a large selection. If the use case means systems are going to visit all of the endpoints anyway to get the work done, you won't regret consolidating.

Pete's proposal or something similar is definitely the way to go.
Dagobert I Duck
THE HANG0VER
#40 - 2016-02-27 15:00:02 UTC

  • General Rate Limit: 150 requests per second
  • Concurrent Connections: 20


Keeping this numbers in mind I can make 130 -140 req/s without any errors. Increasing the concurrent connections to 21 leads to a lot of errors...

I fetch the market within 2 hours.
Previous page123Next page