These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

Authed CREST rate limits

First post
Author
Previous Topic Next Topic
Pete Butcher
The Scope
Gallente Federation
#1 - 2015-03-10 20:48:17 UTC  |  Edited by: Pete Butcher
Since we have a public CREST rate limits thread, let's make this one also visible to the developers without the need to search.
So, let's dig into the problem. Here I got info that we should not care about limits for authed CREST. I followed that advice and pulled market data+history for < 10k items. I didn't actually measure the requests/s, since I was told not to care, but they all got send in a few s. At some point, the CREST server started throwing "Bad gateway" errors and eventually stopped responding at all. By lowering the rate, everything went fine. Therefore the official question is:

Is there or is there not a rate limit for authed CREST, and what it is if it exists (pretty sure it does)?

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Ortho Loess
Escalated.
OnlyFleets.
#2 - 2015-03-10 21:15:01 UTC
I asked back in November.

The answer was that it's still being decided.

FoxFour: maybe you have enough data to have set it now?
Pete Butcher
The Scope
Gallente Federation
#3 - 2015-03-10 21:25:56 UTC
Whatever it is, I can already say a standalone app on a typical desktop PC can easily reach it. And I really, really hope it's not on per-app basis.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Kali Izia
GoomWaffe
#4 - 2015-03-11 06:25:02 UTC
AFAIK you should adhere to the public rate limit for market history, even on authed CREST. It's just market orders you're allowed to hammer.
That might be contributing to the errors if it's rate limit related.

For what it's worth, I tried to reproduce this issue and I was only able to get errors when I set my max sockets to a stupidly high number (10000).
In that case I was getting 502s with "Gateway error communicating with EVE proxy".
I think that might actually just be overloading CREST. I believe rate limits are supposed to throw a 503 instead.

I'm sure FoxFour will be able to answer though.
Pete Butcher
The Scope
Gallente Federation
#5 - 2015-03-11 07:06:11 UTC
Kali Izia wrote:
AFAIK you should adhere to the public rate limit for market history, even on authed CREST. It's just market orders you're allowed to hammer.
That might be contributing to the errors if it's rate limit related.


I hope that's not the case, since it would be pretty damn stupid. Either we're using authed CREST with all its benefits or not. There's no middle ground here.
But, on the other hand, we have authed urls in public CREST which violates both common sense and HATEOAS, which crest seems trying to adhere to.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
CCP FoxFour
C C P
C C P Alliance
#6 - 2015-03-11 09:01:08 UTC  |  Edited by: CCP FoxFour
The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Pete Butcher
The Scope
Gallente Federation
#7 - 2015-03-11 09:06:02 UTC
CCP FoxFour wrote:
The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry.


If you want to see what's causing this, I can fire up the import at any time. The gateway errors are reproducible. What's more disturbing is the total lack of any response after some time. The server just seems to hang in there doing nothing.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Pete Butcher
The Scope
Gallente Federation
#8 - 2015-03-13 17:31:08 UTC
Any info on that one? It's still happening.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Ereshgikal
Wharf Crusaders
#9 - 2015-03-16 18:36:38 UTC
Pete Butcher wrote:
CCP FoxFour wrote:
The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry.


If you want to see what's causing this, I can fire up the import at any time. The gateway errors are reproducible. What's more disturbing is the total lack of any response after some time. The server just seems to hang in there doing nothing.


Just a random idea: Could it be your ISP (or someone else on the network between you and CCP) that considers your traffic harmful and blockholes it for a while? Given the behavior of some ISPs this wouldn't surprise me.
Though that wouldn't explain the 503 unless the ISP injects itself into your SSL stream... :(
But it might explain the loss of contact later.
Pete Butcher
The Scope
Gallente Federation
#10 - 2015-03-16 18:48:26 UTC
Ereshgikal wrote:
Pete Butcher wrote:
CCP FoxFour wrote:
The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry.


If you want to see what's causing this, I can fire up the import at any time. The gateway errors are reproducible. What's more disturbing is the total lack of any response after some time. The server just seems to hang in there doing nothing.


Just a random idea: Could it be your ISP (or someone else on the network between you and CCP) that considers your traffic harmful and blockholes it for a while? Given the behavior of some ISPs this wouldn't surprise me.
Though that wouldn't explain the 503 unless the ISP injects itself into your SSL stream... :(
But it might explain the loss of contact later.


No, it's crest. You just need to fire a lot of requests fast and it will die despite never reaching the official limit. Well, I'll be happy to actually see it die in a deterministic fashion, but it sometimes swallows requests without responding and my application just sits there wondering wtf.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Pete Butcher
The Scope
Gallente Federation
#11 - 2015-04-07 18:08:25 UTC  |  Edited by: Pete Butcher
For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Kali Izia
GoomWaffe
#12 - 2015-04-07 21:30:27 UTC
Pete Butcher wrote:
For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it.

From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy.

To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now.
Pete Butcher
The Scope
Gallente Federation
#13 - 2015-04-08 04:32:29 UTC
Kali Izia wrote:
Pete Butcher wrote:
For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it.

From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy.

To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now.


150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
CCP FoxFour
C C P
C C P Alliance
#14 - 2015-04-08 08:33:45 UTC  |  Edited by: CCP FoxFour
Pete Butcher wrote:
Kali Izia wrote:
Pete Butcher wrote:
For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it.

From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy.

To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now.


150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone.


Is this on pulling market data? What kind of rate are you hitting? As an FYI, and this will be explained in an upcoming dev blog, the authed rate limit will be lowered to about 100 req/s, possibly less. Public CREST will be bumped to 150 req/s.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Pete Butcher
The Scope
Gallente Federation
#15 - 2015-04-08 09:32:14 UTC
CCP FoxFour wrote:
Pete Butcher wrote:
Kali Izia wrote:
Pete Butcher wrote:
For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it.

From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy.

To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now.


150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone.


Is this on pulling market data? What kind of rate are you hitting? As an FYI, and this will be explained in an upcoming dev blog, the authed rate limit will be lowered to about 100 req/s, possibly less. Public CREST will be bumped to 150 req/s.


Did a little benchmark now and I sent 11364 requests in about 90 milliseconds, which translates to theoretical 126k req/s (in reality i make max couple of tens of thousands total). 100 req/s is quite disconnected from reality in comparison. Evernus users typically download orders and history for major market groups, which is about 20k requests per region. Imagine the time needed to download it all with 100/s. This limit may work for some Web applications with cron running in the background, but it's extremely bad for standalone or real time apps.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
CCP FoxFour
C C P
C C P Alliance
#16 - 2015-04-08 10:30:26 UTC
Pete Butcher wrote:
CCP FoxFour wrote:
Pete Butcher wrote:
Kali Izia wrote:
Pete Butcher wrote:
For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it.

From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy.

To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now.


150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone.


Is this on pulling market data? What kind of rate are you hitting? As an FYI, and this will be explained in an upcoming dev blog, the authed rate limit will be lowered to about 100 req/s, possibly less. Public CREST will be bumped to 150 req/s.


Did a little benchmark now and I sent 11364 requests in about 90 milliseconds, which translates to theoretical 126k req/s (in reality i make max couple of tens of thousands total). 100 req/s is quite disconnected from reality in comparison. Evernus users typically download orders and history for major market groups, which is about 20k requests per region. Imagine the time needed to download it all with 100/s. This limit may work for some Web applications with cron running in the background, but it's extremely bad for standalone or real time apps.


... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.

This is considered acceptable by us. It's less than the 5 minute cache for the market.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Pete Butcher
The Scope
Gallente Federation
#17 - 2015-04-08 10:51:39 UTC
CCP FoxFour wrote:

... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.

This is considered acceptable by us. It's less than the 5 minute cache for the market.


You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500?

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
CCP FoxFour
C C P
C C P Alliance
#18 - 2015-04-08 11:00:09 UTC
Pete Butcher wrote:
CCP FoxFour wrote:

... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.

This is considered acceptable by us. It's less than the 5 minute cache for the market.


You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500?


No. And I am. Response time while using public CREST is down a fair bit, at least on Sisi and TQ has better hardware. Go ahead and try it your self on Sisi: public-crest-sisi.testeveonline.com

I just timed a complete dump of the Tenerifis region using fairly simple and optimized script to pull the whole region (buy and sell orders for every time on the market). It took 3 minutes and 53 seconds. Pretty sure I could get that down with a bit of effort.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Pete Butcher
The Scope
Gallente Federation
#19 - 2015-04-08 11:16:40 UTC
CCP FoxFour wrote:
Pete Butcher wrote:
CCP FoxFour wrote:

... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.

This is considered acceptable by us. It's less than the 5 minute cache for the market.


You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500?


No. And I am. Response time while using public CREST is down a fair bit, at least on Sisi and TQ has better hardware. Go ahead and try it your self on Sisi: public-crest-sisi.testeveonline.com

I just timed a complete dump of the Tenerifis region using fairly simple and optimized script to pull the whole region (buy and sell orders for every time on the market). It took 3 minutes and 53 seconds. Pretty sure I could get that down with a bit of effort.


Ok, we'll see how it's going to perform when it gets live. I hope people won't get too upset, especially those doing region trading between hubs.

One last question - what will the response be when the limit gets exceeded?

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
CCP FoxFour
C C P
C C P Alliance
#20 - 2015-04-08 11:23:54 UTC
Pete Butcher wrote:
CCP FoxFour wrote:
Pete Butcher wrote:
CCP FoxFour wrote:

... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.

This is considered acceptable by us. It's less than the 5 minute cache for the market.


You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500?


No. And I am. Response time while using public CREST is down a fair bit, at least on Sisi and TQ has better hardware. Go ahead and try it your self on Sisi: public-crest-sisi.testeveonline.com

I just timed a complete dump of the Tenerifis region using fairly simple and optimized script to pull the whole region (buy and sell orders for every time on the market). It took 3 minutes and 53 seconds. Pretty sure I could get that down with a bit of effort.


Ok, we'll see how it's going to perform when it gets live. I hope people won't get too upset, especially those doing region trading between hubs.

One last question - what will the response be when the limit gets exceeded?


503 is what you should get when breaking the limit.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
12Next page