These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

CREST headers - CacheControl not accurate?

First post
Author
Previous Topic Next Topic
Zifrian
The Frog Pond
Ribbit.
#1 - 2016-01-04 03:10:57 UTC  |  Edited by: Zifrian
I've been working with CREST a bit and I've noticed that the CacheControl values isn't helpful for a few major uses. For instance I've noticed the following:
  • market/{region_id}/types/{type_id}/history/ - returns max-age=3600 (1 hour), but the only time Market history is updated is at midnight GMT each day.
  • /industry/facilities/ - returns max-age=3600 (1 hour) but these can currently only change at downtime

In each of these cases, the Date in the header is the date of the call.

For the others it seems to work fine but I'm wondering if there is something I'm reading wrong or if it's possible to get the next cache time from the server? Maybe this isn't easy to do (I'm not very experienced with web stuff), but if the server returned the time that the data would be available for update in the header, it would make it much easier to work with. I'm sure it would cut down on the number of meaningless calls to the server as well. The API seems to work like this so is this a limitation with CREST?

Note, it looks like the /market/prices was updated to 82800, which is 23 hours (it had been much less in the past).

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
CCP FoxFour
C C P
C C P Alliance
#2 - 2016-01-04 09:36:59 UTC
market/{region_id}/types/{type_id}/history/

God I need to rewrite this whole resource. It isn't linked from anywhere and cannot easily be linked. It's just a mess. Arg. Anyways, aside from ranting about my own terrible code the caching stuff. Those prices only update at midnight anyways since they don't include the "todays" price history information. So when the page is first requested we cache it "until midnight", however many seconds that might be, but set a cache time on the header to 1 hour.

The industry stuff we probably just cache for 1 hour because that is usually what we do for things that don't change much.

I will probably change the history resource to just be cached for 1 hour when I rewrite it. That was sort of a optimizing before it was needed sort of thing.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Zifrian
The Frog Pond
Ribbit.
#3 - 2016-01-04 17:00:21 UTC
CCP FoxFour wrote:
market/{region_id}/types/{type_id}/history/

God I need to rewrite this whole resource. It isn't linked from anywhere and cannot easily be linked. It's just a mess. Arg. Anyways, aside from ranting about my own terrible code the caching stuff. Those prices only update at midnight anyways since they don't include the "todays" price history information. So when the page is first requested we cache it "until midnight", however many seconds that might be, but set a cache time on the header to 1 hour.

The industry stuff we probably just cache for 1 hour because that is usually what we do for things that don't change much.

I will probably change the history resource to just be cached for 1 hour when I rewrite it. That was sort of a optimizing before it was needed sort of thing.

So maybe I don't understand caching but it seems like you would only want people to access data when there is data to retrieve. Setting them to one hour doesn't really help does it? I can add special cases but I like how the api cache dates work since I know I can just check against that and not over call stuff. The headers don't seem to have a cache date per-say but is that something we can get? Like for the market prices the date would just be 00:00:00 gmt on the next day. It would remove a lot of uncertainty to the calls...plus I can blame you all when it doesn't work heh.

Also, with market/prices...is there a reason for 23 hours? It seems kind of odd and I don't know why it's like that. For market purposes these change more often than that I thought.

Thanks

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
CCP FoxFour
C C P
C C P Alliance
#4 - 2016-01-05 08:24:53 UTC
Zifrian wrote:
So maybe I don't understand caching but it seems like you would only want people to access data when there is data to retrieve. Setting them to one hour doesn't really help does it?


Yes and no. The primary concern is we cache data correctly on our NGINX boxes which prevents requests from hitting the TQ cluster. So long as the data is cached in NGINX we could take a very large number of hits and not give a ****. NGINX is so very performant and on some pretty beefy machines. So we set the cache publicly to 1 hour because a number of libraries out there take the cache time for something and cache further requests to the same resource at the same time as the first request. It is a terrible practice but by setting the public cache time to 1 hour but caching for "until midnight" we protect ourselves and help our users.

Zifrian wrote:
The headers don't seem to have a cache date per-say but is that something we can get? Like for the market prices the date would just be 00:00:00 gmt on the next day. It would remove a lot of uncertainty to the calls...plus I can blame you all when it doesn't work heh.


I have been unable to find anything in NGINX that would allow me to do that. Our caching is done in NGINX. The CREST proxies will set the cache length and then NGINX handles the rest. If you are aware of a way to get NGINX to include that information I would gladly add it.

Zifrian wrote:
Also, with market/prices...is there a reason for 23 hours? It seems kind of odd and I don't know why it's like that. For market purposes these change more often than that I thought.


Market prices is just the rolling average, it doesn't change any more than once a day. Real time market data comes from the orders information. The resource you are talking about is just that average price used for estimates in the client.

Zifrian wrote:
Thanks


No problem! :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.