These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

[PROPOSAL] Reduce API cache time by few minutes
Author
Previous Topic Next Topic
Max Kolonko
Caldari Provisions
Caldari State
#1 - 2011-12-14 07:41:18 UTC
Those of us that run CRON JOBS to get API data regularly (every hour/day etc...) suffer from the fact that API cache times usually exactly X Hours.

If this can be reduced by few minutes at least, that would make scheduled scripts make less cache request and would always get newest data.

When I set schedule to run every hour and cache is one hour here is what happening:

API cache time = 12:00

12:00 - job should starts.
12:01 - job actually starts (a lot of different jobs on server make for this delay)
12:02 - job finishes it tasks - API cache time somewhere between 13:01 and 12:02

13:00 - api hits cache data
14:00 -- job should starts.
14:01 - job actually starts (a lot of different jobs on server make for this delay)
14:02 - job finishes it tasks - API cache time somewhere between 15:01 and 15:02

So even though my data should be available every hour, script only can get it every other hour.

Reducing API cahce time for all >1h APIs by lets say 5 minutes should fix all delays and script execution times.

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Lutz Major
Austriae Est Imperare Orbi Universo
#2 - 2011-12-14 08:20:48 UTC
Although I'm with you that some cache timers could be reduced, I must say that your way of (mis)using the cache timers is a bit strange:

You shouldn't fetch new pages unless the cachedUntil expires. That way it is regardless when the timers run out and you don't have to have dozens of different cron jobs.
Talaan Stardrifter
State War Academy
Caldari State
#3 - 2011-12-14 12:07:29 UTC  |  Edited by: Talaan Stardrifter
Cron runs cache checker (every 5 minutes, at your pleasure)
Cache checker finds expired caches
cache checker runs cache refresher for expired caches
cache refresher sets new cache expiry at: cacheUntil + 5 minutes

At least, that's how my Agent works (Windows Service, not a cron, but same difference)
PsyKzz
Republic Military School
Minmatar Republic
#4 - 2011-12-14 14:58:02 UTC
Why not do save the time from the xml response, save to a db.

Then check cron every 5 mins.

If cache time > now then
update
else
;do nothing
end if

Simple.

(i hate having to write code like basic.)

Meh.
Dragonaire
Here there be Dragons
#5 - 2011-12-14 15:34:20 UTC
Or simply use an API library that does all of the above suggestions correctly to start with like Yapeal Blink

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Karbowiak
Sacred Templars
Fraternity.
#6 - 2011-12-14 16:06:20 UTC
The way we do it with EVE-KILL, is that we store every api key in the database, attach its last error code to it and its cached till time.

And everytime it runs out we re-request it. For every single key (we have 8k)
So, yeah.. lots of requests.. Shocked

oh and we use Pheal, **** Yapeal :P
Max Kolonko
Caldari Provisions
Caldari State
#7 - 2011-12-14 19:24:01 UTC  |  Edited by: Max Kolonko
PsyKzz wrote:
Why not do save the time from the xml response, save to a db.

Then check cron every 5 mins.

If cache time > now then
update
else
;do nothing
end if

Simple.

(i hate having to write code like basic.)



Un fortunatly i cant run cron every 5 minutes. I dont own server, so i cant set them up at my pleasure. I have available to me interface that allows them to run at most every full hour. so no matter if i check my timer (i do, i also use pheal) i still can get data 1 hour AFTER the cache expired.

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Tonto Auri
Vhero' Multipurpose Corp
#8 - 2011-12-14 22:46:21 UTC
Well, it's problem with your server, not with API.
Talk to your server provider, probably there's more than one way to skin your cat.

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison