These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Silly question

First post
Author
Previous Topic Next Topic
Bob Niac
Aliastra
Gallente Federation
#1 - 2011-09-23 23:40:32 UTC
Why is the API polling all of its data directly from TQ? Isn't there a way to do a mini-dump of sorts to a database that is optimized for html/php/whatever voodoo you guys use? Not all of the info that is called needs to be real time.


I mean:

I tell program E to get data N.
E queries a proxy server (P) for N
P asks E for credentials, and succeeds in auth.
P then asks TQ for N.
N is converted to a usable format, and is cached on P for later use.
P hands off N to E.
???
Profit?

I would imagine a CPU limited poll of the TQ database would be going on in the background anyway, to update P. P would also get a full incremental dump during downtime. Err .. I think that works?

[u]I <3 Logistics:[/u] Pilot of all  T2 logi and my shiny Archon [deceased.] Also a Chimera which may or may not be horrid. I don't make games, I play them. I get that ppl are passionate about change. I post here to plant seeds. You see your idea as is? Holy **** you win! So let's post, and see what the DEVs and our peers use.
Johnathan Roark
Quantum Industries
#2 - 2011-09-24 04:23:59 UTC
What makes you think the api is querying a live database? Have you ever noticed that the api servers are available during downtime and often during patch deployments? I think its been mentioned by devs that its a replication of the database.

EVEVERIFY - A recruiting API Verification and Audit Tool

Also try out Yapeal for your php api needs
Bob Niac
Aliastra
Gallente Federation
#3 - 2011-09-25 15:43:20 UTC
Original dev blog thread from last month stated the api wasn't an optimized database.

[u]I <3 Logistics:[/u] Pilot of all  T2 logi and my shiny Archon [deceased.] Also a Chimera which may or may not be horrid. I don't make games, I play them. I get that ppl are passionate about change. I post here to plant seeds. You see your idea as is? Holy **** you win! So let's post, and see what the DEVs and our peers use.
Fishsticks Fred
Caldari Provisions
Caldari State
#4 - 2011-09-25 21:23:52 UTC
Johnathan Roark wrote:
What makes you think the api is querying a live database? Have you ever noticed that the api servers are available during downtime and often during patch deployments? I think its been mentioned by devs that its a replication of the database.

Pretty sure it's not available during patches, at least not the big expansion ones.
Dragonaire
Here there be Dragons
#5 - 2011-09-25 23:53:40 UTC
It is sometimes during most of the patch time but we also usually don't get it back for a day or two afterwards as well until they are sure everything is ok in game before turning the API back on. It all depends on the patch.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
CCP Prism X
C C P
C C P Alliance
#6 - 2011-09-26 08:56:01 UTC  |  Edited by: CCP Prism X
The API does query TQ-DB proper. It then uses heavy caching strategies to avoid fetching the same data.

There were multiple discussions to move it over to our archives for, at least, archived data and only query TQ for recent updates to the simulation but we never got any further than just talking about it when I was on the team with Stillman. I doubt there are any talks about it right now.

The API used to really tick me off when it came to DB scraping. I imagine that is why they decided to put a DB programmer into C# web service programming. First thing I did was implement the caching I mentioned earlier and now the API is a pretty cool dude by me. Given any run of TQ the API is hogging around 10%+-2% of the used CPU resources (that does not mean it actually uses 10% of our entire CPU resources). This number also applies to reports I've taken when certain, much used, caches are broken so they always hit the DB. In those cases we see abnormal execution counts, pages read from the service.. but the CPU utilization doesn't really suffer for it. The calls are all quite basic relational queries and ordered on the cluster so CPU isn't much of an issue. Of course in the cases of broken caches the API can start shared-locking up resources way more aggressively than it should preventing writes, but that's a worst case scenario. Up until now only inconsequential caches that store completely static data whose tables never actually need an exclusive lock, except on our content authoring servers, have broken like that.

That's this weeks Monday DB lecture, kids!

CCP Prism X

Programmer on Team Gridlock & Expert Ranter