These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVE API and Public CREST discussion

First post First post First post
Author
Previous Topic Next Topic
Leebe
The Scope
Gallente Federation
#401 - 2014-05-26 17:03:24 UTC
CCP FoxFour wrote:
There are some new things deployed to Sisi today for testing (release date TBA):


Hmm.. looks very interesting .. but the overview link is just the id, so it's not really meaningfull. Maybe it would be possible to at least add the main parties to better help deciding if I'm interested to fetch the details ?

And will there be some kind of cache header we should obey ?
CCP FoxFour
C C P
C C P Alliance
#402 - 2014-05-26 17:10:40 UTC
Leebe wrote:
CCP FoxFour wrote:
There are some new things deployed to Sisi today for testing (release date TBA):


Hmm.. looks very interesting .. but the overview link is just the id, so it's not really meaningfull. Maybe it would be possible to at least add the main parties to better help deciding if I'm interested to fetch the details ?

And will there be some kind of cache header we should obey ?


All CREST calls have a cache header, please obey it.

This resource is for those interested in wars. Not a specific corporations wars. Later once you guys have access to the corporation resource on CREST there will be a link to that corporations wars. This is all wars.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
ItsmeHcK1
Immortalis Inc.
Shadow Cartel
#403 - 2014-05-27 21:30:36 UTC
Will all the API changes be listed in the patch notes?
I ask mainly because I am a lazy bastard.
CCP FoxFour
C C P
C C P Alliance
#404 - 2014-05-28 09:06:16 UTC  |  Edited by: CCP FoxFour
ItsmeHcK1 wrote:
Will all the API changes be listed in the patch notes?
I ask mainly because I am a lazy bastard.


No. Watch this thread or specifically this post from this thread: https://forums.eveonline.com/default.aspx?g=posts&m=4384137#post4384137

I also will announce this stuff on my twitter and generally r/EVETech if you prefer to monitor those places.

The reason for this is because the API and CREST will very likely be deployed separately from the EVE server. Kronos goes out on Tuesday, the API and CREST will probably go out on Wednesday, but maybe Tuesday depending how things go, but mainly I am just not 100% sure.

Sorry!

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Sentenced 1989
#405 - 2014-05-28 15:46:27 UTC  |  Edited by: Sentenced 1989
http://public-crest.eveonline.com/incursions/

How often is this updated

Atm there is new incursion up, spawned about 35 minutes. On crest it shows up every few minutes, stays on for 2-5 minutes then goes down for 2-5 minutes, then shows up again, etc

Can we get some clarification if its supposed to do that, or how long till it gets there and stays there after spawn, etc?

Regarding cache, where can I see how long the cache is for incursions?

The Incursion Guild

Epic Arc Guide
CCP FoxFour
C C P
C C P Alliance
#406 - 2014-05-28 15:55:44 UTC
Sentenced 1989 wrote:
http://public-crest.eveonline.com/incursions/

How often is this updated

Atm there is new incursion up, spawned about 35 minutes. On crest it shows up every few minutes, stays on for 2-5 minutes then goes down for 2-5 minutes, then shows up again, etc

Can we get some clarification if its supposed to do that, or how long till it gets there and stays there after spawn, etc?

Regarding cache, where can I see how long the cache is for incursions?


The cache time for that endpoint is 1 hour. 3600 second. You can see it in the headers as: Cache-Control →public, max-age=3600

There really is no way for it to change every few minutes, that cache is on the nginx box. The server only gets asked once every hour.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Aquila Sagitta
Blue-Fire
#407 - 2014-05-28 16:42:33 UTC
============================================
Idea: Allow us to send isk via corp or personal wallet
Description: I'm currently building tools for my wh based corp and I'd love it if you could send isk with one click instead of having to go to wallet, go to corp wallet, transfer isk, find the person your paying, double check its the right person, type in amount, and then pay them. When you have to do this with 10-20 people 1-3 times a week it gets quite tedious!
============================================

Blue-Fire Best Fire
Sentenced 1989
#408 - 2014-05-28 16:55:36 UTC  |  Edited by: Sentenced 1989
CCP FoxFour wrote:
Sentenced 1989 wrote:
....


The cache time for that endpoint is 1 hour. 3600 second. You can see it in the headers as: Cache-Control →public, max-age=3600

There really is no way for it to change every few minutes, that cache is on the nginx box. The server only gets asked once every hour.


That's what I figured. I'll update my end of stuff so it doesn't ask every time, just once per hour, however you have some error then.

So far I haven't obeyed cache (just implemented the functionality first time, so my bad, sorry, I'll fix it asap), but CREST was reporting 5 incursion up, then a minute or two later 6 incursions up, then a minute or two later 5 incursions up, etc...

If server is asked once every hour, how come I've got different result every few minutes out of CREST?

One more thing though, if I might ask, how is this handled on multiple queries from different locations?
As in I know dotlan does the query as well, will their query produce same cache result for rest of us?
More in detail, all I see in header is Cache-Control: public, max-age=3600, I don't see exact time when it is over as in "cached until", so what I am asking will somebodies else call trigger this countdown, or does every caller get desperate cache file with his own timer?

The Incursion Guild

Epic Arc Guide
Tek Handle
Vanishing Point.
The Initiative.
#409 - 2014-05-28 18:40:56 UTC  |  Edited by: Tek Handle
Idea: Complete the /corp/OutpostList by missing station settings
Description:
useAllianceStanding=1/0
reprocessingOutputDivision=someDivisionFlag
reinforcedModeTime=0000...2300
(reinforcedTimestamp=timestampWhenItLeavesReinforcedModeIfReinforcedOtherwiseBlank)

Idea: Add an endpoint which allows us to fetch existing clone contracts on a player outpost
Description: This should return rowsets with a corpID attribute and characterIDs as nested rows

Idea: Add an endpoint which allows us to fetch a list of offices in a player outpost
Description: rows with attributes
officeNumber=1...n
publiclyAvailable=1/0
rentedByCorporationID=12345789
startDate=someTimestamp
rentPeriod=always30anyway?!

Idea: Add an endpoint which allows us to read the reinforcement time configured for iHubs (and a state as well as the stateTimestamp comparable to the /corp/StarbaseList endpoint)

Idea: Add an endpoint which allows us to fetch a list of customs offices and their setting (standings+reinforced time+timestampWhenItLeavesReinforcedMode)
Wollari
Dirt Nap Squad
#410 - 2014-05-29 10:39:32 UTC  |  Edited by: Wollari
Question for the WAR API

when the attacker or defender joins or leaves an alliance: how will this be displayed in the war data? Will the attacker change from a corporation to an alliance? Does the alliance become an ally or does this war gets closed and a new war pops up for the alliances? Same applies to allies. If an ally joins an alliance or leaves an alliance during the war.

That's a question when tracking wars I've to be aware what data CAN change and which data IS static

DOTLAN EveMaps | Your out-of-game map, navigation toolset, sov database, etc. since 2008
CCP FoxFour
C C P
C C P Alliance
#411 - 2014-05-29 12:54:02 UTC
Wollari wrote:
Question for the WAR API

when the attacker or defender joins or leaves an alliance: how will this be displayed in the war data? Will the attacker change from a corporation to an alliance? Does the alliance become an ally or does this war gets closed and a new war pops up for the alliances? Same applies to allies. If an ally joins an alliance or leaves an alliance during the war.

That's a question when tracking wars I've to be aware what data CAN change and which data IS static


So, take this with a grain of salt as I don't know for sure, some testing will be required.

However from what I understand for attacker: If a corp is in an alliance the alliance declares the war. A corp leaving will not change the attacker as it doesn't matter who in the alliance declared it, the alliance is the attacker.

If an attacking corp leaves an alliance though I believe a new war is started. Same with if a defending corp joins or leaves an alliance. There should be, I think I included it, a fromWarID or fromWar field if that happens... not 100% sure as its a day off here and I don't have the code or docs in front of me.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#412 - 2014-05-29 13:01:56 UTC
Aquila Sagitta wrote:
============================================
Idea: Allow us to send isk via corp or personal wallet
Description: I'm currently building tools for my wh based corp and I'd love it if you could send isk with one click instead of having to go to wallet, go to corp wallet, transfer isk, find the person your paying, double check its the right person, type in amount, and then pay them. When you have to do this with 10-20 people 1-3 times a week it gets quite tedious!
============================================



This MAY happen with CREST but definitely not anytime soon and even then we really don't know if we want to allow this.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#413 - 2014-05-29 13:29:39 UTC
Sentenced 1989 wrote:
CCP FoxFour wrote:
Sentenced 1989 wrote:
....


The cache time for that endpoint is 1 hour. 3600 second. You can see it in the headers as: Cache-Control →public, max-age=3600

There really is no way for it to change every few minutes, that cache is on the nginx box. The server only gets asked once every hour.


That's what I figured. I'll update my end of stuff so it doesn't ask every time, just once per hour, however you have some error then.

So far I haven't obeyed cache (just implemented the functionality first time, so my bad, sorry, I'll fix it asap), but CREST was reporting 5 incursion up, then a minute or two later 6 incursions up, then a minute or two later 5 incursions up, etc...

If server is asked once every hour, how come I've got different result every few minutes out of CREST?

One more thing though, if I might ask, how is this handled on multiple queries from different locations?
As in I know dotlan does the query as well, will their query produce same cache result for rest of us?
More in detail, all I see in header is Cache-Control: public, max-age=3600, I don't see exact time when it is over as in "cached until", so what I am asking will somebodies else call trigger this countdown, or does every caller get desperate cache file with his own timer?


I really shouldn't have posted, I was clearly very tired.

TL;DR yes, that makes sense, it can happen, just obey the cache timer.

Long version: We have 2 nginx boxes serving public CREST on a DNS round robin. The incursion endpoint is very rarely hit. What likely happened is someone made a single GET which hit one of the nginx boxes. A while later someone made another GET which hit the second nginx box. The second one got newer data. I was under the impression that once a person started making requests they would only get one of the nginx boxes which is why I said you should not be getting different results. If however you were looking up the IP every request and going back and forth between boxes there is a chance you would be getting different results.

This should basically answer your second question though. The two nginx boxes have local caches, they can be out of sync, depends where you end up.

We very much on purpose don't tell you when the cache expires. This is pretty standard from my knowledge of the web. If we told everyone when the cache expires everyone would query at that time. Instead we just tell you how long the cache lasts and you obey that. It does mean you may not get the data at exactly the newest possible time, but for us it means the load is spread instead of our servers getting crushed. :)

Hope that helps! :D

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Sentenced 1989
#414 - 2014-05-29 14:02:17 UTC
CCP FoxFour wrote:


I really shouldn't have posted, I was clearly very tired.

TL;DR yes, that makes sense, it can happen, just obey the cache timer.

Long version: We have 2 nginx boxes serving public CREST on a DNS round robin. The incursion endpoint is very rarely hit. What likely happened is someone made a single GET which hit one of the nginx boxes. A while later someone made another GET which hit the second nginx box. The second one got newer data. I was under the impression that once a person started making requests they would only get one of the nginx boxes which is why I said you should not be getting different results. If however you were looking up the IP every request and going back and forth between boxes there is a chance you would be getting different results.

This should basically answer your second question though. The two nginx boxes have local caches, they can be out of sync, depends where you end up.

We very much on purpose don't tell you when the cache expires. This is pretty standard from my knowledge of the web. If we told everyone when the cache expires everyone would query at that time. Instead we just tell you how long the cache lasts and you obey that. It does mean you may not get the data at exactly the newest possible time, but for us it means the load is spread instead of our servers getting crushed. :)

Hope that helps! :D


Yep, helped a lot, thanks for taking your time to explain it.
Anyways, I've redone some stuff on my end, so now I pull your response into local file and my server won't check for new data in 60 minutes. Also to even more limit the load, I didn't tie it to cron job, so only on request of data (when somebody actually opens my page) it asks for new data if the local copy is older then 60 minutes, should make everybody happy :)

And I've even learned how to add user agent to my request :D \o/

The Incursion Guild

Epic Arc Guide
CCP FoxFour
C C P
C C P Alliance
#415 - 2014-05-29 14:04:47 UTC
Sentenced 1989 wrote:
CCP FoxFour wrote:


I really shouldn't have posted, I was clearly very tired.

TL;DR yes, that makes sense, it can happen, just obey the cache timer.

Long version: We have 2 nginx boxes serving public CREST on a DNS round robin. The incursion endpoint is very rarely hit. What likely happened is someone made a single GET which hit one of the nginx boxes. A while later someone made another GET which hit the second nginx box. The second one got newer data. I was under the impression that once a person started making requests they would only get one of the nginx boxes which is why I said you should not be getting different results. If however you were looking up the IP every request and going back and forth between boxes there is a chance you would be getting different results.

This should basically answer your second question though. The two nginx boxes have local caches, they can be out of sync, depends where you end up.

We very much on purpose don't tell you when the cache expires. This is pretty standard from my knowledge of the web. If we told everyone when the cache expires everyone would query at that time. Instead we just tell you how long the cache lasts and you obey that. It does mean you may not get the data at exactly the newest possible time, but for us it means the load is spread instead of our servers getting crushed. :)

Hope that helps! :D


Yep, helped a lot, thanks for taking your time to explain it.
Anyways, I've redone some stuff on my end, so now I pull your response into local file and my server won't check for new data in 60 minutes. Also to even more limit the load, I didn't tie it to cron job, so only on request of data (when somebody actually opens my page) it asks for new data if the local copy is older then 60 minutes, should make everybody happy :)

And I've even learned how to add user agent to my request :D \o/


Awesome! That is exactly the way it should work and huge thanks for the user agent! :D

Any chance of sharing a link to what you are working on?

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Ideki
Wolf Brothers INC
United Neopian Federation
#416 - 2014-05-29 18:41:49 UTC
Hey FoxFour, love the Dev Spolight.
always nice to be able to put a real face on a name Big smile

Creator of The EVE Planetary Planner
CCP FoxFour
C C P
C C P Alliance
#417 - 2014-05-29 18:51:09 UTC
Ideki wrote:
Hey FoxFour, love the Dev Spolight.
always nice to be able to put a real face on a name Big smile


:) Glad you enjoyed it.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Malus Sentio
Subsidy H.R.S.
Xagenic Freymvork
#418 - 2014-05-29 21:11:44 UTC  |  Edited by: Malus Sentio
Idea: Allow us to query entire killmail history
Description: We need to be able to pull entire killmail histories to be able to develop better killboards that can compete with the established ones. Killboards play a massive part in telling the stories that happen in EVE.
CCP FoxFour
C C P
C C P Alliance
#419 - 2014-05-29 21:56:28 UTC
Malus Sentio wrote:
Idea: Allow us to query entire killmail history
Description: We need to be able to pull entire killmail histories to be able to develop better killboards that can compete with the established ones. Killboards play a massive part in telling the stories that happen in EVE.


Thats not a very good argument considering established killboards (aka zKillboard) offers an API to pull all the killmails they have.

You don't need us to help you compete.

To be honest, they also store them in a slightly better and more efficient way. If they were not offering an API to get killmails, which is actually easier than getting them from us because you don't need API keys and such, then I would be more considering of this.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Ogir Tenno
Marvel-Yukon
#420 - 2014-05-29 22:13:22 UTC
CCP FoxFour wrote:
Malus Sentio wrote:
Idea: Allow us to query entire killmail history
Description: We need to be able to pull entire killmail histories to be able to develop better killboards that can compete with the established ones. Killboards play a massive part in telling the stories that happen in EVE.


Thats not a very good argument considering established killboards (aka zKillboard) offers an API to pull all the killmails they have.

You don't need us to help you compete.

To be honest, they also store them in a slightly better and more efficient way. If they were not offering an API to get killmails, which is actually easier than getting them from us because you don't need API keys and such, then I would be more considering of this.


There's a clear argument against that: We are dependent from those APIs then. zKillboards shuts us down - we're screwed. Supporting that there should be a neutral API.