These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVE API and Public CREST discussion

First post First post First post
Author
Previous Topic Next Topic
CCP FoxFour
C C P
C C P Alliance
#81 - 2014-04-20 09:30:53 UTC
qu1ckkkk wrote:
Polarina wrote:
I must've been completely blind! I've been on the lookout for these names for quite a while now. And FoxFour, thank you for all these improvements you've been making to the API, very much appreciated.

On a unrelated note, is it possible to change /corp/Locations.xml.aspx so that it doesn't fail completely on invalid asset IDs? In the minute it takes me to fetch and process the assets list, someone undocks in a ship, invalidating one of the asset IDs. For me, that's a relatively frequent event.

Currently, when that happens, I repeatedly split the list of asset IDs I request into two halves and call the API again on those two lists. This usually results in 9 to 11 requests instead of just one that would've otherwise been required.


The ContractItems API's have a similar thing going like explained in the quote, where I have to split the requests up in order to get the max details. This after you do the ContractsItems call for contractID's you just got via Contracts API...

But, what grinds me the most is the fact that a request for the corp AssetList would contain IDs that did not move, though, when you request the Locations API for those IDs (moments after receiving the IDs from the AssetList. endpoint), you'd get "135: Owner is not the owner of all itemIDs..." : \ I also have to split the assetIDs up and kinda ignore the errors though, which sucks.


Possibly because I just woke up and also because it's the weekend... but I am not really understanding what you are saying. Could you explain in more detail. Sorry. :(

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Max Kolonko
Caldari Provisions
Caldari State
#82 - 2014-04-20 10:09:24 UTC  |  Edited by: Max Kolonko
I think he meant that api call for corp assets gives him list of x ships
So he makes api locations call for those ships ids. And for se he gets error. Presumably because of those ships no lpnger being in corp hangar.

So i assume there is time diffrence beyween those api caching time.

Yep, assets 6 hous, locations 1 hour

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Polarina
EVE University
Ivy League
#83 - 2014-04-20 18:17:50 UTC  |  Edited by: Polarina
CCP FoxFour wrote:
Possibly because I just woke up and also because it's the weekend... but I am not really understanding what you are saying. Could you explain in more detail. Sorry. :(

Let me try to explain in a reproducible manner.

I pull the assets list from the API. From it, I compute what assets are new in it compared to what I already have in my database. Most of the time, it's just new ships appearing in our ship maintenance array, hangars being restocked; very ordinary normal stuff going on.

Downloading the assets list from the API takes about 20 seconds (it is huge), processing it and adding it to the database takes about 40 seconds. Finally, I now have a list of all the asset IDs of new ships / containers / corpses / stuff that came with the last API pull.

Now it's time to query the names of all those new assets with the /corp/Locations.xml.aspx endpoint. In order to save load on the API, I request the names in batches of 50 IDs at a time.

In the 60 seconds it took me to get to this point, someone has undocked in a ship that happens to be one of the asset IDs I'm just about to request. This means that the ship is no longer the corporation's, it would no longer appear in our assets list (if it weren't cached, that is).

I fire off the request! The API notices that one of the asset IDs does not belong to us (but did, just a minute ago), invalidates the whole request and throws an error.
"The API" wrote:

<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
<currentTime>2014-04-20 18:08:13</currentTime>
<error code="135">Owner is not the owner of all itemIDs or a non-existant itemID was passed in. If you are not trying to scrape the API, please ensure your input are valid locations associated with the key owner.</error>
<cachedUntil>2014-04-21 06:08:13</cachedUntil>
</eveapi>

I have no idea which asset ID failed; could be just one, could be more if a fleet undocked or something. All I can do now is divide & conquer.

Where are my clothes?
Proddy Scun
Doomheim
#84 - 2014-04-21 04:46:24 UTC
PI updates in WH seem to be less than 100% reliable. And now I got an idea of maybe why. Two on demand services (PI & wh system) --

no wonder 1 in 6 time you can reconfigure or renew extraction, double check it at end before logging off hours later...and find out when you go back days later to collect that it all stopped at the next DT.

I guess the on demand nature of things means stuff doesn't always get stored in non-volatile DB if server priorities and log off mix wrong. On demand server processes simply do not always flush all data to non-volatile DB if coders missed one caffeine injection when reviewing the code logic.
CCP FoxFour
C C P
C C P Alliance
#85 - 2014-04-21 09:31:37 UTC
Proddy Scun wrote:
PI updates in WH seem to be less than 100% reliable. And now I got an idea of maybe why. Two on demand services (PI & wh system) --

no wonder 1 in 6 time you can reconfigure or renew extraction, double check it at end before logging off hours later...and find out when you go back days later to collect that it all stopped at the next DT.

I guess the on demand nature of things means stuff doesn't always get stored in non-volatile DB if server priorities and log off mix wrong. On demand server processes simply do not always flush all data to non-volatile DB if coders missed one caffeine injection when reviewing the code logic.


WHs are no more on demand than any other solar system, if you can find a way to reproduce this defect please file a bug report. If you have filed one recently, last few months, let me know under what character so I can try and look it up.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#86 - 2014-04-21 09:32:38 UTC
Polarina wrote:
CCP FoxFour wrote:
Possibly because I just woke up and also because it's the weekend... but I am not really understanding what you are saying. Could you explain in more detail. Sorry. :(

Let me try to explain in a reproducible manner.

I pull the assets list from the API. From it, I compute what assets are new in it compared to what I already have in my database. Most of the time, it's just new ships appearing in our ship maintenance array, hangars being restocked; very ordinary normal stuff going on.

Downloading the assets list from the API takes about 20 seconds (it is huge), processing it and adding it to the database takes about 40 seconds. Finally, I now have a list of all the asset IDs of new ships / containers / corpses / stuff that came with the last API pull.

Now it's time to query the names of all those new assets with the /corp/Locations.xml.aspx endpoint. In order to save load on the API, I request the names in batches of 50 IDs at a time.

In the 60 seconds it took me to get to this point, someone has undocked in a ship that happens to be one of the asset IDs I'm just about to request. This means that the ship is no longer the corporation's, it would no longer appear in our assets list (if it weren't cached, that is).

I fire off the request! The API notices that one of the asset IDs does not belong to us (but did, just a minute ago), invalidates the whole request and throws an error.
"The API" wrote:

<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
<currentTime>2014-04-20 18:08:13</currentTime>
<error code="135">Owner is not the owner of all itemIDs or a non-existant itemID was passed in. If you are not trying to scrape the API, please ensure your input are valid locations associated with the key owner.</error>
<cachedUntil>2014-04-21 06:08:13</cachedUntil>
</eveapi>

I have no idea which asset ID failed; could be just one, could be more if a fleet undocked or something. All I can do now is divide & conquer.


OK, thats what I thought you meant. Your wording previously just made me a little confused, sorry.

I will have a look at this ASAP, I think I know why it was done but have a better idea for fixing what it tried to fix.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Max Kolonko
Caldari Provisions
Caldari State
#87 - 2014-04-21 10:50:55 UTC  |  Edited by: Max Kolonko
CCP FoxFour wrote:
Polarina wrote:
CCP FoxFour wrote:
Possibly because I just woke up and also because it's the weekend... but I am not really understanding what you are saying. Could you explain in more detail. Sorry. :(

Let me try to explain in a reproducible manner.

I pull the assets list from the API. From it, I compute what assets are new in it compared to what I already have in my database. Most of the time, it's just new ships appearing in our ship maintenance array, hangars being restocked; very ordinary normal stuff going on.

Downloading the assets list from the API takes about 20 seconds (it is huge), processing it and adding it to the database takes about 40 seconds. Finally, I now have a list of all the asset IDs of new ships / containers / corpses / stuff that came with the last API pull.

Now it's time to query the names of all those new assets with the /corp/Locations.xml.aspx endpoint. In order to save load on the API, I request the names in batches of 50 IDs at a time.

In the 60 seconds it took me to get to this point, someone has undocked in a ship that happens to be one of the asset IDs I'm just about to request. This means that the ship is no longer the corporation's, it would no longer appear in our assets list (if it weren't cached, that is).

I fire off the request! The API notices that one of the asset IDs does not belong to us (but did, just a minute ago), invalidates the whole request and throws an error.
"The API" wrote:

<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
<currentTime>2014-04-20 18:08:13</currentTime>
<error code="135">Owner is not the owner of all itemIDs or a non-existant itemID was passed in. If you are not trying to scrape the API, please ensure your input are valid locations associated with the key owner.</error>
<cachedUntil>2014-04-21 06:08:13</cachedUntil>
</eveapi>

I have no idea which asset ID failed; could be just one, could be more if a fleet undocked or something. All I can do now is divide & conquer.


OK, thats what I thought you meant. Your wording previously just made me a little confused, sorry.

I will have a look at this ASAP, I think I know why it was done but have a better idea for fixing what it tried to fix.


Speaking of that. My way around it is making a lot of 1 item calls. I have 20 calls to single items every 10 minutes on CRON (so keeping well under 32 per second as the job itself takes about 5 seconds on avarage to compleete it could be increased to a 100 calls per job, but my assets have less than 1000 entries for named objects, so i dont mind going slow on this). Initially I populated my corp assets DB with name values until i only got errors for the rest. Then my CRON job is doing the job of populating whatever new shows up in my assets DB. It also updates names if they change (my script takes oldest 20 entries in terms od time of name checking, and all new items have date set to 0, so if less than 20 new items shows up in DB whatever if left goes to updating the names of existing items and updating time of name checking).

If i get error I also update the time of name checking, so this item is not checked again until it turn comes around. It have a lot of downsides and its not perfect in any means, its just a way to always get names for things that are in assets and dont have to worry that api return error for whole call.

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
CCP FoxFour
C C P
C C P Alliance
#88 - 2014-04-21 13:41:01 UTC
Max Kolonko wrote:
CCP FoxFour wrote:
Polarina wrote:
CCP FoxFour wrote:
Possibly because I just woke up and also because it's the weekend... but I am not really understanding what you are saying. Could you explain in more detail. Sorry. :(

Let me try to explain in a reproducible manner.

I pull the assets list from the API. From it, I compute what assets are new in it compared to what I already have in my database. Most of the time, it's just new ships appearing in our ship maintenance array, hangars being restocked; very ordinary normal stuff going on.

Downloading the assets list from the API takes about 20 seconds (it is huge), processing it and adding it to the database takes about 40 seconds. Finally, I now have a list of all the asset IDs of new ships / containers / corpses / stuff that came with the last API pull.

Now it's time to query the names of all those new assets with the /corp/Locations.xml.aspx endpoint. In order to save load on the API, I request the names in batches of 50 IDs at a time.

In the 60 seconds it took me to get to this point, someone has undocked in a ship that happens to be one of the asset IDs I'm just about to request. This means that the ship is no longer the corporation's, it would no longer appear in our assets list (if it weren't cached, that is).

I fire off the request! The API notices that one of the asset IDs does not belong to us (but did, just a minute ago), invalidates the whole request and throws an error.
"The API" wrote:

<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
<currentTime>2014-04-20 18:08:13</currentTime>
<error code="135">Owner is not the owner of all itemIDs or a non-existant itemID was passed in. If you are not trying to scrape the API, please ensure your input are valid locations associated with the key owner.</error>
<cachedUntil>2014-04-21 06:08:13</cachedUntil>
</eveapi>

I have no idea which asset ID failed; could be just one, could be more if a fleet undocked or something. All I can do now is divide & conquer.


OK, thats what I thought you meant. Your wording previously just made me a little confused, sorry.

I will have a look at this ASAP, I think I know why it was done but have a better idea for fixing what it tried to fix.


Speaking of that. My way around it is making a lot of 1 item calls. I have 20 calls to single items every 10 minutes on CRON (so keeping well under 32 per second as the job itself takes about 5 seconds on avarage to compleete it could be increased to a 100 calls per job, but my assets have less than 1000 entries for named objects, so i dont mind going slow on this). Initially I populated my corp assets DB with name values until i only got errors for the rest. Then my CRON job is doing the job of populating whatever new shows up in my assets DB. It also updates names if they change (my script takes oldest 20 entries in terms od time of name checking, and all new items have date set to 0, so if less than 20 new items shows up in DB whatever if left goes to updating the names of existing items and updating time of name checking).

If i get error I also update the time of name checking, so this item is not checked again until it turn comes around. It have a lot of downsides and its not perfect in any means, its just a way to always get names for things that are in assets and dont have to worry that api return error for whole call.


Yeaaaaaa the fact that you are having to do things like that is just... /face_palm

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
qu1ckkkk
The Warp Core Stabilizers
#89 - 2014-04-21 15:01:38 UTC
CCP FoxFour wrote:

Yeaaaaaa the fact that you are having to do things like that is just... /face_palm


Yeah, like this horrible piece of code? :D

https://github.com/eve-seat/seat/blob/dev/app/eveapi/corporation/CorporationStarbaseDetail.php#L48

Proud developer of SeAT! A Simple Eve API & Corporation Management Tool.

Project Page: https://github.com/eveseat/seat
Max Kolonko
Caldari Provisions
Caldari State
#90 - 2014-04-22 08:50:14 UTC  |  Edited by: Max Kolonko
CCP FoxFour wrote:
Max Kolonko wrote:
CCP FoxFour wrote:
....


Speaking of that. My way around it is making a lot of 1 item calls. I have 20 calls to single items every 10 minutes on CRON (so keeping well under 32 per second as the job itself takes about 5 seconds on avarage to compleete it could be increased to a 100 calls per job, but my assets have less than 1000 entries for named objects, so i dont mind going slow on this). Initially I populated my corp assets DB with name values until i only got errors for the rest. Then my CRON job is doing the job of populating whatever new shows up in my assets DB. It also updates names if they change (my script takes oldest 20 entries in terms od time of name checking, and all new items have date set to 0, so if less than 20 new items shows up in DB whatever if left goes to updating the names of existing items and updating time of name checking).

If i get error I also update the time of name checking, so this item is not checked again until it turn comes around. It have a lot of downsides and its not perfect in any means, its just a way to always get names for things that are in assets and dont have to worry that api return error for whole call.


Yeaaaaaa the fact that you are having to do things like that is just... /face_palm


So CCP FoxFour, would not it be best to just move Name and XYZ coorodinates into assets API? Just merge those two endpoints and leave 1 hour cache on it. that would be briliant.

Also - btw what XYZ coordinates means exactly? Are those XYZ coords inside the system or inside the grid or what? Are those exactly the same for a ship and SMA that it is stored in? I need to check that, never bothered to use Locations API other than for names

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
qu1ckkkk
The Warp Core Stabilizers
#91 - 2014-04-22 09:57:13 UTC
Max Kolonko wrote:

Also - btw what XYZ coordinates means exactly? Are those XYZ coords inside the system or inside the grid or what? Are those exactly the same for a ship and SMA that it is stored in? I need to check that, never bothered to use Locations API other than for names


Not sure what you mean by "inside the grid" but they are locations in space if populated.

Proud developer of SeAT! A Simple Eve API & Corporation Management Tool.

Project Page: https://github.com/eveseat/seat
Laendra
Universalis Imperium
Goonswarm Federation
#92 - 2014-04-22 13:15:47 UTC
CCP FoxFour: ♥♥♥♥♥ for all the API ♥
CCP FoxFour
C C P
C C P Alliance
#93 - 2014-04-22 13:16:43 UTC
Laendra wrote:
CCP FoxFour: ♥♥♥♥♥ for all the API ♥


:)

You're more than welcome.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#94 - 2014-04-23 09:44:27 UTC
PI code is in and such, will try and get Sisi updated ASAP.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#95 - 2014-04-23 17:52:59 UTC
Right, the PI endpoints are on Sisi right now if you want to try them:



All of the endpoints want:

  • characterID
  • planetID (all but colonies on this one)
  • a key and code with access to the assets list


GO MY MINIONS! TEST AND WORK WITH THIS! Show me what you can do! :D

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#96 - 2014-04-23 17:55:58 UTC
For those that don't have working API keys on Sisi, here is some sample data I just took from it:

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Querns
Science and Trade Institute
Caldari State
#97 - 2014-04-23 18:05:49 UTC
Thanks for your hard work! I will play around with this some tonight, when I get out of work.

This post was crafted by the wormhole expert of the Goonswarm Economic Warfare Cabal, the foremost authority on Eve: Online economics and gameplay.
Laendra
Universalis Imperium
Goonswarm Federation
#98 - 2014-04-23 19:36:18 UTC
Unable to even get api.testeveonline.com to come up...keeps timing out.
CCP FoxFour
C C P
C C P Alliance
#99 - 2014-04-23 19:41:21 UTC
Laendra wrote:
Unable to even get api.testeveonline.com to come up...keeps timing out.


Yea sorry. The ports for that host were closed for some reason. I have a ticket in to get this fixed ASAP, should hopefully be fixed tonight.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#100 - 2014-04-23 20:24:11 UTC
And the issue should now be fixed. api.testeveonline.com should now be accessible from outside CCP. :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.