These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Yapeal PHP API library (2.0)

Author
Dragonaire
Here there be Dragons
#241 - 2015-06-05 17:16:21 UTC
Any docs that say the mask is optional is for the old 1.x version. 'activeAPIMask' in utilRegisteredKey is required to receive any API data for any API key now and has been since at least version 2.0.x.

Riley Masterson is correct removing the key record from utilRegisteredKey does not remove any of the of the data from the other tables if you need to do that you'll have to do it in your own code.

Remember also for any API the API key you are using must have the needed access active which means to get privileged info from some APIs like 'corpMemberTracking' it need to be from CEO or a Director since they are the only one with that level access. Yapeal logical ANDs together the 'accessMask' bitmap in accountAPIKeyInfo that comes from CCP with what you set in 'activeAPIMask' from utilRegisteredKey to decided which APIs are active from a given API key. This allows you to decrease which API info to get but not try to increase beyond what the API key allows and cause extra pointless errors from the API server.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Legedric Striker
Brave Newbies Inc.
Brave Collective
#242 - 2015-06-09 08:56:02 UTC  |  Edited by: Legedric Striker
So I updated to the most recent version using composer (had a version from mid December 2014 running) but I think I am somehow in trouble now... as I am getting this:

Quote:
[2015-06-09 09:11:39] yapeal.ALERT: Eve Error (904): Received from API account/AccountStatus - Your IP address has been temporarily blocked because it is causing too many errors. See the cacheUntil timestamp for when it will be opened again. IPs that continually cause a lot of errors in the API will be permanently banned, please take measures to minimize problematic API calls from your application. [] []


My project is containing about 4,500 API keys having 6,500 characters tied to them.

However, the yapeal.log is showing this error quite often:

Quote:
Could NOT get XML data {"exception":"[object] (Guzzle\\Http\\Exception\\ClientErrorResponseException(code: 0): Client error response\r\n[status code] 403\r\n[reason phrase] Forbidden\r\n[url] https://api.eveonline.com/eve/CharacterInfo.xml.aspx at \\vendor\\guzzle\\guzzle\\src\\Guzzle\\Http\\Exception\\BadResponseException.php:43)"} []


and

Quote:
Could NOT get XML data {"exception":"[object] (Guzzle\\Http\\Exception\\ClientErrorResponseException(code: 0): Client error response\r\n[status code] 403\r\n[reason phrase] Forbidden\r\n[url] https://api.eveonline.com/account/AccountStatus.xml.aspx


So I think some people deleted their API keys for my project and my project is still trying to pull data using theses keys. Having over 4,500 keys in it there may of course be many keys that got invalid during the past months and so my application is causing multiple errors that may get my project blocked from API calls.

Am I right thinking that I need to develop another API call by myself that checks for forbidden API keys and removes them from the "utilregisteredkey" table?

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!

Dragonaire
Here there be Dragons
#243 - 2015-06-10 16:37:42 UTC
Legedric Striker - So probably was happening with the older version as well it just didn't do as good a job of logging what was going on. Since the event system that was planned hasn't been added to Yapeal yet you'll have to do a couple things to make it so Yapeal can handle this automatically for you. The easiest thing to do is extend from lib/Xml/GuzzleNetworkRetriever and add code to the catch RequestException code to inspect the returned error and handle the 403 errors. Probably not good idea to add all the code there but in another protected method but that the point where you can first intercept the normal process of just report the exception. The other thing you'll need to do is have Yapeal use your version of GuzzlwNetworkRetreiver instead of the default one. For that you just need to set 'Yapeal.Xml.Retriever' in the ContainerInterface $dic you pass to Yapeal class when it is called. The default is set in lib/Configuration/Wiring::wireRetriever() and you should be able to basically copy the code there and just change the class name in the new statement for retriever. Yapeal because of the if statement at the start of the method will not try overriding what you have set for 'Yapeal.Xml.Retriever'. If you need some additional help with this just Email and I'll help out as I can.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Legedric Striker
Brave Newbies Inc.
Brave Collective
#244 - 2015-06-10 17:39:30 UTC
Thanks for the tips Dragonaire!

I already started to just deactivate all API keys for accounts that didn't login to my projekt for more than 90 days and pretty much like 30% of the API keys got deactivated that way.
Furthermore I deactivated some other APIs I somehow missed set isActive to 0 like "YapealCorporationSheet" etc. which I just don't need.

I also contacted CCP Foxfour as there seems to be a general issue occuring at their side of the API blocking API callers too fast and I asked if the "Forbidden" is really what I thought or something else before I start adjusting the GuzzlwNetworkRetreiver class.

In addition I also added something like a run-tracker to my cronjob so I do not start Yapeal's auto-magic when the last run is still running for example etc.
Like just a couple of minutes, after I applied all the above mentioned updates to my project Yapeal seems to be running like I know it, taking just less than 2 minutes to complete all necessary tasks for all ~3,000 remaining API keys Big smile

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!

Dragonaire
Here there be Dragons
#245 - 2015-06-10 18:24:35 UTC
I also just got to thinking about something and I don't think Yapeal checks the expires column from accountAPIKeyInfo table either before making the other API calls so that might be something that could be implemented as well. I'll look into it. That might help with at least some of them. Might also try just run query on DB to look for those yourself as well and it might help find some others that you might have missed.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#246 - 2015-06-10 19:46:26 UTC
Legedric Striker - https://github.com/Yapeal/yapeal/issues/97 I've push a possible fix for this if you could give it a try and see if it clears up any remain API errors and let me know that would be helpful Smile

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Legedric Striker
Brave Newbies Inc.
Brave Collective
#247 - 2015-06-11 07:24:23 UTC  |  Edited by: Legedric Striker
I pulled an update but it's hard for me to tell right now if it resolved many errors as I've said yesterday, Yapeal is running like a charm since I removed many inactive API keys Blink

I am still getting some "Forbidden" errors while running the APIKeyInfo update but it seems to be less than yesterday.

On a side note, I added another log entry to yapeal/lib/Yapeal.php which you may apply to your code as well.

I changed line 135 to:

$logger->info('Magic completed!');
return 0;


The reason is, I got some execution timeout exceptions due to my hosting provider applying a new limit of 30 seconds to his global script timeouts.
This way I know from looking at the Yapeal.log if the API update got completed or somehow ran into trouble and crashed.

I know it is not really necessary as the error.log is containing teh actual exception but I added it anyway Blink

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!

Dragonaire
Here there be Dragons
#248 - 2015-06-11 19:01:36 UTC
You shouldn't be having timeout problems except if you are using the webserver SAPI version of PHP as the CLI SAPI ignores the timeout setting from php.ini. Normally the CLI version is at /usr/bin/php but usually if you just call bin/yapeal.php directly from your crontab it'll find it using env in the first line just like most scripts find bash or python etc.

I've also been thinking about adding back in the soft timeout timer like Yapeal had before. It got removed during the refactor for version 2.x but could be useful to have something like it again. Would prefer to make it configurable in yapeal.yaml if possible as well so if I get a chance I'll look at how I can add it back in.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#249 - 2015-06-14 20:29:04 UTC
Hi all, so decided to rename the 'isActive' column in several of the util* tables to just 'active'. The old name was really a hold over from version 1.x and I'd been thinking about change it for a while. This will probably be most noticeable in utilRegisteredKey to most people. I haven't update any of the docs yet to reflect the change but will be doing that soon(tm)Blink.

Good news is I also have done some work on bin/UtilRegisteredKey class to make it more useful. It now acts as a more complete model for the table and can both read exist rows and do upserts. I also reduced the dependencies to just PDO which should make it easier to integrate into you own projects. The changes did break backwards compatibility but I think the benefit of the new abilities will out weight the pain. I don't expect to need to make any other changes to it now and it now is the recommend way to make changes to the table going forward.

I'll be looking at adding a class to help with interact with utilEveApi as well for those of you that turn APIs off etc. Let me know if there are any of the other util tables you like to see classes done for. Most of them actually are not used right now but I've left them in case the functionally gets implemented in the future.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Mintoko
Taedium In Perpetuam
#250 - 2015-06-22 02:38:04 UTC  |  Edited by: Mintoko
I just deleted a 1.68GB log file. Threshold settings are at the recommended defaults, but it's still logging debug information and growing about 4.25MB per hour.

What settings do I need to stop this?

Using latest code from 6/15/15

My current settings are:

yapeal.error.threshold = 400
yapeal.log.threshold = 300


Also, is there no longer a setting to use utilXmlCache?
Dragonaire
Here there be Dragons
#251 - 2015-06-22 04:10:42 UTC
Mintoko - Look for something that's above the threshold that is triggering the logging. It only takes one WARNING to turn on logging for the rest of that run for all levels of logging.

Currently the only cache is filesystem as I never wrote the DB one for version 2.x. When I asked around no one seemed to be using it so with the limited developer time I have it lost out to other things. I've left in the tables for it so it can be added again by someone in the future. They would probably need some updates to be more inline with things but shouldn't be a lot. If you wanted to add it I'd be more than happy to merge it. Also would have a little time to answer questions via Email to help out.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Mintoko
Taedium In Perpetuam
#252 - 2015-06-22 04:20:51 UTC  |  Edited by: Mintoko
All that's listed in the log file is debug and info.

Actually...I found some warnings, although they're not the first thing listed. It's complaining that the api mailbody data is invalid.

http://pastebin.com/raw.php?i=a53GJhuv
Dragonaire
Here there be Dragons
#253 - 2015-06-22 19:02:29 UTC  |  Edited by: Dragonaire
y default it print the proceeding 20 error messages when triggered and all following. I think I might know what is going on if you can try something for me we'll see if it'll fix it. In Database/common.xsd edit line 38 from

[xs:pattern value="[0-9]+(,\s?[0-9]+)*"/]


and change it to

[xs:pattern value="\s*[0-9]+(,\s?[0-9]+)*"/]

Replacing the starting '[' and ending ']' with '<' and '>' since CCP still can't get code tags to work right [;P]
Just looking at the output from the reported error it I think maybe that the XML has some extra spaces in front of the list and the above change should allow for that. If that doesn't seem to fix it then if you can post to PasteBin or Email me one of the actual invalidMessageBodyXXX.xml file I'll look at it and see if I can figure out why it's failing. Either way let me know and I'll look at push out a fix to everyone.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Mintoko
Taedium In Perpetuam
#254 - 2015-06-23 11:02:04 UTC
The new string seems to work as it's no longer coming up as invalid and no errors are being thrown in my face.

Thanks
Dragonaire
Here there be Dragons
#255 - 2015-06-23 21:50:41 UTC
Ok leave it to CCP to mess up formatting P I'll try pushing it out today for everyone.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Mintoko
Taedium In Perpetuam
#256 - 2015-06-24 20:05:35 UTC
Got another one for ya. I think I know what's going on with this one, but don't know where to look. My guess is that you're currently looking for currentTime > apiTime instead of currentTime >= apiTime


[2015-06-24 03:07:01] yapeal.DEBUG: Started filesystem retrieve for account/APIKeyInfo [] []
[2015-06-24 03:07:01] yapeal.WARNING: CachedUntil is invalid was given 2015-06-24 08:01:02 and currentTime is 2015-06-24 08:01:02 [] []
Dragonaire
Here there be Dragons
#257 - 2015-06-25 18:14:05 UTC
Mintoko - Actually to get that error requires that the XML from the API server is invalid. The current date-time in the XML is less than or equal to the cachedUntil date-time they gave which means they served stall data for you to use. If you look starting at line 255 in lib/Xml/FileCacheRetriever.php you see I'm comparing the times directly from the XML itself. I added that check because they had several APIs doing this and was trying to help them find the ones that did. If you'd file a bug report and go ahead and ref this post it would be great. Most of the APIs have been fixed now but looks like APIKeyInfo still has a problem. Hopefully CCP FoxFour will get a chance to look for it and fix it for us.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#258 - 2015-06-25 22:25:11 UTC
https://github.com/Yapeal/yapeal/issues/98
Pushed fixed to master.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#259 - 2015-07-02 23:41:12 UTC
Hi all,
So after reading about the eve/CharacterAffiliation API in this blog post: https://developers.eveonline.com/blog/article/characteraffiliations-and-the-image-server I decided to go ahead and implement it. Do to the nature of the API and the expected use it is not run as part of the Yapeal::autoMagic() process but is available for anyone to use directly with oneShot() and to give everyone an example of how to do something like that I'm put the short bit of test code I wrote up on PasteBin for you all to have a look at.
http://pastebin.com/hT3uc47f
Its NOT pretty nor using the best coding practices but people often seem to have questions on how to use parts of Yapeal so thought seeing some working code might be useful.

Anyway hope someone finds the new API useful and as always thank you for using Yapeal, Dragonaire

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#260 - 2015-07-13 05:39:24 UTC
So looks like we have another new developer working on the Eve APIs again so some things are changing more quickly again I'll try to get to everything ASAP but if you notice any new errors popping up in your logs let me know so I can take a look at them.

https://github.com/Yapeal/yapeal/issues/99 fixed.

In other news I'm been working on a new developer related command I'll be adding to 'yc'. It automates/helps with all the things that are needed to add Eve APIs that have a simple rowset type structure to Yapeal. They are the most common API style so should prove very useful to me or anyone else wishing to add/update an API as CCP makes changes. I'll write more about it when I feel it's more useable and out of alpha development. I only start creating it a couple days ago but it's actually coming together nicely and I in part used it for the fix above.

As always thank you for using Yapeal, Dragonaire

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.