These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Yapeal PHP API library (2.0)
Author
Previous Topic Next Topic
Dragonaire
Here there be Dragons
#261 - 2015-07-13 22:03:40 UTC  |  Edited by: Dragonaire
https://github.com/Yapeal/yapeal/issues/100 fixed
https://github.com/Yapeal/yapeal/issues/101 fixed

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Legedric Striker
State War Academy
Caldari State
#262 - 2015-07-20 10:53:06 UTC  |  Edited by: Legedric Striker
Hi Dragonaire,

yes, it's me again ;)

As several users keep reporting their SkillQueue won't get updated through the API recently I digged into this by myself as the issue was also reproducable for my own account.

So if I check my SkillQueue API manually using this link https://api.eveonline.com/char/SkillQueue.xml.aspx?characterID=90750214&keyID=xxxxxxxxxxxxx&vCode=YYYYYYYYYYYYYYYYYYYYYYYYYYYYY

I see a skill queue matching the one in game, however, Yapeal isn't pulling it right into the database and the log is telling me about any obvious errors.

However, I check the timers and while the above mentioned direct link is showing this:
-cachedUntil-2015-07-20 11:09:41-/cachedUntil-


The database table "utilCachedUntil" is showing this recordset:
SkillQueue     2015-07-20 21:36:42    90750214     Char


So completely different times having the Yapeal cacheduntil time way longer than it probably should be.

I don't know why or where this is coming from but the times seem to be off somehow :/

When I delete the entry the next run inserted a new recordset with a correct "expires" entry. So it seems like the cachedUntil sometimes saves wrong values into the "expires" field.

I know you are not taking the cachedUntil timers from CCP but instead take teh "currentTime" and add the interval to it. But maybe you should check it against the cachedUntil time and take the lower one of both for cases where CCP's "currentTime" is nonsense?

/Edit:
Now I got even the following for my character here:

CharacterSheet     2015-07-21 11:25:37     90750214     Char
SkillQueue     2015-07-21 11:28:16     90750214     Char


So like 24h cache timer? This cannot be right...
Well, for the time being I built a workaround and delete all cachedUntilTimers having expires more than 1 hours ahead of now for SkillQueue, CharacterSheet and APIKeyInfo...

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!
Mintoko
Taedium In Perpetuam
#263 - 2015-07-20 12:10:35 UTC
Using latest version from 7/14 @ 06:41
Three APIs are broken.

eveCharacterInfo
charContactList
charCharacterSheet



[2015-07-20 06:53:32] yapeal.DEBUG: Element 'ancestry': This element is not expected. Expected is one of ( alliance, allianceDate, allianceID, bloodline ).  [] []
[2015-07-20 06:53:32] yapeal.WARNING: The data retrieved from Eve API eve/CharacterInfo for 957626873 is invalid [] []

[2015-07-20 06:53:38] yapeal.DEBUG: Element 'allianceContactLabels': This element is not expected. Expected is ( allianceContactList ).  [] []
[2015-07-20 06:53:38] yapeal.WARNING: The data retrieved from Eve API char/ContactList for 957626873 is invalid [] []

[2015-07-20 06:53:46] yapeal.DEBUG: Element 'ancestryID': This element is not expected. Expected is ( balance ).  [] []
[2015-07-20 06:53:46] yapeal.WARNING: The data retrieved from Eve API char/CharacterSheet for 957626873 is invalid [] [
Dragonaire
Here there be Dragons
#264 - 2015-07-20 15:09:31 UTC  |  Edited by: Dragonaire
Legedric Striker - Glad you can seem to reproduce it as that makes it much easier to try figuring it out Lol Maybe they have changed the cache time on that API so I'll look into it. BTW the cache interval is stored in utilEveApi table so you can just update it there instead of having another script go through the other table for you. I just looked at it and for SkillQueue it's set to 15 min (900 sec) which should be right. I'm wondering if there's something going on with the currenttime in the XML that is throwing it off instead? Might look at those if you can and see if that what was causing it. I know the API was all kinds of screwed up after this last update but it sounds like you might have had this happening before that as well. Any more info you can find out on it that might help me narrow it down would be great.

Mintoko - Thanks for info I'll try getting the new columns they added to those APIs updated. I'm sorry I didn't ended up having as much time as planned to work on Yapeal before the update so I'd have a few more of them ready to go when the update came out but I'll get to them soon(tm) Smile I expect I'll have the character ones done first since the contact one requires adding another table or tables I think but I'll have better idea once I get a look at the actual XML here on my end.

In other news made some progress on the experimental branch of Yapeal so hope to have it in available in an early alpha test for everyone to try out. I'm still trying to decide if I'll call it 2.1 or 3.0 but lending to 3.0 because some of the public interface is change enough to require it by semantic version standards.

As always thank you for using Yapeal, Dragonaire

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Legedric Striker
State War Academy
Caldari State
#265 - 2015-07-20 17:56:59 UTC
Yes I think the "currentTime" is off at CCP from time to time producing these weird times. That's why I delete all cachedUntil timers from the database more than 1 hour in the future.

Adjustint the interval won't help for the broken currentTime at CCP and that's why I implemented the workaround for me ;)

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!
Dragonaire
Here there be Dragons
#266 - 2015-07-20 18:30:14 UTC
Okay kind what I was thinking if you can find an example where it's doing it might try bug report with copy of XML and maybe they can track down in their logs what's going on. Might just be one of the API server setup wrong or something and it'll help them figure it out as it's probably giving wrong current time but cachedUntil is still right in the XML.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Dragonaire
Here there be Dragons
#267 - 2015-07-20 22:45:36 UTC
Okay think I got everything updated for the changes. Let me know if something still seems to be missing. Note that several tables have been added so make sure you run bin/yc D:U after installing the update so all the DB changes happen.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Legedric Striker
State War Academy
Caldari State
#268 - 2015-07-21 07:17:32 UTC
Hi Dragonaire,

most characters are working again now, but there still seems to be some not-adapted data within the characterSheet API part:


[2015-07-21 07:07:02] yapeal.DEBUG: Given XSL name /www/htdocs/w0123f51/_libext/vendor/yapeal/yapeal/lib/Database/Char/CharacterSheet.xsl [] []
[2015-07-21 07:07:02] yapeal.DEBUG: Started XSD validating [] []
[2015-07-21 07:07:02] yapeal.DEBUG: Element 'row': This element is not expected.  [] []
[2015-07-21 07:07:02] yapeal.WARNING: The data retrieved from Eve API char/CharacterSheet for yyyyyyyyis invalid [] []


I checked the XML manually and I found several "row" entries in there:


[rowset name="implants" key="typeID" columns="typeID,typeName"]
      [row typeID="27258" typeName="Zainou 'Snapshot' Cruise Missiles CM-605" /]
      [row typeID="24637" typeName="Zainou 'Deadeye' Missile Projection MP-705" /]
      [row typeID="24640" typeName="Zainou 'Deadeye' Guided Missile Precision GP-805" /]
      [row typeID="24639" typeName="Zainou 'Deadeye' Target Navigation Prediction TN-905" /]
      [row typeID="24638" typeName="Zainou 'Deadeye' Rapid Launch RL-1005" /]
      [row typeID="20121" typeName="High-grade Crystal Alpha" /]
      [row typeID="20157" typeName="High-grade Crystal Beta" /]
      [row typeID="20158" typeName="High-grade Crystal Gamma" /]
      [row typeID="20159" typeName="High-grade Crystal Delta" /]
      [row typeID="20160" typeName="High-grade Crystal Epsilon" /]
      [row typeID="33329" typeName="Genolution 'Auroral' AU-79" /]
    [/rowset]



[rowset name="skills" key="typeID" columns="typeID,skillpoints,level,published"]
      [row typeID="3394" skillpoints="90510" level="4" published="1" /]
      [row typeID="3442" skillpoints="226275" level="4" published="1" /]
[/rowset]


Furthermore there are also many other "rowsets" in the XMl not containing rows, probably because they are empty for this character but you may wanna expect a "row" element in there, too.


[rowset name="jumpClones" key="jumpCloneID" columns="jumpCloneID,typeID,locationID,cloneName" /]
[rowset name="jumpCloneImplants" key="jumpCloneID" columns="jumpCloneID,typeID,typeName" /]
[rowset name="certificates" key="certificateID" columns="certificateID" /]
[rowset name="corporationRoles" key="roleID" columns="roleID,roleName" /]
[rowset name="corporationRolesAtHQ" key="roleID" columns="roleID,roleName" /]
[rowset name="corporationRolesAtBase" key="roleID" columns="roleID,roleName" /]
[rowset name="corporationRolesAtOther" key="roleID" columns="roleID,roleName" /]
[rowset name="corporationTitles" key="titleID" columns="titleID,titleName" /]


I hope this helps ;)

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!
Dragonaire
Here there be Dragons
#269 - 2015-07-21 15:46:07 UTC
Actual most of CharacterSheet is a collection of rowsets so that all looks fine it is just that that char doesn't have any corp roles etc which isn't that unusual. The unexpected row error does suggest something is off. Make sure you got the latest update I pushed out a couple of them yesterday. Also worth delete the XMLs in cache/ just to make sure it's not a bad XML that's messing stuff up from when the API was screwy. If after do that you still have the same error for the same ones if you could email me the part of your log just before and after the error (about 30 lines each side) and the Invalid XML file I'll take a look and see what's going on.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Ramov Tinoga
Tinoga Enterprises
#270 - 2015-07-29 11:50:54 UTC
Thanks for the continued development, Dragonaire!

Today I installed the latest master from git and everything went fine except for one small issue. I was getting a lot of DEBUG and INFO output, so I searched for the WARNINGS that caused the verbose output and I found this:

[2015-07-29 13:21:00] yapeal.WARNING: Failed to upsert data from Eve API char/ContactList for 722849004 {"exception":"[o
bject] (PDOException(code: 42S02): SQLSTATE[42S02]: Base table or view not found: 1146 Table 'yapeal2.charAllianceContac
tLabels' doesn't exist at /home/scripts/yapeal-git/lib/Database/Char/ContactList.php:103)"} []


Grepping for charAllianceContactLabels, I found 
lib/Sql/updates/201507202210.sql
, which should have created the tables on 
bin/yc D:I [...]
, but they were indeed not there.

As 
bin/yc D:U
claimed

Skipping SQL file 201507202210.sql since its <= the latest database version 201507202210

I manually reset the entry in utilDatabaseVersion to 201507132213 and ran the update again, which created the missing tables. On the next yapeal run, the WARNING (and verbose logging) went away and the tables were populated accordingly.

Now, I have two entries in utilDatabaseVersion (201507132213 and 201507202210).
Does that mean that other updates were also possibly missed and I have to force them similar to described above, or will I be fine, now that no more WARNINGS occur?
Dragonaire
Here there be Dragons
#271 - 2015-07-29 15:55:44 UTC
Ramov Tinoga - Sounds like maybe I didn't get one of the tables added to the CreateCharTables.sql file that is in the update. I'll take a look and make sure I correct it. The way it is suppose to work is the update files act kind of like a diff between the DB version that you have and what the current Create*Tables.sql create on a new install would be like but seems like in this case the update is ahead of the create Blink. You shouldn't have to do anything else if you aren't seeing any more warnings in the logs and I'll push the updated create out either today or tomorrow as I get time.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Dragonaire
Here there be Dragons
#272 - 2015-07-29 16:56:28 UTC
https://github.com/Yapeal/yapeal/issues/102

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Sona Arnst
Regnum demens Deus
#273 - 2015-09-15 11:51:33 UTC
Getting the bellow when running "php bin/yc D:U" and I'm no longer getting any data imported into the tables. The XML pulls fine.

DatabaseUpdater::addDatabaseProcedure

Skipping SQL file 201408281814.sql since its <= the latest database version 201507202210
Skipping SQL file 201409272333.sql since its <= the latest database version 201507202210
Skipping SQL file 201410152135.sql since its <= the latest database version 201507202210
Skipping SQL file 201411031600.sql since its <= the latest database version 201507202210
Skipping SQL file 201412130159.sql since its <= the latest database version 201507202210
Skipping SQL file 201502120216.sql since its <= the latest database version 201507202210
Skipping SQL file 201503120437.sql since its <= the latest database version 201507202210
Skipping SQL file 201505150647.sql since its <= the latest database version 201507202210
Skipping SQL file 201506131923.sql since its <= the latest database version 201507202210
Skipping SQL file 201507022249.sql since its <= the latest database version 201507202210
Skipping SQL file 201507130432.sql since its <= the latest database version 201507202210
Skipping SQL file 201507131620.sql since its <= the latest database version 201507202210
Skipping SQL file 201507132213.sql since its <= the latest database version 201507202210
Skipping SQL file 201507202210.sql since its <= the latest database version 201507202210

DatabaseUpdater::dropDatabaseProcedure



I have tried deleting all .sql files apart from 201507202210.sql but I still get the errors.
Dragonaire
Here there be Dragons
#274 - 2015-09-18 17:59:01 UTC  |  Edited by: Dragonaire
Sona Arnst - Sorry for the late reply been busy and not checking the forums for last few days. All of the skipping messages is normal for an update it just means you already have a newer 'version' of the DB tables already. The first time you run the update when there has been a DB change it will do the one or two updates you need but skip any you have already done or for other reasons is unneeded (for example during new installs). All the update SQL file names are made up of the date and time they where created by me, so for example 201507132213 = 2015-07-13 at 10:13 pm GMT but your DB tables are already at 2015-07-20 at 10:10 pm GMT which if you look is the last one listed in the output.

If you do a fresh install and use 'php bin/yc D:I' it will also know it's at the current version. Yapeal tracks which 'version' of the DB tables is on using the utilDatabaseVersion table. Depending on how many times you done an update vs a new install you'll see one or more records in the table and the update always check for the highest one that it finds and adds a record for each update that it completes fully.

Hopefully my explanation help you better understand what's going on.

Also after re-reading your post I decided I should make clear that the 'php bin/yc D:U' is used to update the DB table structure (column name, type, add new ones ) but is NOT used to update your user data in the tables. You use bin/yapeal for that.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Koron Eldoran
Center for Advanced Studies
Gallente Federation
#275 - 2015-09-30 12:05:39 UTC  |  Edited by: Koron Eldoran
How can i delete all date from a specific account/corp/alliance?
Dragonaire
Here there be Dragons
#276 - 2015-10-01 18:08:41 UTC
Depends on the table but most are going to have corresponding ID columns or in the case of most of the char/corp/account tables you have the ownerID column which is based on those IDs so a simple 'delete * from yapeal.theTable where ownerID = id ' should remove anything with that ownerID

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Legedric Striker
State War Academy
Caldari State
#277 - 2015-12-11 07:16:59 UTC
Hi Dragonaire,

it's been a while and I don't know if you are still active but I seem to ran into some trouble with expired API keys lately.

In my project database several API keys seem to have expired and unfortunately Yapeal does not recognize it.

The problem is, that EVE's API only responds with error code 222 "Key has expired. Contact key owner for access renewal." even when you try to get the APIKeyInfo for a key that expired (or got deleted?) so Yapeal is not able to get the new expired date and so thinks the key is still active leading to an error everytime Yapeal tries to fetch something using that spefici key.

So my project has got well above 7k API keys in it and you can image that many people just remove their API key from their account, let it expire or whatever and all these keys lead to the following line in Yapeal's log:

[2015-12-10 23:38:02] yapeal.DEBUG: Could NOT get XML data {"exception":"[object] (Guzzle\\Http\\Exception\\ClientErrorResponseException(code: 0): Client error response\n[status code] 403\n[reason phrase] Forbidden\n[url] https://api.eveonline.com/account/APIKeyInfo.xml.aspx at /www/htdocs/w0123f51/_libext/vendor/guzzle/guzzle/src/Guzzle/Http/Exception/BadResponseException.php:43)"} []

I don't know why it is handled as "Forbidden" but that's what I see in my logs.
Of course due to Yapeal's amazing speed to work on API calls I produce many many errors crawling all API keys of my project for different APIs like APIKeyInfo, CharacterSheet or SkillQueue and this leads to my IP getting blocked due to producing too many errors in a short amount of time everytime my cronjob runs the auto_magic call.

Do you see any chance on deactivating API keys once they ran into this type of error to protect peopl using your API library from getting banned due to producing too many errors?

I will have a look at Yapeal's code to see if I can do it on my own but I would feel more comfortable if you, as the original developer would implement a solution for this Blink

Regards,

Legedric

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!
Legedric Striker
State War Academy
Caldari State
#278 - 2015-12-11 08:14:48 UTC
While I was working on a workaround I found other errors leading to the same IP ban when too many of them happen.

For example when I try to fetch a skill queue for a character I also get "Forbidden" messages every now and then.

One key I checked specifically threw this error in EVE's API:
201: Character does not belong to account.

So for whatever reason those error messages from EVE's API do not even get through to Yapeal but result in a 403 Forbidden instead. I only see these detailed error moessages when I use my browser to get the xml from EVE's API directly.

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!
Mintoko
Taedium In Perpetuam
#279 - 2015-12-11 13:12:33 UTC
'utilRegisteredKey' has a 'active' column and I thought that 'accountCharacters' at one time had one as well. A character on one of my accounts no longer exists and as long as the character is listed, attempts are made to pull from the API. I'd like to keep this character listed as it references other tables. Can an active column be added to 'accountCharacters' or is there something else I could do?
Dragonaire
Here there be Dragons
#280 - 2015-12-11 22:02:43 UTC
Mintoko wrote:
'utilRegisteredKey' has a 'active' column and I thought that 'accountCharacters' at one time had one as well. A character on one of my accounts no longer exists and as long as the character is listed, attempts are made to pull from the API. I'd like to keep this character listed as it references other tables. Can an active column be added to 'accountCharacters' or is there something else I could do?

'accountCharacters' didn't have an active flag but before Yapeal had 'utilRegisteredChar' and 'Corp' tables and they had active flags that could be used but with the APIs moving to 'keyID' and 'vCode' those tables needed to be dropped since everything is based on the 'keyID' now and nothing else in the APIs.

You can do what you want to do you just need to find the row in 'accountKeyBridge' that is linking the keyID to the charID and remove it. The APIKeyInfo class that manages that tables does NOT delete any old rows so references don't break but you can do so as long as you insure everything you use refs to accountCharacters table only. Note that deleting rows in accountKeyBridge only works if the charID is no longer associated with the keyID which seems to be your case. If you look at the getActiveRegisteredCharacters() method in Sql/CommonSqlQueries class you'll see how Yapeal decides which char APIs it checks. getActiveRegisteredCorporations() methods does the same for corp APIs and they both use the accountKeyBridge table to select the list of IDs so deleting the correct row there in effect makes Yapeal ignore the old char/corp you want to keep around in the other tables.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.