These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Yapeal PHP API library (2.0)

Author
Dragonaire
Here there be Dragons
#61 - 2014-08-08 15:48:22 UTC
Tiberius Zol - Sorry you lost all the data without backups but there is a reason I called it init. I also clearly state in the instructions to make backups before starting. Something I will try to do going forward which I really have been mean to do is make an update SQL with just the changes. I'll have to come up with a naming system that make sense and is easy to understand as well but it is something that will be needed more and more as time goes on. I just had another idea how to do it as well that might be easier to maintain so give me a couple days and I'll see if it might work the way I think it could.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Tiberius Zol
Moira.
Villore Accords
#62 - 2014-08-08 16:49:06 UTC
nevermind. had a backup from yesterday. was the first shock only. :) i will keep in mind to change all sql creation files until we have a update mechanism. :)

Mr. Tibbers on twitter: @Mr_Tibbers

Mr. Tibbers Blog: www.eve-versum.de

Legedric Striker
Brave Newbies Inc.
Brave Collective
#63 - 2014-08-08 19:31:57 UTC
Well, of course there is always the possibility to just take the SQL files from the most recent Yapeal version, take your favourite version control (like SVN or git) and see for the diffs yourself and just adopt the changes from there. No need to recreate the tables at all ;)

Of course you should know what you are doing Blink

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!

Tiberius Zol
Moira.
Villore Accords
#64 - 2014-08-08 19:53:09 UTC
yeah.. learned about it.. never do things after 30 hours without sleep. ;-)

Mr. Tibbers on twitter: @Mr_Tibbers

Mr. Tibbers Blog: www.eve-versum.de

Legedric Striker
Brave Newbies Inc.
Brave Collective
#65 - 2014-08-08 20:39:54 UTC
Tiberius Zol wrote:
yeah.. learned about it.. never do things after 30 hours without sleep. ;-)


Seems like you are doing that pretty often as I think I may have read that you were saying this a couple of days ago Blink

EVE-Skillplan.net - Plan your pilot skill training online on PC, Mac, tablet or smartphone!

Jack Haydn
Magellanic Itg
Goonswarm Federation
#66 - 2014-08-09 20:57:53 UTC
Dragonaire wrote:
Actual Yapeal does locking on a per API basis which allows multiple instances to run without stepping on each others toes just like it always has for a long time. AbstractCommonEveApi::gotApiLock() handles getting the lock. Using the database for locking is faster plus more cross platform compatible for what it needs. It also using transactions for all deletes and upsert so it makes sure multiple table APIs are updated correctly and keeps everything ACID. So you shouldn't need to do any extra locking in your crontab. You said it would cause weird behavior, have you actually experienced this or just assumed because you didn't see any locking that it would?

I'm a little puzzled by what you mean when you say 'delete APIs' maybe you can explain what you mean just a little more.



Sorry for the long delay, was quite busy.

I have not looked into what kind of locking yapeal does specifically, so I did not notice any row or table level locking. Somer did recommend using a file system based lock file though, since that is what he does aswell. I know little about yapeal, but I guess Somer kind of knows what's going on there. So I'm the wrong one to ask what exactly is happening. Maybe the locking logic is bugged, maybe Somer and I have set something up utterly wrong.

What I do know, is that CCP eventually blocked me and I was getting API data in stretches with lots of holes inbetween. Once I manually went through the database, weeded out a lot of expired/changed/deleted API keys and set my cron interval to 15 minutes, everything is running smooth again and I'm getting full API coverage. This kind of self-maintenance would be nice to have in the API library. Maybe it's even there, but then apparently bugged or I'm dumb.

This brings me to my second suggestion, about deleting APIs: AFAIK yapeal only provides methods to make API keys inactive (setting the isActive flag to zero), but no methods to actually completely remove API keys from the system.
Dragonaire
Here there be Dragons
#67 - 2014-08-09 22:15:48 UTC
Jack Haydn - Sound like your application needs to do a better job of check keys before they get added to utilRegisteredKeys but to why Yapeal doesn't delete 'bad' keys and just deactivates them is randomly the API servers will return error codes suggesting it's a bad key. If it actual did delete them every time the API reported a problem you'd very soon end up with no keys at all.

My suggestion is you have a crontab that check for deactivated keys and use oneShot() calls to APIKeyInfo to check the keys are still valid etc and decided what you want to do directly. Another reason Yapeal doesn't do it automatically is some sites might want to send eve-mail or do something else to let the site admin or user know there is a problem. By simple making it not active the problem can be corrected and reactivate it. I try NOT to decide how things like that have to be handled because that's not Yapeal job that's the application's job Yapeal just tries to keep things from happening like your IP from being banned. Now if you find something in the logs where it's not deactivating a key when it should be do let me know and I'll make sure it gets fixed.

What might be helpful to figure out why you ended up getting banned, which by the way is very hard to do, except if there's something true wrong with your app setup if we could see the log files. There should be no way Yapeal would be breaking any of the API caching rules which would cause this but if somehow it is I would have to see the log files to figure out where the error might be. I would suggest instead of trying to paste all or part of your log file here and fight the forum formatting etc that you put it on Pastebin or something like it and just include a link. If you prefer you can also E-mail it to me at the E-mail listed on GitHub just make sure you ID yourself so it doesn't end up in my junk bin Blink.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Jack Haydn
Magellanic Itg
Goonswarm Federation
#68 - 2014-08-09 23:00:51 UTC
Okay, fair enough, just deactivating them in the automatic process is reasonable. But apparently that failed somehow. Unfortunately, the bad keys are now purged and the logs were not set to debug or similar, so at this point there is not much I can provide for debugging purposes.

What I am asking for regarding the deletion of keys is a method which facilitates deletion. Not automatically triggered, but just a function I can call, pass it a keyid, and it'll remove the key and all relevant entries. It's quite a pain to figure out all the util and account joins needed to remove all the entries associated with it and hopefully not miss one ;)

IMO the library should abstract maintenance of the stuff as much as possible, so that the developer does not need to look into all the details. Otherwise one might just write the thing on their own :)

Another possible idea: In the spirit of above paragraph, a suitable abstraction for api key maintenance would be to provide a function which takes an API keyID and returns the state of the API. It can be good or display the number of times or the time interval it consecutively failed to receive a valid response from the API server. Then the developer can easily call that, and send out email warnings etc. Figuring out the inner workings of yapeal key maintenance is not needed anymore, then, in combination with a library-provided delete method.
Dragonaire
Here there be Dragons
#69 - 2014-08-10 05:41:34 UTC
delete from utilRegisteredKey where isActive = 0;

Now if you want to remove all the data from the API tables that is a little more complex and I can give you some pointers on that but if you are already selecting the data to show it in your application you know how to do it really. Just change your selects into deletes and you're done. All the joins etc will be the same or nearly the same and I'm sure you can figure it out.

Let me quote something from the main page of Yapeal here.
Quote:
Yet Another Php Eve Api Library also know as Yapeal which can be used to access the Eve Online API data and place it into a database.

I think it does that and I'll assume you did too or you wouldn't have used it. The idea of Yapeal is not to abstract everything database away for you but to simple have a auto-magically updating database so you don't have to deal with all the details of the XML. The original idea of Yapeal is the database is the interface that you should only need to deal with it. That has changed in some small ways but in the end I still use that as the guide to what is done and included in Yapeal. It is NOT Yapeal's job to handle everything related to the database because it really can't. Every application or even programmers writing the same application are going to do things differently and what might work great for one type of application or one programmer could bring tears to the eyes of another programmer. The code in some of the older versions of Yapeal and some of my other projects have been know to nearly do this at times actually for me when I look at them now Blink

Think about what Yapeal does do and really how simple it is to work with.

  • You have one small config file that is set and forget.
  • You turn on and off which APIs it does anything with from a single column in one table (utilEveApi) isActive.
  • You only have to add site user API key information to one table (utilRegistereedKeys) and Yapeal takes care of the details afterwards.
  • If you need to stop Yapeal from getting some site user data once again there's only one table you need to work with utilRegisteredKey.
  • Do a one time setup of a crontab or scheduled task on a preferred one minute timer.
  • Yapeal goes to work making sure all the API data you have requested is as up to date as is allowed by the servers.


So in the end you only have two tables in the database your application has to manage so Yapeal can update over 100 tables for all your site users. I think that a pretty good deal.

Now with that said if you look I did start some work on a class to help manage the utilRegisteredKey table which you'll find in bin/ directory. It currently just helps you add a new key but I do plan on it being a more complete management tool for people to use on that table. If you want to add to it and share it with everybody please do as I'm not sure when I might get back to working on it as I have many other things to do in Yapeal and otherwise. I also would like to see something to help manage the utilEveApi table to make turning APIs on and off easier with a more OOP interface etc.

I do want to thank you for the feedback and ideas even if I probably wouldn't change Yapeal in the ways you want and as always thank you for using Yapeal.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#70 - 2014-08-11 20:09:42 UTC
Just a quick note I've added corp/Facilities for those of you working with IndustryJobs. Note that I include an update SQL file in bin/sql/update/. It's mostly an early idea about doing this so will have some changes to it in the future but at least it can be used manually for now with edits until I develop the code to use it more directly.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Kolb
Fancy Fox Enterprises
#71 - 2014-08-12 15:30:34 UTC  |  Edited by: Kolb
I wrote a slightly fancy bash script for my yapeal cron job:

http://pastebin.com/f9KJUBhd

It keeps the cron's logfile around 1 MB is size by trimming out the oldest records.

| It checks if the API server is working if so,
|---- it checks if Yapeal is running if so,
|-------- it keeps the scripts you put in it's body from starting again till yapeal is finished.
|---- if Yapeal isn't running it processes any number of scripts you wish to execute in series with yapeal somewhere in the series.
| if the API server isn't responsive it stops the script

Happy yapealing,

Kolb
Dragonaire
Here there be Dragons
#72 - 2014-08-12 17:43:07 UTC
Kolb - Thanks for sharing just have a few suggestions.


  • First instead of trying to manage the log files yourself look at logrotate it's made to handle that type of stuff and is what most people use.
  • Instead of copying rows from one table to another you could look at replacing utilRegisteredKey with a view or change you app so it uses the keyID as a foreign key in your table. You might find CreateCustomTables.sql helpful here.
  • Why check for the API server when Yapeal already does that? It's a waste of your time writing code to try replacing something Yapeal already handles.
  • Once again THERE IS NO PROBLEM WITH MORE THEN ONE YAPEAL RUNNING AT THE SAME TIME IT WAS MADE TO DO SO. It does locking on a per API and keyID basis and no additional locking is needed or management is needed. If you want to have Yapeal NOT be able to keep up with increasing load continue NOT letting this happen and it'll get slower and slower with more users. Running Yapeal with the one minute crontab is basically a simple way to make it act like it's threaded which can't be done in a cross platform way at this time in PHP or I'd use it. Yapeal spends 99% of it's time waiting on either your network connection to get the XML or your database/ filesystem to store it.


In summary stop re-inventing the wheel.
Stop try to second guessing over 5 years of Eve API development experience that has gone into Yapeal from me and others. Everything you are trying to do I've tried in Yapeal at some point in the past and if it's NOT there now there is a reason for it. The reason is usually I found a better way or it didn't work or even that it simple wasn't needed.

Just a suggestion try it my way. One of three things will happen.

  1. You'll find I'm right and it does work better then what you've been trying.
  2. You won't be able to measure any difference and my way is simpler.
  3. You find something in Yapeal that can be improved because you have tested it both ways and can show that and everyone benefits when I give you credit when adding it to Yapeal.


One final thing don't really take the above personally in anyway I've just been down this road a few times and needed to blow off some steam about it. I'm glad you want to improve things and help others and if you can show me I'm wrong on something that great as well.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Kolb
Fancy Fox Enterprises
#73 - 2014-08-12 18:57:03 UTC
Dragonaire wrote:
Kolb - Thanks for sharing just have a few suggestions.


  • First instead of trying to manage the log files yourself look at logrotate it's made to handle that type of stuff and is what most people use.
  • Instead of copying rows from one table to another you could look at replacing utilRegisteredKey with a view or change you app so it uses the keyID as a foreign key in your table. You might find CreateCustomTables.sql helpful here.
  • Why check for the API server when Yapeal already does that? It's a waste of your time writing code to try replacing something Yapeal already handles.
  • Once again THERE IS NO PROBLEM WITH MORE THEN ONE YAPEAL RUNNING AT THE SAME TIME IT WAS MADE TO DO SO. It does locking on a per API and keyID basis and no additional locking is needed or management is needed. If you want to have Yapeal NOT be able to keep up with increasing load continue NOT letting this happen and it'll get slower and slower with more users. Running Yapeal with the one minute crontab is basically a simple way to make it act like it's threaded which can't be done in a cross platform way at this time in PHP or I'd use it. Yapeal spends 99% of it's time waiting on either your network connection to get the XML or your database/ filesystem to store it.


In summary stop re-inventing the wheel.
Stop try to second guessing over 5 years of Eve API development experience that has gone into Yapeal from me and others. Everything you are trying to do I've tried in Yapeal at some point in the past and if it's NOT there now there is a reason for it. The reason is usually I found a better way or it didn't work or even that it simple wasn't needed.

Just a suggestion try it my way. One of three things will happen.

  1. You'll find I'm right and it does work better then what you've been trying.
  2. You won't be able to measure any difference and my way is simpler.
  3. You find something in Yapeal that can be improved because you have tested it both ways and can show that and everyone benefits when I give you credit when adding it to Yapeal.


One final thing don't really take the above personally in anyway I've just been down this road a few times and needed to blow off some steam about it. I'm glad you want to improve things and help others and if you can show me I'm wrong on something that great as well.




I'm not second guessing you Dragonaire, I wrote the bash for my specific needs in my environment is all. I put some effort into my cron and thought it might be something others could use in their work or even as an educational example.

:-)

I have a reason for not wanting to run my three scripts over themselves and it doesn't involve me worrying about the state of the yapeal tables but wanting yapeal to be finished with it's work before I continue on with the following steps in my process. Again this script isn't just about kicking off yapeal, I wanted to keep all of my tools running during an API outage.

I wasn't judging or commenting about the state or capabilities of yapeal, just building a tool for my needs and sharing my lessons learned with others out there that might not have experience with building a bash script to run as a cron.
Tiberius Zol
Moira.
Villore Accords
#74 - 2014-08-13 19:09:40 UTC  |  Edited by: Tiberius Zol
Is the Posfuel stuff broken? I have two old entries in corpPosfuel but no actual data.

Edit: yeah. Seems that StarbaseDetail is broken. Get another xml file as shown in the cache files.

Mr. Tibbers on twitter: @Mr_Tibbers

Mr. Tibbers Blog: www.eve-versum.de

Dragonaire
Here there be Dragons
#75 - 2014-08-13 23:27:55 UTC
Make sure they aren't for the same POS the cache files are on a per KeyID/characterID basis. So if you have say 2 CEO keys or a CEO and Director key for the same corp you'll have 2 cache files but only one set of data in the DB.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Tiberius Zol
Moira.
Villore Accords
#76 - 2014-08-14 06:10:56 UTC
I have 4 Characterkeys and 2 Corpkeys (1 Director, 1 CEO but different Corps and different Chars)..
I also have no Membertracking for the Corp with Director Key.. on the old yapeal both works.

Mr. Tibbers on twitter: @Mr_Tibbers

Mr. Tibbers Blog: www.eve-versum.de

Dragonaire
Here there be Dragons
#77 - 2014-08-14 07:02:09 UTC
Tiberius Zol - Do all the POSes show in StarbaseList table?

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Tiberius Zol
Moira.
Villore Accords
#78 - 2014-08-14 07:44:21 UTC
yes.

Mr. Tibbers on twitter: @Mr_Tibbers

Mr. Tibbers Blog: www.eve-versum.de

Dragonaire
Here there be Dragons
#79 - 2014-08-14 19:00:40 UTC
Tiberius Zol - So it seems to be working here fine but I only have a single POS to work with. I'll go through some checks you can do to maybe narrow down where something might be breaking.

  1. If you look in cache/corp you should find a couple of StarbaseList*.xml files which list all the POSes you see in the DB table.
  2. You should see one StarbaseDetail*.xml for each POS as well.
  3. Looking at the StarbaseDetail*.xml files they should look something like http://pastebin.com/hVXtxk5Y. Notice if there is a 'fuel' element and make sure there are rows in it.

If all the above is good then you will next need to check log/yapeal.log and look for the messages on the POS stuff. Easy way to find them is do a find/search for 'Corp/Starbase'. Should see where StarbaseDetail calls StarbaseList to make sure it's up to date then processes each POS in turn. Make sure it doesn't report invalid XML when checking with XSD. Next it inserts the info into the 'corpStarbaseDetail', 'corpFuel', 'corpCombatSettings', and 'corpGeneralSettings' tables. Look for any exceptions especially having to do with the inserts. Also note how many rows it says for each table with fuel being the only one that should have more than a single row.

If you do find something in the log if you can put is up on Pastebin or some where I can take a look I'll try figuring out what's causing the issue. If you don't find anything when you look at the log you might still put it up somewhere where I can have a look at it as I might spot something that looks wrong to me still.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Tiberius Zol
Moira.
Villore Accords
#80 - 2014-08-15 06:33:27 UTC  |  Edited by: Tiberius Zol
Checked several cache files now. I have no cachefile for my online pos. there several cache files for my offline poses including one offline pos with some fuel left (this entrys are in my db, so its fine).
I have checked the logfile also.
Found the insert part but it wrotes only the two lines from the offline pos with the fuel left.
here a pastebin from the logpart: http://pastebin.com/SNmeztzA

Mr. Tibbers on twitter: @Mr_Tibbers

Mr. Tibbers Blog: www.eve-versum.de