These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Yapeal PHP API library (2.0)

Author
Dragonaire
Here there be Dragons
#221 - 2015-02-07 02:23:22 UTC
Hi everyone so just realized it had been a while since I'd posted anything here so decided I better update everyone on some things I've been do with Yapeal. First Yapeal has moved again but not very far Blink After experimenting with Github organizations I decided to go ahead and move Yapeal into its own. What does this mean for you you ask? Not much Github should auto redirect you to the new URL but you might make sure any browser bookmarks etc you have saved get updated as I think the forwarding end at some point.

Install - If you use composer you don't have to do anything I've already update Packagist so everything should just work Smile If you use git directly you'll either need to update the remote repo info in Git or another option is to simply delete your current git directory for Yapeal and recreate it using the new info. Make sure you back up any config/yapeal.yaml and bin/sql/CreateCustomTables.sql files you might have there etc before you delete.

Updates - While I was at it I have done some maintenance type things on Yapeal based in part on stuff I've been work with in the experimental branch. First I've decided to go as fulled to PSR-1,2 standards as I can with Yapeal. As I've noted before version 2.0+ has generally been to PSR1 standards but now that more of the developer tools have started understand the PSR-2 stuff and I've got more comfortable with the way code should look when using it and I've also had a chance to use it on some of my other projects I decided it's time for Yapeal to change over as well. Note it doesn't really change how it runs but if you look at the code you may notice some difference in spacing and how the code wrap etc.

Something that may be more noticeable is I've converted all the files in Yapeal to use Linux (Unix) style line ends including the README.md etc. I the past to make them easier to access for people on Windows platforms I've kept them with Windows line endings but now that there are some many free editors out there that understand both I've decided to switch and make my life easier. Only people this might effect is anyone trying to use notepad etc on Windows trying to view them will see just one long line. I'm guessing anyone that going to be using Yapeal already has an editor that can handle this plus all of them can be view directly on GitHub as well.

Another new 'tool' I've been starting to use is SensioLabs Insight. I kind of have a love/hate thing going on with it mostly because it thinks Yapeal is really badly written Sad It isn't but there are some things I have done that they don't think much of mostly in the code used by 'yc'. Like many PHP tools they assume the only thing people use PHP for is in webpages. I've submitted some bug reports etc to them to look at making it better in a few areas for console type apps as well. The main reason I decided to say something about it is I'll probably being doing some commits over the next few days to fix stuff it reports but I also thought I bring it to everyone's attention since I'm sure many of your apps are web based and it can be a neat tool to help you improve things.

You may notice I've delete the old PhpUnit test that were in tests/. They have been failing for a while because of some changes that were done to Yapeal a while back. I've basically been having to ignore travis on each commit complaining about them. I originally was going to fix them so left it going but I've decided that fix the existing tests and trying toaddd all the others that I should write for the over 100 classes in Yapeal was just to much. Don't get me wrong I think tests are great and I'm not happy about not having them but if you've even wrote PhpUnit tests you know they are the easiest thing to work with. PhpUnit is a great testing tool but for something like Yapeal I just have come to feel it's not a good match with all the DB and network code that Yapeal uses plus there are some other things reason as well that I'm not going into here. What I am looking at is switching to something like PhpSpec instead for at least any new code I add. I may also do a few specs and examples for the more critical stuff in the existing code as well if I rewrite it. Just a quick word on why I think PhpSpec might be a better match is it talks to the behavior of the code which is what is more important than the details of how the code goes about doing it. Both PhpUnit and PhpSpec let you test for the behavior but PS just makes it easier and it doesn't bog you down in a lot of setup details or let you start trying to check the internal state of class like I as least always seen to end up doing when using PU trying to make sure everything is good.

Okay I think that enough for one post and if you manage to read the whole thing your gate camp must be really boring and slow or the mining is in Eve for you.Big smile

I'll try not to go so long between post so they don't get as long hopefully.

As always thank you for using Yapeal, Dragonaire.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Tiberius Zol
Moira.
#222 - 2015-02-10 09:27:02 UTC  |  Edited by: Tiberius Zol
Hey man,

first of all, i want to thank you, that you still working on this.
I tried to update yapeal yesterday and run in several problems:

First. The Procedure creates an error for me:

DatabaseUpdater::addDatabaseProcedure
.CREATE PROCEDURE "yapeal"."AddOrModifyColumn"(
    IN param_database_name  VARCHAR(100) CHARACTER SET utf8 COLLATE utf8_unicode_ci,
    IN param_table_name     VARCHAR(100) CHARACTER SET utf8 COLLATE utf8_unicode_ci,
    IN param_column_name    VARCHAR(100) CHARACTER SET utf8 COLLATE utf8_unicode_ci,
    IN param_column_details VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci)
    BEGIN
        IF NOT EXISTS(SELECT NULL
                      FROM
                          "information_schema"."COLUMNS"
                      WHERE
                          "COLUMN_NAME" COLLATE utf8_unicode_ci = param_column_name AND
                          "TABLE_NAME" COLLATE utf8_unicode_ci = param_table_name AND
                          "table_schema" COLLATE utf8_unicode_ci = param_database_name)
        THEN
/* Create the full statement to execute */
            SET @StatementToExecute = concat('ALTER TABLE "',
                                             param_database_name, '"."',
                                             param_table_name,
                                             '" ADD COLUMN "',
                                             param_column_name, '" ',
                                             param_column_details) ;
/* Prepare and execute the statement that was built */
            PREPARE DynamicStatement FROM @StatementToExecute;
            EXECUTE DynamicStatement;
/* Cleanup the prepared statement */
            DEALLOCATE PREPARE DynamicStatement;
        ELSE
/* Create the full statement to execute */
            SET @StatementToExecute = concat('ALTER TABLE "',
                                             param_database_name, '"."',
                                             param_table_name,
                                             '" MODIFY COLUMN "',
                                             param_column_name, '" ',
                                             param_column_details) ;
/* Prepare and execute the statement that was built */
            PREPARE DynamicStatement FROM @StatementToExecute;
            EXECUTE DynamicStatement;
/* Cleanup the prepared statement */
            DEALLOCATE PREPARE DynamicStatement;
        END IF;
    END
Sql failed in DatabaseUpdater::addDatabaseProcedure on statement 1 with (42000) 
SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right syntax to use
near 'COLLATE utf8_unicode_ci,
    IN param_table_name     VARCHAR(100) CHARACTER SET ' at line 2

But in my understanding it was only for the three DropColumn statements.
Everything after it works (i have run it manually after it).

Second:
All my running CorpIndustryJobs have 1970-01-01 00:00:01 as completedDate and endDate.

Third:
I found several Problems in the log:
example:
[2015-02-10 10:08:29] yapeal.DEBUG: Started network retrieve for account/AccountStatus [] []
[2015-02-10 10:08:29] yapeal.INFO: Could NOT find /eve/yapealneu/lib/Database/Account/AccountStatus.xsl [] []
[2015-02-10 10:08:29] yapeal.DEBUG: Given XSL name /eve/yapealneu/lib/Database/common.xsl [] []
[2015-02-10 10:08:29] yapeal.DEBUG: Started XSD validating [] []
[2015-02-10 10:08:29] yapeal.DEBUG: Element 'Offers': This element is not expected.  [] []
[2015-02-10 10:08:29] yapeal.WARNING: The data retrieved from Eve API account/AccountStatus for 3451731 is invalid [] []


Same for corpStarbaseDetail-

The most important thing for me is the industryJobs thing, as my industry tool doesn't work anymore. Any clue how to fix it?

Mr. Tibbers on twitter: @Mr_Tibbers

Mr. Tibbers Blog: www.eve-versum.de

Dragonaire
Here there be Dragons
#223 - 2015-02-11 02:47:26 UTC
Tiberius Zol - I think there is currently so kind of split going on between newer vs older versions of MariaDB and maybe also it and MySQL which is effecting stuff. Could you try running this
show variables like "version"
and let me know what you get back that might help me track it down more. I'm currently running '10.0.14-MariaDB' and in the process of updating to 10.0.16.

On the failing AccountStatus could you maybe send a copy of the XML as I don't have any example that have the offer part to use to update the XSD. Dummy IDs and names etc are fine just need to be able to see the structure. Just send it to my e-mail on GitHub is probably the easiest thing to do.

On the IndustryJobs stuff not sure why you might be having problems but you probably need to look for it in your log files and post that for me to have an idea. It sounds like you're getting the default values from the DB so somehow they are getting there from the XML might turn the file cache on and look at the actual XML and see if something looks off there as well.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#224 - 2015-02-11 17:24:34 UTC
Okay sorry guys looks like in that last push I made a '!=' into a '===' by mistake instead of a '!==' which broke the attribute preserver stuff Oops I just pushed the fix for it. Tiberius Zol this should fix what you noticed with the IndustryJobs and it was effecting other things to but just hadn't noticed it apparently Blink

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#225 - 2015-02-11 20:59:30 UTC  |  Edited by: Dragonaire
So on a better note that the last post I've added something that some of you will probably find useful if you deploy Yapeal in you app in some way. I the past you could have a bin/sql/CreateCustomTables.sql file to do additional DB setup like adding additional tables etc. The main problem with it is if you use composer to install Yapeal there was no easy way to put you file there so Yapeal would see it. I have now made it so Yapeal will also look in the config/ and if installed from composer into vendor/ it'll additional look for it in vendor/../config/ so it can be with any other per install config you do. So you can have it in the same directory with your yapeal.yaml file which should make it much easier to integrate with your app.

PS: Forgot to say it will allow you to have a CreateCustomTables.sql in all three places at once they don't replace the one another just in case you find that useful.

Something I'm probably going to do in the near future (next couple days possibly) is start move some stuff around to what I feel are better places for them. One of the first will probably be moving all the stuff currently in bin/sql over to lib/Sql/ instead. At the time I put them in bin it was simply because I need them somewhere else different than where they were in the old 1.x version of Yapeal and couldn't really think of a place I like at the time. Of course with time what was just a few basically static files used to do one time init of the DB and tables has grow into a full init and update system with a lot more files etc. There also some other Sql related stuff laying around other places in Yapeal that could stand a place of their own so I decided it was time to just go with the lib/Sql directory and that would also let me centralize some of the other stuff into a Yapeal/Sql namespace as well.

There are a couple other thing that I'd like to move and that the *.xsd and *xsl file that now are with the *php under lib/Database/*/ directories. Have them there is some what easier to work with when design things but gets very messy with the char and corp section because of the number of APIs. So look for them to be moved as well in the future.

None of this changes should really effect how you use Yapeal or anything, they will just mean you'll need to look for things in some different places as will I Blink.

As always thank you for using Yapeal, Dragonaire

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Tiberius Zol
Moira.
#226 - 2015-02-14 11:10:46 UTC
As a short info. It's working again. Thx man. :)

Mr. Tibbers on twitter: @Mr_Tibbers

Mr. Tibbers Blog: www.eve-versum.de

Jahn Ntago
Ntago Family Enterprises
#227 - 2015-02-15 09:48:37 UTC  |  Edited by: Jahn Ntago
Hello,

I seem to be having a new problem since 2.0.0 and remaining with the latest builds:

toons that leave a corporation remain in the corpMemberTracking table associated with that corp. In my case they are leaving a corp whose API I have to go to an NPC corp.

EDIT: the character API properly shows the character in the NPC corp.

Any ideas?
Dragonaire
Here there be Dragons
#228 - 2015-02-16 01:31:23 UTC
Jahn Ntago - Sounds like there a missing clearing of the table per corp in that API that no one has notice before I'll see about fixing it.

https://github.com/Yapeal/yapeal/issues/93

See if that fixed it for you.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Jahn Ntago
Ntago Family Enterprises
#229 - 2015-02-17 01:28:56 UTC  |  Edited by: Jahn Ntago
I loaded up the two modded files from issue#93 and reran the processing twice.

Unfortunately the issue remains as it was.
Dragonaire
Here there be Dragons
#230 - 2015-02-17 17:39:54 UTC
Jahn Ntago - So I know its clearing all rows in the table and than applying the new info from the XML so I'm pretty sure its not an issue in Yapeal. I'm wondering how you are testing and if it's just something having to do with the cache times. MemberTracking uses a 6 hour cache so the XML can reflect them as still in the corp up to almost 12 hours after they leave a corp with bad timing. If you are still having problems with this please enable the XML filesystem cache so you have the XML for MemberTracking plus the other table's XML you that shows them as no longer in the corp so you can send them to me along with the log so I can try figuring out why it might still be happening. Also you should change the logging level so its including debug to get more details. You can E-mail everything to the one listed on GitHub and I'll take a look and see if I can figure out what is going on.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Jahn Ntago
Ntago Family Enterprises
#231 - 2015-02-18 02:11:26 UTC
I ran the processing again this morning and it worked. Must have been a caching issue?

corpMemberTracking now works properly for me.

Thank you.
Nick Partridge
Deep Core Mining Inc.
Caldari State
#232 - 2015-03-06 16:19:33 UTC
Any clues on the following error from the yapeal.log.

yapeal.WARNING: Failed to upsert data from Eve API corp/MemberTrackingLimited for 1542130595 {"exception":"[object] (PDOException(code: 22001): SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'base' at row 21 at G:\\server\\WEB\\public_html\\eve_the-partridges_co_uk\\yapeal\\lib\\Database\\EveApiToolsTrait.php:111)"
Mintoko
Taedium In Perpetuam
#233 - 2015-03-11 22:07:28 UTC
Nick Partridge wrote:
Any clues on the following error from the yapeal.log.

yapeal.WARNING: Failed to upsert data from Eve API corp/MemberTrackingLimited for 1542130595 {"exception":"[object] (PDOException(code: 22001): SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'base' at row 21 at G:\\server\\WEB\\public_html\\eve_the-partridges_co_uk\\yapeal\\lib\\Database\\EveApiToolsTrait.php:111)"



I was just about to report this myself. The column size for 'base' is too small. You can alter the table corpMemberTracking and change the base size to 100.
Dragonaire
Here there be Dragons
#234 - 2015-03-12 05:08:30 UTC
https://github.com/Yapeal/yapeal/issues/94 fixed

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#235 - 2015-04-23 19:33:03 UTC
Hi all give thought I'd let people I'm still around but been busy with work so not been working on Yapeal lately and it seems the last few changes I've made a couple months ago have largely killed most of the common bugs that get reported Smile. Hopefully I'll have some time again to get back to some stuff I was working on in another couple months.

As always thank you for using Yapeal, Dragonaire.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Cwittofur Cesaille
Sniggerdly
Pandemic Legion
#236 - 2015-04-29 18:42:27 UTC
I stumbled upon yapeal while browsing for a new framework to use the API. I have been using custom code since then. It seems like Yapeal will handle everything I need it to do... however I feel stupid and have no idea where to get started. The old documentation doesn't seem overly helpful. I'm not completely new to PHP; but I'm new enough to be lost! (I'm a .NET Developer). If there is anyone who could give me some insight on how to create a *basic* site that'd be super helpful. I can hit the ground running from there. I'm just not sure how/where to start :) I have it installed on my server already and in my web site area (going to symlink once I know it's all set).

Thanks in advanced!
Dragonaire
Here there be Dragons
#237 - 2015-05-06 21:03:19 UTC
Sorry I haven't even been on the forums for a week because of stuff IRL. Using the new Yapeal and the old aren't that different in many ways you have Yapeal running in the background via crontab or task scheduler and it 'automagically' keeps the database updated. Just start php use bin/yapeal.php and it'll take care of everything else for you.

If you need to do something like verify an API key using Account/APIKeyInfo you can call the required class directly. You can find each per API class in lib/Database/ and use the oneShot() method for sign-in type stuff. In bin/ you'll find a UtilRegisterKey class that should help with add the required APIKey info to the DB tables needed for Yapeal's automatic stuff to work.

All the database and tables creation stuff is covered in the *.md files using the bin/yc console commands. The best way to understand how Yapeal works is to look at the code itself starting with the lib/Yapeal class which kicks off all the other stuff. You can see how it's called by looking in bin.yapeal.php which just insures the composer class autoload is started then creates and start the class passing it it's one required parameter.

If you have any other questions just post them here I shouldn't be away so long again and I or someone else should be able to help you out.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Emma Otto
Mirai Industries
#238 - 2015-05-12 21:10:38 UTC
Thanks very much for this library. Like many others I'd started coding my own library to grab API data. But then I found this and you appear, at first glance, to have saved me so much time with your wonderful code!

Couple of questions:

1) I've added a corp API key but the corpDivisions and corpAccountBalance tables don't seem to be pulling in any data?
2) If I remove a key from the utilRegisteredKey table will this purge the data related to this key from the database on the next pull of data?
Riley Masterson
Mechanical Basterds
#239 - 2015-05-29 01:50:36 UTC
Hi Emma.

I just noticed that a corp key seems to require an access mask or it will not pull data, if you're not supplying one that could be the issue. the documentation said somewhere that the column can be null but that appears to only be valid for character keys.

For the second, it has been my experience that removing the key does not purge data during the next update.
Vaneel Tremurn
Mechanical Basterds
#240 - 2015-05-29 13:41:52 UTC
So, how did I play this game without yapeal again?

Question regarding corpMemberTracking. I don't seem to get any data for it. Most of the other corp keys (assetList, walletTransactions, industryJobs) all work fine. Any suggestions?