These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Yapeal PHP API library (version 1.0.5)

First post
Author
Previous Topic Next Topic
PsyKzz
Republic Military School
Minmatar Republic
#81 - 2011-10-30 19:33:24 UTC
Dragonaire wrote:
Not really since that ability is going away when old keys do from what I understand and once again by design Yapeal expects to have keys etc for anything in account, char, or corp sections in the requests plus the database expect the full data you only get from having the keys.

Yapeal is a made to be used when you have the required keys etc not for just general random access to the APIs if that's what you need there are several other libraries for that but of course it's up to you then to parser the XML and put it in the database yourself too Blink

Instead of just trying to say indirectly that Yapeal is doing something wrong why don't you learn how to use it correctly first.


Okay, so to help me out then, when adding / updating data to the db does yapeal overwrite data if it finds any?
For example if i created my own to add the data to the database would yapeal, if it ever got a key for such corp, replace it?

Meh.
Dragonaire
Here there be Dragons
#82 - 2011-10-30 19:42:22 UTC
Miko Kiun wrote:
Dragonaire wrote:
If you are talking about /eve/CharacterID.xml.aspx and /eve/CharacterName.xml.aspx they are really outside of the scope of what Yapeal does and it's unlikely I'll add support for them except maybe in some kind of util script if I find and need for it or a few extra days with nothing else to do Blink

If you mean /eve/CharacterInfo.xml.aspx it would have already been done except CCP was stupid and put an API that requires apiKeys and vCodes in the eve section instead of where it belonged in char or account. You can get all the same information from the other APIs anyway so I've never seen why anyone would find it that useful as it is. I've got some ideas for changes to Yapeal that MAY in some distance future be able to overcome CCP's stupidity on the placement of this API but let's just say being able to support this API is going to be purely an afterthought in any changes.


I'm confused, characterinfo doesn't require keys.
https://api.eveonline.com/eve/CharacterInfo.xml.aspx?characterID=90338329
Is corp history accessed somewhere else? I can't find it anywhere in the yapeal tables
To get the full information from that API you do need the keys etc. I believe you need a key with at least the low access level to get the corp history info but I could be wrong as I've not looked at it since they added that to it and the examples on eve-id aren't really clear on that point.

The main thing is Yapeal doesn't currently support having APIs with and without parameters in the same section of the API as it now works so any API that doesn't follow the 'normal' pattern for a section can't be added at this time. As I've said in some of my other posts I've had some ideas on how to make changes that might lift that limitation but it will require basically a total rewrite of much of Yapeal so it's not something that can happen overnight.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
PsyKzz
Republic Military School
Minmatar Republic
#83 - 2011-10-30 19:45:15 UTC
Dragonaire wrote:
Miko Kiun wrote:
Dragonaire wrote:
If you are talking about /eve/CharacterID.xml.aspx and /eve/CharacterName.xml.aspx they are really outside of the scope of what Yapeal does and it's unlikely I'll add support for them except maybe in some kind of util script if I find and need for it or a few extra days with nothing else to do Blink

If you mean /eve/CharacterInfo.xml.aspx it would have already been done except CCP was stupid and put an API that requires apiKeys and vCodes in the eve section instead of where it belonged in char or account. You can get all the same information from the other APIs anyway so I've never seen why anyone would find it that useful as it is. I've got some ideas for changes to Yapeal that MAY in some distance future be able to overcome CCP's stupidity on the placement of this API but let's just say being able to support this API is going to be purely an afterthought in any changes.


I'm confused, characterinfo doesn't require keys.
https://api.eveonline.com/eve/CharacterInfo.xml.aspx?characterID=90338329
Is corp history accessed somewhere else? I can't find it anywhere in the yapeal tables
To get the full information from that API you do need the keys etc. I believe you need a key with at least the low access level to get the corp history info but I could be wrong as I've not looked at it since they added that to it and the examples on eve-id aren't really clear on that point.

The main thing is Yapeal doesn't currently support having APIs with and without parameters in the same section of the API as it now works so any API that doesn't follow the 'normal' pattern for a section can't be added at this time. As I've said in some of my other posts I've had some ideas on how to make changes that might lift that limitation but it will require basically a total rewrite of much of Yapeal so it's not something that can happen overnight.


Honestly that didnt answer my question, i understand for yapeal to work it requires more keys with access.
What i am planning to do in the mean time is my own calls and fill the DB with my own data.

What i am worried about is when yapeal goes to input something is it going to spurg and be all like 'oh **** data already exists' or is it going to just delete and replace?

Meh.
Dragonaire
Here there be Dragons
#84 - 2011-10-30 19:52:47 UTC
Quote:
Okay, so to help me out then, when adding / updating data to the db does yapeal overwrite data if it finds any?
For example if i created my own to add the data to the database would yapeal, if it ever got a key for such corp, replace it?
If you put something into an API table that Yapeal works with it will overwrite it with the newer data yes. Some tables it truncates or deletes rows from before adding the new ones and others it just updates any overlapping existing data. So for things like the corpSheet it'll overwrite but an API like assetList it clears out the old data first. The easiest way to know which it does currently is to see if the per API class has a method named prepareTables() and see what queries it might be running. If an API doesn't have that method it's just overwriting the data.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Miko Kiun
Doomheim
#85 - 2011-10-30 20:31:40 UTC
corp history doesn't require a key, that link I posted previously shows that. I noticed that you have a CharacterInfo placeholder file, so I guess I'll play around with making something work to get corp history and store it in a table...

I don't think that the walletjournal/transaction history is walking backwards. I know I have significantly more transactions in my journal than what is getting store in the database.
Dragonaire
Here there be Dragons
#86 - 2011-10-30 23:33:01 UTC
Miko Kiun wrote:
corp history doesn't require a key, that link I posted previously shows that. I noticed that you have a CharacterInfo placeholder file, so I guess I'll play around with making something work to get corp history and store it in a table...
One great thing with open source software is there is always that option of course and if you do come up with something that can be integrated back into Yapeal I'll be happy to look at it.

Miko Kiun wrote:
I don't think that the walletjournal/transaction history is walking backwards. I know I have significantly more transactions in my journal than what is getting store in the database.
It may not be working right as I said but I've been busy working on many other things and not having a lot of test accounts to work with which have more than half a dozen rows a week at most it's not easy to work on either. I will let you know that just because you have a lot of history that you can see in-game doesn't mean it will all be available through the APIs. CCP limits you to only walking back for the last week or so worth of rows. They use to allow more but with the latest changes they seem to have cut it down farther than before. So to put it another way you can't walk back to see that first frigate you bought when you started play Eve three years ago. To do that you would have had to started collecting it at the time to have it now.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Norian Lonark
Center for Advanced Studies
Gallente Federation
#87 - 2011-10-31 01:14:41 UTC  |  Edited by: Norian Lonark
Dragonaire wrote:
Generally just copying everything should work but if your host gives you ssh access it's usually better to install it and then just copy over your config/yapeal.ini and anything you need from the util* tables. The advantage to doing it this way is updates to Yapeal should be easier to do since you can just follow instructions from INSTALL.txt and don't have an extra copy step to get from your local machine to the host machine and maybe miss something.


Hi Dragonaire,

thanks for getting back to me so quickly. I have to say using it locally it is absolutely fantastic and taken my little project from being an absolute nightmare to something quite manageable (so far Big smile).

I would like to ask when copying the files to my host (as I think SSH access is a problem) which files / folders should I not copy i'm obviously thinking the install folder but are their any others that I should miss?

Thanks again,

Nori.

Start wide, expand further, and never look back
Dragonaire
Here there be Dragons
#88 - 2011-10-31 01:29:31 UTC
HI all thought it was time I post a little update on what I've been working on in Yapeal over the last week or so. Some of you may have already noticed it if you been looking at the web sites especially the subversion. For starters it's no longer a subversion repository but now uses Mercurial instead. You might wonder why the change and all I can say is I've been looking around at some of the newer distributed version control software like Git, Mercurial, etc because everyone seems to think they are better plus they say they have features that might be useful. I really looked at moving to Git but for me how it was an improvement over what Subversion had to offer I could never figure out. I never really looked at Mercurial because I never saw where it was really different from Git and so didn't see any reason to go with something that isn't used by many people from my experience anyway Blink.

That changed after Callean Drevus decided to release the source code for EMT. He happens to use Hg (Mercurial) for his project and since he uses Yapeal in it I wanted to take a look and see if there was anything I could learn from how he was using it and improve on Yapeal or that I could suggest to him some things about using it more effectively in his project. Originally I only figured I'd checkout his code and maybe sent him a few Eve-mails or E-mail and maybe chat with him a little about stuff but while looking at the version of Yapeal he was using I noticed it was a little behind the current version. Since it needed updating I decided this might be a good time to really try out Hg and work with it. So I setup an account on Bitbucket and forked his project to try stuff out. You can see the fork here if you're interested.

https://bitbucket.org/Dragonaire/eve-marketeer/overview

So now I had my own sandbox to work on but no real idea how to do so with Hg. I did what I always do and started looking around at Bitbucket, Googling more about Hg, and looking at the man pages for Hg. While I was doing so I came across an interesting site about Mercurial called: Hg Init: a Mercurial tutorial which has a section just for people like me coming from a Subversion background. I don't know if the author is brilliant or if all the other reading I had done on the differences finally just came together when I read what he wrote but I decided I liked Hg and could finally see where it might be be better then Subversion for Yapeal.

I won't bore you with any more of the background reasoning that lead to me deciding to switch Yapeal from Subversion to Mercurial but that what I did a few days ago.

Continued in next post.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Dragonaire
Here there be Dragons
#89 - 2011-10-31 01:46:47 UTC
... Continued

So how does this effect you is what you really want to know right?

If you use the archives not much. You might notice that instead of having one or more folders on SourceForge where you can download versions of Yapeal there is only one called archives/ with multiple versions in it. You may notice that the numbering system used for the versions has changed. With Subversion I had a nice simple number that increased every time I committed something to the repository but with a distributed version control system like Hg I don't have that anymore.

Old school three part numbering (1.0.2) doesn't really work well either or at least are very hard to manage automatically so I decided to go with a system based on the date and time I build the archives but make it looks more like the old school three part numbering system. Does it really matter to anyone? Not really the only time it'll probably be used is when I ask you what version you are using when reporting a issue or when you are wondering if you need to update your version. The only ones you'll probably ever really care about are the ones that have '-latest' in their names as they are of course the latest version you'll be wanting to upgrade to as features are added and bugs fixed. If you always make sure you are regularly updating to the latest version it doesn't really matter what the version number is.

If you use the repository to get Yapeal you'll have to use Mercurial or a GUI that understands it but if you only care about what use to appear in trunk/ before it'll be in default/tip which is what Hg defaults to when you clone it.

If you are like me and need to see everything because I am often working on several things like bug fixes and new features in branches they are still there for now but there are better ways to do things like that in Hg and you'll notice many of the current branches will disappear over time as I start taking advantage of them. A couple of them have already like admin/ and wiki/. Admin was where I kept stuff that was only used during development for things like making the archives etc. I've now done what I should have done before and moved them into a separate repository which is private at least for now since there really isn't anyone else doing releases or needing any of the other stuff that was found there. If / When needed I can make it public or give others access to it as well but there is no need to confuse everyone by having it there. The wiki move is because GoogleCode uses a separate Hg repository for it now. With Subversion it was part of the main one but now is on its own. I also decided to do a little cleanup and have moved pics/ out of the repos while I was at it and have them just as a download on SF instead. That last change with pics/ should make the download smaller but still make them available to anyone that needs them.

There are probably several other changes you might notice but they shouldn't really matter to anyone besides me. If you do notice something you used and can't find it now let me know and I'll tell you where it went to. I've got a full archive of everything in the old Subversion plus you'll find most everything in the Hg history since I did a full import of the tags, branches, and trunk to it.

One last thing that everyone will probably notice when they go to install or upgrade Yapeal is the README and INSTALL.txt now are named README.md and INSTALL.md. They are still basically plain text files but can be converted directly to HTML pages for display with the right software. They have been updated with the new instructions needed for working with the changes to Mercurial and the archives while I was at it Smile

If you have any questions I'll help where I can. If you just have comments they are welcome as well.

version 11.303.0923

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Dragonaire
Here there be Dragons
#90 - 2011-10-31 02:09:17 UTC
Quote:
thanks for getting back to me so quickly. I have to say using it locally it is absolutely fantastic and taken my little project from being an absolute nightmare to something quite manageable (so far Big smile).

I would like to ask when copying the files to my host (as I think SSH access is a problem) which files / folders should I not copy i'm obviously thinking the install folder but are their any others that I should miss?
You only need to copy the directory structure of cache/ and the *.xsd file but not the *.xml or *.sql files. install/ is optional but if you are having problems getting it to work you might need some of the scripts from it. They won't run with CGI by default but might still be modifiable to work with it should you need them. You don't need any log files in log/ either just the folder and the .htaccess file. Most directories have a .htaccess file in them so if you do happen to copy something you wish you hadn't no one should have access to them. I do still highly suggest you don't put it anywhere the web server can see it but with care it should be minimally secure to do so. Everything else you'll be needing.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Norian Lonark
Center for Advanced Studies
Gallente Federation
#91 - 2011-10-31 17:47:22 UTC  |  Edited by: Norian Lonark
Hi,

Ok I tried to get the installation working via SSH and its been successful. Big smile

When I run Yapeal.php I get the following:

Warning: getopt(): No support for long options in this build in yapeal/inc/parseCommandLineOptions.php on line 82

everything seems to have updated in the sql database apart from the charindustryjobs (which is unfortunatly the main thing I am after Sadv)

Any ideas or pointers?

Many thanks,

Nori.

Start wide, expand further, and never look back
Miko Kiun
Doomheim
#92 - 2011-10-31 20:43:10 UTC
Is the jabber room still used? I may not have set things up correctly, I'm being asked what settings I want to use to create the room, and I'm the only one there...

I'm getting an error I haven't been able to sort out.

WARNING: Could not prepare eveAllianceList API tables to accept new data for 0

EXCEPTION: Failed to get ADOFieldObjects for columns in YapealQueryBuilder
Code: 4

All the other APIs appear to be correct, and I verified the table exists and has all of it's columns...


Also, Dragonaire, I have CharacterInfo working. I had to modify the eveCharacterInfo and SectionEve php files. I created an XSD, as well as two new tables. Do you have a preference on how you would like me to deliver this to you for your perusal???
Miko Kiun
Doomheim
#93 - 2011-10-31 21:06:35 UTC
Miko Kiun wrote:
Is the jabber room still used? I may not have set things up correctly, I'm being asked what settings I want to use to create the room, and I'm the only one there...

I'm getting an error I haven't been able to sort out.

WARNING: Could not prepare eveAllianceList API tables to accept new data for 0

EXCEPTION: Failed to get ADOFieldObjects for columns in YapealQueryBuilder
Code: 4

All the other APIs appear to be correct, and I verified the table exists and has all of it's columns...


Also, Dragonaire, I have CharacterInfo working. I had to modify the eveCharacterInfo and SectionEve php files. I created an XSD, as well as two new tables. Do you have a preference on how you would like me to deliver this to you for your perusal???



Nevermind on the error, I resolved it... My other questions still stand tho Smile
Dragonaire
Here there be Dragons
#94 - 2011-11-01 06:03:05 UTC
Quote:
Warning: getopt(): No support for long options in this build in yapeal/inc/parseCommandLineOptions.php on line 82
Seems even in Linux support for long options was uncommon before 5.3 in PHP so I'll be looking at changing it so it only tries to use them with newer versions I guess on all platforms P

Quote:
everything seems to have updated in the sql database apart from the charindustryjobs (which is unfortunatly the main thing I am after
If the other char APIs seem to be working and not the one you want make sure you didn't somehow turn it off in one of the masks.

Quote:
Is the jabber room still used? I may not have set things up correctly, I'm being asked what settings I want to use to create the room, and I'm the only one there...
Looks like the server died I've sent an E-mail to the owner to see if he can give it a poke. You can always try private IM on Gmail as well. Just try dragonrun1 there and you'll get me. Keep in mind I'm West Coast US and work an afternoon/evening shift that can make it hard to catch me for some peopleBlink

Quote:
WARNING: Could not prepare eveAllianceList API tables to accept new data for 0
No idea how you're getting that error. Yapeal doesn't use prepare SQL statements anywhere so only thing I can think of is you were trying to use them somewhere in your new code maybe?

Quote:
EXCEPTION: Failed to get ADOFieldObjects for columns in YapealQueryBuilder
Code: 4
I'm guessing you aren't passing the table name to the constructor for YapealQueryBuilder as the first parameter or maybe you're not passing in the correct DSN for connect in $dsn?

Do you have a preference on how you would like me to deliver this to you for your perusal???

If you can do it as pull request with Hg that would be great but I'm not sure I have that setup right yet else a unified diff would work sent to me via E-mail. The one other way would be just a copy all the files you've changed in an archive using zip or tar.gz etc. That last option might be the easiest overall and you can use the same E-mail account from above as well.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Norian Lonark
Center for Advanced Studies
Gallente Federation
#95 - 2011-11-01 10:58:39 UTC
Hi Dragonaire,

I just wanted to say a big thank you again for a wonderful tool and great support you give.

I have got all my issues resolved now and Yapeal is working brilliantly and doing everything that I want it to do. Big smile

Cheers,

Nori.

Start wide, expand further, and never look back
Miko Kiun
Doomheim
#96 - 2011-11-01 17:33:03 UTC
Diffed files sent over via email...

I was able to get those error messages sorted out. The problem was actually a setting in MySQL. Some of the XML files are pretty decent sized, like AllianceList. If you have your max_allowed_packet set to low yapeal throws those errors. I didn't figure it out till I snagged the SQL statement that was failing and tried running it through my SQL admin tool to see if the statement was valid.
Dragonaire
Here there be Dragons
#97 - 2011-11-01 22:04:10 UTC
Ah yeah there are a few there that are very large was something on the old thread about the packet size issue. Looks like you get it sorted out but the other way to fix it for anyone that can't make changes to their config file for MySQL is to lower the default number of records Yapeal tries to store at a time by changing YapealQueryBuilder::MAX_UPSERT from 1000 to something lower until you stop having problems. You want to keep it as high as possible to have less inserts so they are faster but small enough not to cause errors.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#98 - 2011-11-02 03:01:17 UTC
Question: I know that Yapeal is supposed to run periodically from a cronjob, but how would I go about updating 1 characters info via my application? For example, if the logged in user wishes to update his character's information to get the most up-to-date info, how can I go about doing that (updating a character or account on demand rather than on a schedule)?

Thanks!
Zaepho
Goosewarms
Tactical Hunters and Gatherers
#99 - 2011-11-02 03:48:24 UTC
Sable Blitzmann wrote:
Question: I know that Yapeal is supposed to run periodically from a cronjob, but how would I go about updating 1 characters info via my application? For example, if the logged in user wishes to update his character's information to get the most up-to-date info, how can I go about doing that (updating a character or account on demand rather than on a schedule)?

Thanks!


The data is always as up to date as possible within the confines of the cache intervals. Yapeal will collect and store the data on it's schedule as often as possible.
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#100 - 2011-11-02 03:52:18 UTC
Zaepho wrote:
Sable Blitzmann wrote:
Question: I know that Yapeal is supposed to run periodically from a cronjob, but how would I go about updating 1 characters info via my application? For example, if the logged in user wishes to update his character's information to get the most up-to-date info, how can I go about doing that (updating a character or account on demand rather than on a schedule)?

Thanks!


The data is always as up to date as possible within the confines of the cache intervals. Yapeal will collect and store the data on it's schedule as often as possible.


of course, but it's still on a schedule. Is there any way for an on demand function that updates specific chars/corps? I don't mind either way, was just a question that popped into my head. =)