EVE Technology Lab

The only APIs where there is a direct link from the key to ownerID is in account section. The eve, map, and server sections don't really have an owner so they default to 0. Char section APIs use the characterID and corp section uses the corporationID but since Yapeal can work with multiple keys for each there's really no direct link to the key used. To put it another way Yapeal doesn't care if there are one or multiple keys active for the same character or corporation and will use them in a random order when trying to retrieve the XML data from the API servers so there really isn't a way to say which key will be used.

If you enforce only one key of a type on a per character/corporation basis in your application you can then try to do a one to one mapping but watch out for account type keys as you actually have a one to many relationship with them.

Hope that helps you.

Or maybe learn to use the accountApiKeyInfo, accountCharacters, and accountKeyBridge which really does the linking you were asking about there just isn't a real one to one link except as I said you enforce one in your application. You can do that with a few SQL queries on the exist info from the above tables.

I updated Yapeal last Saturday (zip version) and I'm having high CPU usage since Sunday downtime. I see all log files empty with default Yapeal config. How can I get logging to work?

Thanks!

@Bado Roul look in the config/yapeal.ini file for the cache settings you probably need to change.

I haven't had a chance to really add any extra debug stuff to Yapeal since the change to new error logging code but you can try trace but it'll probably not have much interesting either but at least you'll know what all code it does run . Have you run install/checkForRequirements.php yet it might spot something that should help like the cache directory not being writable etc. Clear up any errors it might find and see if that helps. If doing those things doesn't seem to fix it let me know and I'll see what else I can come up with that might give us a clue to why it's not caching anything.

I'll make a bet if you looked at what is using mast of your CPU it will turn out to be MySQL trying to put everything into the tables. Also it really is better to run Yapeal every minute as then it doesn't get behind and have to run for extended periods to catch up all the time. It has some built-in code to try evening out globs of APIs needing retrieved all at the same time that can't do it's job if ran infrequently. Everyone is always shooting themselves in the foot trying to out think me and Yapeal on that

It will be very busy for a while when you first start it as it does have to try getting all of the APIs for everything you have asked it to do but once that initial fill is done it should within a couple days only have an occasional spike when a couple of the larger APIs happen to need to be done at the same time.

Yapeal itself spends 99% of it's time waiting on either the API servers to give it the data it needs from the network or waiting on MySQL to finish storing the data in the database the code itself usually only takes a second or two at most to run and typically only is doing something for 1/4 of a second or less of the time it take to do an API. Disk I/O has been the most common limitation on how many keys it can handle on most shared hosts as they are usually setup for lots of reads but few writes to the file system but Yapeal spends 90% of it's time writing directly for cache or indirectly through database table inserts in MySQL.

Hopefully that helps you understand what's going on.

Hi Everyone,

I have a bit of an odd problem. Yapeal has been working fine for me however last week it suddenly stopped. When I run Yapeal it looks like it has run fine but nothing in the database updates. I have ran the check requirements and check privs scripts again and they still come back as ok.

Also nothing is in the log folders.

Is there anything I can do to get a clue as to what might have gone wrong?

Many thanks,

Nori.

Hi Dragonaire,

Thanks for the tips how I got it working again was I deleted the API information from the utilRegisteredKey table, added it again (didnt change the API) and it started working again.

Will keep an eye on it over the next few days so may contact you for further help and many thanks for the kind offer.

Best regards,

Nori.

I would have been surprised if you didn't have lots of addRow() in trace consider it's called to store every single row (record) into the database that Yapeal does. Let me give an example with a very simple API like server/ServerStatus.

Once Yapeal decides the API needs to be updated it updates utilCachedUntil with new date-time with a short timeout in case something goes wrong on this update it can re-try in about 5 minutes. (BTW for this API that's it's normal cache time but on most APIs this is shorter than normal) It then tries to get the API data from the server. If it gets the data and you have database caching turned on it writes the XML to the utilXmlCache table and then does an upsert to the serverServerStatus table. Yapeal then figures out the new cachedUntil time it's going to use from XML (No it doesn't always use the one from XML directly because often the APIs lie about the true cache time and I got tired of all the errors) It then updates utilCachedUntil with the new date-time. So for just a single record API there are 3-4 writes to the database. 2-3 of those are overhead that don't change with the number of records that need to be updated.

Now let's think about an API like AssetList where there can be 1000s of rows of data for each of them addRow() is called. That's per char/corp as well so with just a dozen or so chars/corps you can easily end up with 10000 calls to it or more. The best way to get a good idea how many times it will be called look at how many rows (records) of data are in the all the API data tables plus the XML cache if you are using it and utilCachedUntil.

Now just to make something clear Yapeal doesn't actual store the data into the database on each call to addRow() it actually saves up the data and wraps up to 2000 of them by default into a single extended insert or upsert inside a database transaction to make them faster.

In all of the above I've ignored the selects that are needed on the database as well for Yapeal to do it's job. There are about six or so per API and do to the nature of them most can't be cached but for the ones that can Yapeal does some internal caching.

Let's now compare the number of writes Yapeal does to the database compared to say a blogging or forum site.

Let's say there are 10000 active people on the site and they are all quick readers and fast typists. They each take 5 seconds to read a post and 5 seconds to add their own and they are doing it constantly (Typical EVE General Discussion poster ). Lets guess each post causes three rows (records) in different tables to be written for each of them and it has to read from two tables to display one. So we have 10000 / 10 * 3 = 3000 rows being written each second and 10000 / 10 * 2 = 2000 reads. Now Yapeal can do that many writes on a single API for a single char/corp but a lot less reads. Most hosting site are setup at best to handle those type of loads and actually most of them are figuring on closer to 100 reads per write to the database since that's much more typical.

Given the above is it any wonder that when you start trying to add a few hundred or thousand users to Yapeal it's usually the database directly or indirectly via file I/O that can't keep up on most shared hosts or even a VPS? The above numbers are ignoring any overhead your applications might be adding which probably make up for some of the reads. The above numbers aren't going to be unique to Yapeal either but generally true for anything that tries to keep the all the API data in a database.

Anyway I hope that gives everyone a better idea of what you are truly asking of your poor server when working with the APIs

First I don't want you to think I was saying there was anything wrong with your VPS I actual think Linode is a great service. I mostly did the post to give everyone some data to work with to understand the differences in the load that API applications in general and Yapeal in particular have so they have something to work with when trying to scale up with them and might have had some baring on your problems since I didn't know how many chars or corps you were trying to run it with.

You are right 1001 inserts on corpWalletJournal doesn't look right at all. I would say somehow the exit conditions aren't right and my fail safe is kicking in at the 1000-1001 mark and keeping it from becoming an infinite loop It's interesting to me that only the Journal was effected as transactions use the same code with only minor diffs but I'll look into it now that I've got a better idea what's happening so thanks for the extra logging it really helps. I'm also wondering if you are seeing the same thing happen with the same 2 APIs in the char section by chance?

Edit I've found one difference that could be causes the problem and I have something you can try to see if it does fix it. In corpWalletJournal on line 218 which reads:

$qb = new YapealQueryBuilder($tableName, YAPEAL_DSN, FALSE);

try changing it to:

$qb = new YapealQueryBuilder($tableName, YAPEAL_DSN);

That's one of only 2-3 minor diffs from corpWalletTransactions and the only one I see that might have an effect. Let me know if that seems to fix the issue.