EVE Technology Lab

TL;DR: we periodically sync your EVE API data to the cloud, you access it anywhere with REST, Java or Javascript. Oh yeah, you can access the EVE static data dump the same way. Try it out here: https://www.eve-kit.org

Hi All,

In the course of creating various simple EVE tools, I found it very convenient to have a copy of all my character/corp data stored in the cloud and accessible via web APIs. This made it possible to do things like create a Google Docs spreadsheet which updates automatically from my data. This is easy to do against the EVE API, but I wanted to go a bit further. Specifically, I wanted to do three things: 1) keep a longer history of my data beyond what the EVE API normally provides; 2) annotate my data so I could track information not normally provided by the EVE API (think: blueprint ME and PE); and 3) make it all accessible from the web, so I could write tools on multiple platforms.

I created a simple backend server which periodically syncs all my data to a database, and allows access to the data through a simple REST API. This worked out pretty well, and seemed like a generally useful tool, so I decided to beef it up and offer the service to the wider community. To that end, I’m happy to announce the beta release of EveKit, a web-based toolkit for building applications based on the EVE API and EVE static data export. The idea is pretty simple: you specify an API key, and EveKit periodically snaps your Character or Corporation data to the cloud (keeping a history in the process). You can then use REST or one of the supported client APIs (currently: Java and Javascript) to access your data. Most tools aren't very useful without the EVE static data export, so I provide web APIs for accessing that data as well.

There’s probably already a service out there that does all this, but I couldn't find one (maybe I didn't look hard enough). Anyway, here are the key features:

• EveKit can snapshot both characters and corporations, at least as frequently as the EVE API servers allow.
• EveKit synchronizes your data on each snapshot and keeps a complete history (of things which have a history). So all your wallet transactions, market orders, industry jobs, etc., are stored forever. However, things like your current wallet balance, character sheet, etc., are updated in place.
• EveKit lets you attach persistent meta data to your synchronized data. I use this to remember the ME and PE on my blueprint originals.
• The EveKit access API supports a key and hash scheme, similar to the EVE API key scheme, so you can give different people different access to your data. Access to static data also uses a key and hash (mainly for metering and auditing).
• Your synchronized data is always available in bulk form. I recognize that not everyone will want to use the EveKit APIs to access their data. EveKit creates a nightly dump of your data (in MySQL dump format) which you can download from your account page. EveKit doesn’t trap your data behind custom APIs, you can get the raw stuff any time you like.

The beta is open to all takers, but each account is limited to at most two “synchronized” accounts (a synchronized account is either a Character or Corporation). I’ll consider upping the number of allowed accounts once I get a handle on usage and performance. I’m continuing to add documentation for many areas of the site. The biggest current gap in documentation is for the static data API. You may find it useful to download the JavaDocs (part of the Java API) even if you plan to use the REST API for static data, at least until I finish filling out this part of the site.

Bug reports, comments and suggestions are very welcome. Thanks for giving it a try!

Why do i get the feeling that in a few years time everyone will be using this website? :)

To be honest: this idea and website is exactly what i am looking for for years now. Up until now i have been using several applications to keep track of my data from everything eve related, but once a certain tool or website stops updating and becomes obsolete when eve gets a new expansion and the developers behind the tool or website for whatever reason do not want to work on the application than my data is pretty much worthless after one, or a few expansions.

I really hope this would resolve that problem, and that off course that your website will stay online in length of years, perhaps even as long as eve will exist....? :P

I am always in the mood to test out a new application, especially a well thought out one like this one! Sign me (Dunpeal Hunter), and my separate industry corp up for a beta key :)

Shouldn't you be sweating over a new release or something :)

EveKit architecture is pretty simple:

• deployed on Google Cloud using: App Engine, Cloud SQL and Compute.
• web site and data access is served from an elastic pool of front ends which are basically just Java Servlets
• sync'ing with the EVE API is handled by a pool of back ends which run on a schedule to pull new data (also Java Servlets)
• all data is stored in a Cloud SQL instance (automatically backed up). I use Hibernate (I know, I know) as a convenient ORM to handle the data. There's a bit of caching in App Engine to make commonly used features more snappy (like list of accounts, latest sync status, etc).
• a daily job runs on Compute and generates per-account MySQL dumps.
• EVE static dump is also stored in a Cloud SQL instance.
• All of this in one of Google's US zones. I'm curious how bad the latency will be for non-US users.

So basically an N-tier web app deployed to the cloud. Happy to answer any other questions.

Thanks for all the kind comments. I plan to stick it out as long as I can. Since this is all on Google Cloud, it's fairly easy to turn this over to someone else if I can't hack it anymore. It's also possible CCP will get very ambitious with CREST, in which case you might not need EveKit anymore :)

P.S. In case it wasn't clear from my post, you don't need to do anything special to participate in the beta. Just log into the site (using google or twitter) and add the account(s) you want to sync.

So you mention you can put metadata on there. How.. am I missing the obvious here? Navigation as is pointed out in the FAQ's is a pain to say the least especially without a list of current plates available.

Or is the metadata you put after you dump the char data and do it that way?

2014-07-29 EveKit Beta Status

Updates:

- Beta release has been active for about a week now, thanks to all participants! Interesting stats below...
- Static DB has been upgraded to the Crius release!
- Participants have uncovered numerous bugs which were fixed during the week. Thanks again!

Next Steps:

- I encourage everyone to give the API section a try. This "plate" lets you test out the API against your sync'd data. You can reach the API section using this link.
- So far so good with the load but there are a few synchronization issues I'd like to fine tune.
- Menu/navigation redesign is on the short list. In the mean time, thanks for bearing with the somewhat unorthodox navigation scheme.
- I assume everyone would appreciate a vid or two showing how to use some of the features. I'll try to cobble something together next weekend.
- I'd like to convert an existing EVE app to use EveKit as the backend in place of some proprietary local store. Please let me know if you have any suggested apps. Going through this exercise will likely drive changes to the API, which is a good thing.

Interesting stats:

- 10 users are participating in the beta so far.
- 13 separate EVE accounts are synchronizing on an automatic basis.
- 3828 account synchronizations have been performed in the last week (about 550/day)
- Average sync time was 260 seconds.

The database took a bit of a holiday earlier this morning, slowing everyone down, but we're back to normal now.

If you created your corp account around 9:44 UTC on 8/2 then you're syncing now but it's a bit slow, I'm guessing this is a decent sized corp. The first sync happened during the DB issues and took around 3 hours to sort itself out. The second sync took about an hour. The third sync just started so I'll keep an eye on it. Typically, the first few syncs are slow as the site builds up wallet history, etc (e.g. lots of DB commits). Subsequent commits should go faster.

As far as sync's go, the site should be feature complete.

UPDATE: and as I was about to hit 'post', your latest sync just completed. It took about 7 minutes.

Thanks, just need some spare time to get more into it. I really shouldn't do so many different things at once :D



EDIT:

Hmm, somehow I'm just getting 20 entries out of https://www.eve-kit.org/access/common/Asset/list?accesskey=********&accesshash=****************************************

Should be a few thousand at least :D
And I already tried to set the maxresults to a high number, still returning 20 entries

Ok, I just bumped up all the default API batch sizes to 1000 entries (if maxresults < 1, then this default is used). You can explicitly select a smaller batch with the "maxresults" parameter. If you have more than 1000 entries, you'll need to make another call passing the continuation parameter.

That's an average time for sync'ing one account. The times vary a bit depending on the type of account (active corporations take much longer to sync). There are also occasional disruptions in Google's cloud, a pretty nasty one has been going on for almost 24 hours now where I see significantly more DB connection drops than typical.

Here's what I'm seeing over the last week or so (these are for sync'ing one account):

min: 0.23 minutes
avg: 5 minutes
max: 136 minutes
99%:40 minutes
95%: 20 minutes
median: 1.93 minutes

This is sync'ing 22 accounts, 10 of which are for corporations. For corporations, the breakdown looks like this:

min: 0.28 minutes
avg: 6.23 minutes
max: 136 minutes
99%: 51 minutes
95%: 25.5 minutes
median: 2.5 minutes

and for characters:

min: 0.23 minutes
avg: 3.93 minutes
max: 89 minutes
99%: 29 minutes
95%: 16.5 minutes
median: 1.6 minutes