These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

EveKit - web accessible EVE character and corp data (with history)

First post
Author
salacious necrosis
Garoun Investment Bank
Gallente Federation
#1 - 2014-07-22 02:48:45 UTC
TL;DR: we periodically sync your EVE API data to the cloud, you access it anywhere with REST, Java or Javascript. Oh yeah, you can access the EVE static data dump the same way. Try it out here: https://www.eve-kit.org

Hi All,

In the course of creating various simple EVE tools, I found it very convenient to have a copy of all my character/corp data stored in the cloud and accessible via web APIs. This made it possible to do things like create a Google Docs spreadsheet which updates automatically from my data. This is easy to do against the EVE API, but I wanted to go a bit further. Specifically, I wanted to do three things: 1) keep a longer history of my data beyond what the EVE API normally provides; 2) annotate my data so I could track information not normally provided by the EVE API (think: blueprint ME and PE); and 3) make it all accessible from the web, so I could write tools on multiple platforms.

I created a simple backend server which periodically syncs all my data to a database, and allows access to the data through a simple REST API. This worked out pretty well, and seemed like a generally useful tool, so I decided to beef it up and offer the service to the wider community. To that end, I’m happy to announce the beta release of EveKit, a web-based toolkit for building applications based on the EVE API and EVE static data export. The idea is pretty simple: you specify an API key, and EveKit periodically snaps your Character or Corporation data to the cloud (keeping a history in the process). You can then use REST or one of the supported client APIs (currently: Java and Javascript) to access your data. Most tools aren't very useful without the EVE static data export, so I provide web APIs for accessing that data as well.

There’s probably already a service out there that does all this, but I couldn't find one (maybe I didn't look hard enough). Anyway, here are the key features:

  • EveKit can snapshot both characters and corporations, at least as frequently as the EVE API servers allow.
  • EveKit synchronizes your data on each snapshot and keeps a complete history (of things which have a history). So all your wallet transactions, market orders, industry jobs, etc., are stored forever. However, things like your current wallet balance, character sheet, etc., are updated in place.
  • EveKit lets you attach persistent meta data to your synchronized data. I use this to remember the ME and PE on my blueprint originals.
  • The EveKit access API supports a key and hash scheme, similar to the EVE API key scheme, so you can give different people different access to your data. Access to static data also uses a key and hash (mainly for metering and auditing).
  • Your synchronized data is always available in bulk form. I recognize that not everyone will want to use the EveKit APIs to access their data. EveKit creates a nightly dump of your data (in MySQL dump format) which you can download from your account page. EveKit doesn’t trap your data behind custom APIs, you can get the raw stuff any time you like.

The beta is open to all takers, but each account is limited to at most two “synchronized” accounts (a synchronized account is either a Character or Corporation). I’ll consider upping the number of allowed accounts once I get a handle on usage and performance. I’m continuing to add documentation for many areas of the site. The biggest current gap in documentation is for the static data API. You may find it useful to download the JavaDocs (part of the Java API) even if you plan to use the REST API for static data, at least until I finish filling out this part of the site.

Bug reports, comments and suggestions are very welcome. Thanks for giving it a try!

Use EveKit ! - Tools for EVE Online 3rd party development

CCP FoxFour
C C P
C C P Alliance
#2 - 2014-07-22 06:45:16 UTC
This is... really cool! :D

Also a really good site to integrate the EVE SSO, which while only in a limited test right now should be out to a broader audience soon. Send me a private mail if you are interested in using it.

If you don't mind me asking, any chance you mind providing more details about how you are using the Google Cloud for hosting?

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Dunpeal Hunter
Zervas Aeronautics
The Bastion
#3 - 2014-07-22 10:52:14 UTC
Why do i get the feeling that in a few years time everyone will be using this website? :)

To be honest: this idea and website is exactly what i am looking for for years now. Up until now i have been using several applications to keep track of my data from everything eve related, but once a certain tool or website stops updating and becomes obsolete when eve gets a new expansion and the developers behind the tool or website for whatever reason do not want to work on the application than my data is pretty much worthless after one, or a few expansions.

I really hope this would resolve that problem, and that off course that your website will stay online in length of years, perhaps even as long as eve will exist....? :P

I am always in the mood to test out a new application, especially a well thought out one like this one! Sign me (Dunpeal Hunter), and my separate industry corp up for a beta key :)

I dont know if this is important or not, but i have chickens.....

Altro Targazen
Doomheim
#4 - 2014-07-22 11:02:05 UTC
CCP FoxFour wrote:

If you don't mind me asking, any chance you mind providing more details about how you are using the Google Cloud for hosting?

I suppose Google App Engine especially it seems to be related to Java ;-) I'm curious as well.
salacious necrosis
Garoun Investment Bank
Gallente Federation
#5 - 2014-07-22 12:36:59 UTC
CCP FoxFour wrote:
This is... really cool! :D

Also a really good site to integrate the EVE SSO, which while only in a limited test right now should be out to a broader audience soon. Send me a private mail if you are interested in using it.

If you don't mind me asking, any chance you mind providing more details about how you are using the Google Cloud for hosting?


Shouldn't you be sweating over a new release or something :)

EveKit architecture is pretty simple:

  • deployed on Google Cloud using: App Engine, Cloud SQL and Compute.
  • web site and data access is served from an elastic pool of front ends which are basically just Java Servlets
  • sync'ing with the EVE API is handled by a pool of back ends which run on a schedule to pull new data (also Java Servlets)
  • all data is stored in a Cloud SQL instance (automatically backed up). I use Hibernate (I know, I know) as a convenient ORM to handle the data. There's a bit of caching in App Engine to make commonly used features more snappy (like list of accounts, latest sync status, etc).
  • a daily job runs on Compute and generates per-account MySQL dumps.
  • EVE static dump is also stored in a Cloud SQL instance.
  • All of this in one of Google's US zones. I'm curious how bad the latency will be for non-US users.

So basically an N-tier web app deployed to the cloud. Happy to answer any other questions.

Use EveKit ! - Tools for EVE Online 3rd party development

salacious necrosis
Garoun Investment Bank
Gallente Federation
#6 - 2014-07-22 12:40:34 UTC
Altro Targazen wrote:
CCP FoxFour wrote:

If you don't mind me asking, any chance you mind providing more details about how you are using the Google Cloud for hosting?

I suppose Google App Engine especially it seems to be related to Java ;-) I'm curious as well.


Yep, GAE and mostly java as detailed in my other post. The daily compute job is a bit of bash and other command line bits. The cool kids seem to be using Python or Go on GAE these days, but since my day job is about 95% Java, that's what I went with. I have Python and PHP client APIs on my todo list.

Use EveKit ! - Tools for EVE Online 3rd party development

salacious necrosis
Garoun Investment Bank
Gallente Federation
#7 - 2014-07-22 13:28:48 UTC
Dunpeal Hunter wrote:
Why do i get the feeling that in a few years time everyone will be using this website? :)

To be honest: this idea and website is exactly what i am looking for for years now. Up until now i have been using several applications to keep track of my data from everything eve related, but once a certain tool or website stops updating and becomes obsolete when eve gets a new expansion and the developers behind the tool or website for whatever reason do not want to work on the application than my data is pretty much worthless after one, or a few expansions.

I really hope this would resolve that problem, and that off course that your website will stay online in length of years, perhaps even as long as eve will exist....? :P

I am always in the mood to test out a new application, especially a well thought out one like this one! Sign me (Dunpeal Hunter), and my separate industry corp up for a beta key :)


Thanks for all the kind comments. I plan to stick it out as long as I can. Since this is all on Google Cloud, it's fairly easy to turn this over to someone else if I can't hack it anymore. It's also possible CCP will get very ambitious with CREST, in which case you might not need EveKit anymore :)

P.S. In case it wasn't clear from my post, you don't need to do anything special to participate in the beta. Just log into the site (using google or twitter) and add the account(s) you want to sync.

Use EveKit ! - Tools for EVE Online 3rd party development

Myopic Thyne
Accounts Payable.
#8 - 2014-07-23 18:52:28 UTC
The thought that comes to my mind is that this very quickly starts heading towards being a way to verify that alts belong to someone. This is actually pretty awesome. Using current systems you can weakly do this by asking them to give you a specific key (since these are editable) but beyond that, with SSO, the only way you should have access to an account is if you have the credentials... that's huge.
Neroo Tal
Order of the Golden Dawn
#9 - 2014-07-25 18:04:36 UTC
So you mention you can put metadata on there. How.. am I missing the obvious here? Navigation as is pointed out in the FAQ's is a pain to say the least especially without a list of current plates available.

Or is the metadata you put after you dump the char data and do it that way?
salacious necrosis
Garoun Investment Bank
Gallente Federation
#10 - 2014-07-25 23:44:21 UTC
Neroo Tal wrote:
So you mention you can put metadata on there. How.. am I missing the obvious here? Navigation as is pointed out in the FAQ's is a pain to say the least especially without a list of current plates available.

Or is the metadata you put after you dump the char data and do it that way?


Yeah, I'm seriously considering nuking the current navigation system. If the site had more content it might make sense, but not now. Anyway, you can get a list of all available plates by typing a "space" into the navbar and letting command completion show you the list.

You add meta data through one of the APIs (e.g. web, java or javascript), or you can try it out on the api "plate". Here's an example using the api plate to show how to tag an asset with meta-data:

1) Log in if you haven't already. You can visit the api plate without being logged in, but you won't be able to try out any calls unless you're logged in.
2) Go to the api plate (type "api" into the navbar).
3) Click on "Capsuleer" under API on the left (you can also use "Corporation" depending on what type of account you have).
4) Now find the meta ID of the asset to tag. You can do this by selecting "Asset" on the left, then "list". The form on the right should fill in with one of your access keys. Enter -1 for continuation and something like 10 for maxresults, then click "SUBMIT". Pick an asset of the result of the call and note it's meta ID.
5) Now select "Meta" in the API panel on the left, then "set".
6) Fill in the API form with the meta ID and a key/value pair. Keys and values are strings with a max size of 255. Hit "SUBMIT" to save the assignment.
7) You can view meta-data by selecting Meta->list, then filling in the form appropriately (don't forget to hit SUBMIT).

Hmm, I wonder if it's worth making a vid or other tutorials for this sort of thing.

Use EveKit ! - Tools for EVE Online 3rd party development

salacious necrosis
Garoun Investment Bank
Gallente Federation
#11 - 2014-07-30 04:01:33 UTC
2014-07-29 EveKit Beta Status

Updates:

- Beta release has been active for about a week now, thanks to all participants! Interesting stats below...
- Static DB has been upgraded to the Crius release!
- Participants have uncovered numerous bugs which were fixed during the week. Thanks again!

Next Steps:

- I encourage everyone to give the API section a try. This "plate" lets you test out the API against your sync'd data. You can reach the API section using this link.
- So far so good with the load but there are a few synchronization issues I'd like to fine tune.
- Menu/navigation redesign is on the short list. In the mean time, thanks for bearing with the somewhat unorthodox navigation scheme.
- I assume everyone would appreciate a vid or two showing how to use some of the features. I'll try to cobble something together next weekend.
- I'd like to convert an existing EVE app to use EveKit as the backend in place of some proprietary local store. Please let me know if you have any suggested apps. Going through this exercise will likely drive changes to the API, which is a good thing.

Interesting stats:

- 10 users are participating in the beta so far.
- 13 separate EVE accounts are synchronizing on an automatic basis.
- 3828 account synchronizations have been performed in the last week (about 550/day)
- Average sync time was 260 seconds.

Use EveKit ! - Tools for EVE Online 3rd party development

Elisa-Verena Elestair
Garoun Investment Bank
Gallente Federation
#12 - 2014-08-02 11:26:35 UTC
Hello there.

I wanted to try your EveKit with a corp key of my corp. So I got this key from my CEO, which is working in EveHQ.
But in EveKit it looks like the API isn't updating. Tried it manually first, then set it to auto-update.
I put one of my account api's there and that one updated soon...

So maybe there's a bug or you didn't put that function in it yet
salacious necrosis
Garoun Investment Bank
Gallente Federation
#13 - 2014-08-02 15:12:40 UTC
Elisa-Verena Elestair wrote:
Hello there.

I wanted to try your EveKit with a corp key of my corp. So I got this key from my CEO, which is working in EveHQ.
But in EveKit it looks like the API isn't updating. Tried it manually first, then set it to auto-update.
I put one of my account api's there and that one updated soon...

So maybe there's a bug or you didn't put that function in it yet


The database took a bit of a holiday earlier this morning, slowing everyone down, but we're back to normal now.

If you created your corp account around 9:44 UTC on 8/2 then you're syncing now but it's a bit slow, I'm guessing this is a decent sized corp. The first sync happened during the DB issues and took around 3 hours to sort itself out. The second sync took about an hour. The third sync just started so I'll keep an eye on it. Typically, the first few syncs are slow as the site builds up wallet history, etc (e.g. lots of DB commits). Subsequent commits should go faster.

As far as sync's go, the site should be feature complete.

UPDATE: and as I was about to hit 'post', your latest sync just completed. It took about 7 minutes.

Use EveKit ! - Tools for EVE Online 3rd party development

salacious necrosis
Garoun Investment Bank
Gallente Federation
#14 - 2014-08-06 02:31:51 UTC  |  Edited by: salacious necrosis
2014-08-05 EveKit Beta Status

Updates:

- Two weeks into the beta, thanks again to all participants!
- We had a DB issue the morning of 8/2. No data was lost, but syncs were very slow (screwed up our stats!)
- Menu has been vastly simplified to make navigation much easier.
- Static DB has been upgraded to the Crius 1.6 release.

Next Steps:

- I've added a section for guides and videos but it's empty at the moment. I'll add a few videos this weekend.
- I'm experimenting with different sync scheduling mechanisms. Sync time will likely be a bit variable over the next week.

Interesting stats for the last 7 days:

- 14 users are participating in the beta so far.
- 18 separate EVE accounts are synchronizing on an automatic basis.
- 5891 account synchronizations have been performed in the last week (about 840/day)
- Average sync time was 5.8 minutes, min was 1 minute, max was 207 minutes (during a DB issue on 8/2).

Use EveKit ! - Tools for EVE Online 3rd party development

Elisa-Verena Elestair
Garoun Investment Bank
Gallente Federation
#15 - 2014-08-08 17:42:51 UTC  |  Edited by: Elisa-Verena Elestair
salacious necrosis wrote:


The database took a bit of a holiday earlier this morning, slowing everyone down, but we're back to normal now.

If you created your corp account around 9:44 UTC on 8/2 then you're syncing now but it's a bit slow, I'm guessing this is a decent sized corp. The first sync happened during the DB issues and took around 3 hours to sort itself out. The second sync took about an hour. The third sync just started so I'll keep an eye on it. Typically, the first few syncs are slow as the site builds up wallet history, etc (e.g. lots of DB commits). Subsequent commits should go faster.

As far as sync's go, the site should be feature complete.

UPDATE: and as I was about to hit 'post', your latest sync just completed. It took about 7 minutes.



Thanks, just need some spare time to get more into it. I really shouldn't do so many different things at once :D



EDIT:

Hmm, somehow I'm just getting 20 entries out of https://www.eve-kit.org/access/common/Asset/list?accesskey=********&accesshash=****************************************

Should be a few thousand at least :D
And I already tried to set the maxresults to a high number, still returning 20 entries
salacious necrosis
Garoun Investment Bank
Gallente Federation
#16 - 2014-08-10 02:15:03 UTC
Elisa-Verena Elestair wrote:
salacious necrosis wrote:


The database took a bit of a holiday earlier this morning, slowing everyone down, but we're back to normal now.

If you created your corp account around 9:44 UTC on 8/2 then you're syncing now but it's a bit slow, I'm guessing this is a decent sized corp. The first sync happened during the DB issues and took around 3 hours to sort itself out. The second sync took about an hour. The third sync just started so I'll keep an eye on it. Typically, the first few syncs are slow as the site builds up wallet history, etc (e.g. lots of DB commits). Subsequent commits should go faster.

As far as sync's go, the site should be feature complete.

UPDATE: and as I was about to hit 'post', your latest sync just completed. It took about 7 minutes.



Thanks, just need some spare time to get more into it. I really shouldn't do so many different things at once :D



EDIT:

Hmm, somehow I'm just getting 20 entries out of https://www.eve-kit.org/access/common/Asset/list?accesskey=********&accesshash=****************************************

Should be a few thousand at least :D
And I already tried to set the maxresults to a high number, still returning 20 entries


That's because each API call has a max results limit and I just noticed some calls are really stingy. I'll up the max batch sizes to something more reasonable.

Anyway, you can get all the results by passing in the continuation ID. When you make a call to asset list (and many other calls), you'll see results like this:

Quote:

{
"eveKitBuild": "4fbd8c5815e833f349bbb03b35cffc87ee91c805-20140809193906 (achonee)",
"responseTime": 1407636489581,
"continuation": "708",
"result": [
{
"metaID": 699,
"itemID": 1005100417883,
"locationID": 60001357,
"typeID": 24698,
"quantity": 1,
"flag": 4,
"singleton": true,
"rawQuantity": -1,
"containerID": -1
},


If the value in the continuation field is not "-1", then there are more results. You can get the next batch of results by making another call with the given continuation value. In the example above, "708" is the metaID of the last asset in the current batch. When you make the call again with "708" in the continuation field, you'll get the next batch of assets with metaID greater than 708 (but only the next 20 until I up the limits). You can keep iterating this way until the continuation field shows "-1".

There's no "cursor" here in the database sense either. If you happen to know the metaID you want to start at, you can just put that in on the first call to the API.

Use EveKit ! - Tools for EVE Online 3rd party development

salacious necrosis
Garoun Investment Bank
Gallente Federation
#17 - 2014-08-10 19:17:41 UTC
Ok, I just bumped up all the default API batch sizes to 1000 entries (if maxresults < 1, then this default is used). You can explicitly select a smaller batch with the "maxresults" parameter. If you have more than 1000 entries, you'll need to make another call passing the continuation parameter.

Use EveKit ! - Tools for EVE Online 3rd party development

Jeronica
Tackled In Belt
xXPlease Pandemic Citizens Reloaded Alliance.Xx
#18 - 2014-08-11 04:21:19 UTC
This looks like a great idea! The added meta-data to certain items seems like it would be very useful to some people.

Just curious, but this statistic here: "- Average sync time was 260 seconds." Was this for one account? Or for all 13 accounts you have.

EVE-MOGUL.COM

Trade Profit Tracking&Analytics

Offering Sotiyo Services In

New Caldari | Ashab

IPOs & Investments

salacious necrosis
Garoun Investment Bank
Gallente Federation
#19 - 2014-08-11 23:22:48 UTC
Jeronica wrote:
This looks like a great idea! The added meta-data to certain items seems like it would be very useful to some people.

Just curious, but this statistic here: "- Average sync time was 260 seconds." Was this for one account? Or for all 13 accounts you have.


That's an average time for sync'ing one account. The times vary a bit depending on the type of account (active corporations take much longer to sync). There are also occasional disruptions in Google's cloud, a pretty nasty one has been going on for almost 24 hours now where I see significantly more DB connection drops than typical.

Here's what I'm seeing over the last week or so (these are for sync'ing one account):

min: 0.23 minutes
avg: 5 minutes
max: 136 minutes
99%:40 minutes
95%: 20 minutes
median: 1.93 minutes

This is sync'ing 22 accounts, 10 of which are for corporations. For corporations, the breakdown looks like this:

min: 0.28 minutes
avg: 6.23 minutes
max: 136 minutes
99%: 51 minutes
95%: 25.5 minutes
median: 2.5 minutes

and for characters:

min: 0.23 minutes
avg: 3.93 minutes
max: 89 minutes
99%: 29 minutes
95%: 16.5 minutes
median: 1.6 minutes



Use EveKit ! - Tools for EVE Online 3rd party development

Elisa-Verena Elestair
Garoun Investment Bank
Gallente Federation
#20 - 2014-08-12 05:15:53 UTC
salacious necrosis wrote:


That's because each API call has a max results limit and I just noticed some calls are really stingy. I'll up the max batch sizes to something more reasonable.

Anyway, you can get all the results by passing in the continuation ID. When you make a call to asset list (and many other calls), you'll see results like this:

Quote:

{
"eveKitBuild": "4fbd8c5815e833f349bbb03b35cffc87ee91c805-20140809193906 (achonee)",
"responseTime": 1407636489581,
"continuation": "708",
"result": [
{
"metaID": 699,
"itemID": 1005100417883,
"locationID": 60001357,
"typeID": 24698,
"quantity": 1,
"flag": 4,
"singleton": true,
"rawQuantity": -1,
"containerID": -1
},


If the value in the continuation field is not "-1", then there are more results. You can get the next batch of results by making another call with the given continuation value. In the example above, "708" is the metaID of the last asset in the current batch. When you make the call again with "708" in the continuation field, you'll get the next batch of assets with metaID greater than 708 (but only the next 20 until I up the limits). You can keep iterating this way until the continuation field shows "-1".

There's no "cursor" here in the database sense either. If you happen to know the metaID you want to start at, you can just put that in on the first call to the API.



Ok I see my failure. I tried it without continuation & maxresults and so I always got just 20 results returned. But now with set maxresults it's a long list, like it should be, thanks.
12Next page