EVE Technology Lab

After a month or so of furious updates and tuning, I'm happy to announce that EveKit - a site for automatically storing your EVE API data to the cloud - is now out of beta. We first announced the site with this post. A big thank you to the beta testers that created accounts. These accounts were extremely valuable in helping us tune the performance of the site.

Most of the recent work has focused on stability and performance of the backend (see the impressive stats below). These are the highlights:

• We've abandoned the MySQL backend in favor of the native data store on Google App Engine. This allows us to eliminate much of the serialization (and therefore slower performance) that was occurring on the backend.
• All account data transferred without issue but we're still in the process of transferring historical data for accounts over to the new data store. As of this writing, we're about halfway done. New data has been syncing for almost a week now.
• We've stopped providing account snapshots as MySQL dumps. Instead, when you request an account snapshot you'll get a zip of CSV files with your data. These are easy to import into a database if that's your desire.
• Not that anyone likely cares, but the EVE static data we provide is still backed by MySQL. This will change in the future as the data is completely static and could be delivered more efficiently from local files.
• Lastly, we've revamped and simplified the Web UI. We expect to post a video sometime this weekend covering the basics of the site.

What's next? Honestly, we're looking for suggestions here. We'll post a call for feature ideas in another thread. Here's what we have for next steps in the short term:

• Add support for the "planetary" EVE APis. We don't yet download this data.
• Consider adding support for sync'ing SiSi data. This is valuable to us for testing new API features, not sure if anyone else cares about their SiSi data.
• Consider exposing a Graphite API to make it easier to build dashboards.

Finally, here are some stats showing the massive performance improvements we've been able to achieve. These are aggregate stats, meaning combined over both character and corporation accounts.

Aggregate Sync Stats Before Improvements:

Average sync time: 4.4 MINUTES
Min: 0.48 MINUTES
Max: 69.3 MINUTES
99%: 19.7 MINUTES
95%: 10.2 MINUTES
Median (50%): 3.6 MINUTES

Aggregate Sync Stats After Improvements:

Average sync time: 43 SECONDS
Min: 8.2 SECONDS
Max: 6.2 MINUTES
99%: 2.8 MINUTES
95%: 1.6 MINUTES
Median (50%): 27 SECONDS

It's still true that the first sync of a large corporation is a little painful, but once the caches are warmed up, syncs go very fast.