EVE Technology Lab

I've written a platform for building killboards and other killmail related applications. It's still in the alpha stage however. Before it goes beta there needs to be a few hundred automated unit tests in place to ensure that as many defects as possible are weeded out and regressions are prevented as development progresses.

Project: pykb

The database is isolated from the code and can be swapped out entirely. I've included the DDL for both PostgreSQL and MySQL illustrating this. Simply edit a config file to switch between the two once you have the databases installed. There's no hard coded SQL aside from a stored function call.

There are aggregate tables containing summary data at the daily, monthly and yearly levels by character, corporation and alliance along with the same for ship type kill and loss statistics. These are maintained via an aggregation function that needs to be set up to run once a minute - only aggregates associated with inserted, deleted or updated kills are recalculated. These aggregates are of course optional and can be ignored of you have no need for them.

Killmails copied from the client, including those with German and Russian keywords can be processed along with feeds from EDK based sites and the EVE API given the appropriate keys.

To do:
- Add in support for uWSGI caching to supplant the built-in persistent cache that's only really useful for single process apps. uWSGI Caching
- Add support for user management / logins / password encryption using py-bcrypt
- Build an API Key kill log fetching task with auto-disabling of keys that fail.
- Add in loads of unit tests and fix whatever defects remain.
- See what more can be done to speed up MySQL aggregation function.

I will start up a new project soon as well to create a killboard and analysis app on top of this using
Flask, Mako, uWSGI, Nginx with Varnish and a CDN - most likely CloudFlare

Looking into this for real time graphs: Graphite Need to see if it'll work with uWSGI instead of mod_python / Apache. I'd like to provide the capability to see data graphically at a high level and enable drilling down further to see details.

Lastly add in support for high speed searches using Sphinx and automated documentation for both projects using Sphinx same name - different products.

Probably easier to grab the latest source instead and browse it locally:

hg clone http://hg.code.sf.net/p/pykb/code pykb

Good to hear. I just finished updating the database schemas and data to adjust for the latest CCP changes. I found a defect last night and checked in a fix shortly afterwards so get the latest from the trunk.

If you could, simply dump data from your existing killboard into either the MySQL or PostgreSQL databases and see if you can spot any discrepancies. Easiest way to do that is to use the backfill.py script to read data from your existing EDK system if you used their code via idfeed starting at id number 1.

All of the scripts support the standard -h --help arguments for options. I'll add a -u option to enable specifying a url on the command line too for the backfill.py script however for now just do this:

insert into killboards(url, pause_secs)
values('http://www.yoursite.com/?a=idfeed', 1);

It'll grab 50 kills at a time, process them, wait a second and fetch the next set, starting from the oldest.

Just run this with no arguments: backfill.py

Note: If your data is hosted on EVE-Kill.net do not set the pause_secs value, just leave it with the default of 1800 secs (30 minutes) as they recommend on their idfeed page to avoid putting too much load on their site. It'll take forever to get your data out however....but if you set it lower they might get mad at you and block your ip address.

To view the results you can use dumpmails.py to extract data back into human readable combat log kill mail format or use the handy views defined in the database to make is easier to see the data as opposed to having to figure out how to join everything.

Thanks Kalstrams. I've started on the web site code using Flask and Mako. It'll serve as an example implementation to show how to leverage the pykb platform for a killboard app. Hardest part so far is getting the CSS to work properly - using the Blueprint CSS framework as a starting point and tweaking it to make kill details match the new in-game kill report fairly closely.

Anyway, once I have a working, fully function killboard, hosted on a VPS - probably Linode, I'm going to focus further on getting automated tests written for the library code to start with so that others can add to it and run the tests, as well as add their own tests and ensure that everything still works 100%.

Here's a screen shot showing a functional kill report web page - Kill Report Here's the corresponding entry on EVE-Kill.net: Kill Details So next I'll add in the points value and API Verified (or not) indicator.

Of course I'll have to include the CCP 3rd party site copyright notice on the bottom as well before release since the images and overall layout / design are theirs.

At any rate, to be clear, the code for the web portion of this is a separate project from pykb, so far much smaller consisting of an 80 line python script and a 200 line html file plus some css and a config file. I'll create a project on sourceforge for this once I have enough worth checking in. The pykb project provides the models and controllers - the backend platform to build the views upon - in this case a web site.