EVE Information Portal

I'm a bit befuddled over the choice of YAML but hey, if it works for you guys then that's great. I know that a lot of guys where I work are moving a lot of our configs over from a simple key=value format to YAML.

I'm not sure I would want it for the massive data sets that Eve uses, but it's a standard format, it isn't XML (which is good and bad), and it's very readable. So, what'ev. : > I'm just happy that CCP is awesome enough to provide us with the data in the first place.

Yes, but why? In what way is the data changing such that YAML is the most appropriate format, and/or in what way is YAML the most appropriate format for the existing data?

There may well be a very good reason. I'm just interested in hearing it.

I figure I'll just answer some questions in an incomprehensible techy way.

As an organization, CCP has decided that we benefit from developers being able to work in branches (in a Perforce sense), and working in branches eventually means that you need to be able to change your data with your code and not affect other people. Binary formats and DBs can be perfect for the run-time data requirements that you have, but they're sucky from the point of view of understanding and merging data as a frail human doing an integration. So we want our data in files, and we want to be able to merge them.

So why not CSV? Yes, it would seem to be more appropriate to some of the data we've shown off so far, but at some point you also have to realize that the reason your data is tabular is because you've been storing it in a tabular storage medium. It's also really sucky to deal with foreign key relations between text files. As we progressively migrate more data, and deal with things like moons being the children of planets, we can decide to represent that in the structure of our data.

And why not JSON? Well, JSON is spiffy and all, but we already use YAML as an institution (if you've ever looked at our .red files for our ship assets etc), and it can deal with some things much more nicely than JSON (we use YAML reference support for some things). JSON somewhat suffers for having been built for a language that hasn't had a standard concept of a map until ECMA script 6 (stringified attribute names on objects just don't count, and this issue carries over into the proposed JSON schema validation standard).

We can output our static data as JSON, it's just not what we want to work with. We asked about this at the fanfest, but I think the detail was probably missed in the noise of the orbital bombardment

So we wanted to use YAML, and a no-sql-ish document-like data setup, but this isn't really appropriate for the runtime. In fact, we don't use it for the runtime... we use a structured binary format that's built from the data (think MessagePack), attempts to minimize memory overhead and disk seek operations. In some cases, we might even use Sqlite (woo, standard python library!) as appropriate for the use-case.

The end result of what we're trying to achieve is better runtime memory overhead performance, with an easier time for our developers to add / remove and change data formats in a human-understandable base format that can be versioned with changes to the code that are associated with it and isolated in branches. Some of the data will get built straight back into relation tables in MS-SQL, but some of it will just sit in a structured format that means that all related data to a particular "thing" is just accessible right there without subsequent join operations or lookups being required. We get to be able to sync our source control and get our data as it was at that point.

An important thing for us is to just try this and see if it can work for us, which is why we're starting so small. Beyond the changes to the data are much bigger changes to our tools and methods of working, not so much based on what the format is, but more on the issue of the data being local, isolated and in files rather than a central authoring DB.

I would suggest that the best long-term solution *might* be to look at NoSQL type databases, of which there are a number of free and very good options, or you can choose to try and maintain scripts that process our data into relational structures.

Thanks CCP Redundacy,

We too are dealing with the difficulties of maintaining some kind of versioning system with DDL and static data; it is really a problem that doesn't have a good solution yet.

I can understand going to files that are easily parseable by CVS/Git/Whatever. It also allows people to check out the files and make their own changes locally without affecting the test database, and without having to load a fresh database copy into their local test environment every time they need something.

VS2010 Premium actually has some good SQL versioning capabilities when used with Team Foundation Server, but I have to say that I'm intrigued by CCP's approach here. Kinda the best of both worlds, in a certain sense (for you guys, a bit less so for us). A bit unwieldy, for multiple join type queries, but that stuff can be handled in code instead of in the database, I suppose.

Cheers. To clarify, does CCP have any plans to add structures which can't easily be represented in a database? (I'm having difficulty imagining what these might be, but YAML can do a lot.)

If there are any performance issues with YAML in third-party applications, I'm sure someone over at the Technology Lab will come up with a more useful package. (Something similar to the binary format that CCP Redundancy mentioned?)

[EDIT]


This, please, more of this! I've recently come back to EVE after playing some other in-development games, and it's refreshing to once again chat with devs who respect the player base and are themselves respectable.

I don't recommend YAML for anything where you worry about performance. Check out MongoDB (BSON) or MessagePack as a starting point (also NoSQL in general is an interesting thing to play with if you've been all-relational, but I won't pretend that it's a good solution to everything). If you need to use YAML, make sure you're using a native parser at least (pyYAML + libYAML, for example).

We'll be sticking to lists and dicts and nested objects (pretty much JSON), and mainly focusing on working out how to convert our existing datasets (that are already in the DB) to this sort of thing without screwing things up for everyone at CCP. Python is very handy at working with this sort of data, so I personally recommend that for transforming it to whatever format you prefer.

This sort of structure: { 1: ['a', 'cat'], 2:['two','dogs'] } is a pain in the ass in a DB... do-able, but I don't want to insist that people build a relational version unless they need to.

..*slow clap*...

Alright, you've convinced me. A unit of Spirits to you!

Bout time. That denormalized table inside SQL made my soul hurt.

AND

I never have to install MSSQL ever again.

With web-based stuff you can cache it either using a nosql approach like mongo, or something sane people use like memcached. If you think about it hard enough, you realize that most data you're pulling from CCP should likely be in a cached state rather than pinging the database for it or parsing it anyway.