EVE Technology Lab

Hello everyone,

We know that some of you have been frustrated with late SDE releases and last-minute-changes in the past. Building and deploying the SDE has always been a manual process that had a tendency to slip through the cracks.

We want to build and ship the SDE to you all sooner, and more often alongside both Tranquility and Singularity releases. This would necessitate some changes to the SDE and its build process in order to achieve continuous delivery.

In the process of migrating static data from our legacy BSD content authoring tool to our new FSD authoring tool; we've been moving away from editing tables in a centralized SQL Server database, to YAML files tracked in revision control. This process is reflected by the SDE, having shipped both a SQL Server backup file and YAML files for some time now. We don't anticipate this migration to finish anytime soon.

One way of simplifying the build process can be achieved by naïvely converting the SQL tables to YAML files.

As an example, the last two entries of the eveUnits.yaml file would look like:

-
  unitID: 141
  unitName: "Hardpoints"
  displayName: "hardpoints"
  description: "For various counts to do with turret, launcher and rig hardpoints"
-
  unitID: 142
  unitName: Sex
  displayName: "1=Male 2=Unisex 3=Female"
  description: null

The first entry of staStations.yaml would look like:

-
  stationID: 60000004
  security: 0
  dockingCostPerVolume: 0.0
  maxShipVolumeDockable: 50000000.0
  officeRentalCost: 10000
  operationID: 26
  stationTypeID: 1531
  corporationID: 1000002
  solarSystemID: 30002780
  constellationID: 20000407
  regionID: 10000033
  stationName: "Muvolailen X - Moon 3 - CBD Corporation Storage"
  x: 1723680890880.0
  y: 256414064640.0
  z: -60755435520.0
  reprocessingEfficiency: 0.5
  reprocessingStationsTake: 0.05
  reprocessingHangarFlag: 4

In order to smooth out such a transition, we would of course ship the good o' SQL Server backup file and the SQLite universe data alongside these new YAML files for the next one or two SDE releases for Tranquility. That way you'd have time to update any tools you may be running on top of the Static Data Export.

What are your thoughts and opinions on this proposal?

Thanks!

I personally would welcome getting all the SDE tables in plain YAML on a more regular basis from an official source. For me it would be much easier to handle than waiting for FuzzySteves MySQL conversion (thanks for that, Steve! You got my vote!) and then converting that again into the custom Django database schemes I use in my projects.

Besides, the ability to just run a plain text diff against the previous version would be really nice, too.

I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files.

One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour.

There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes.

This may not result in the prettiest YAML files, at least not until we migrate more data over to FSD internally. This approach is, in my opinion at least, the simplest and least intrusive method of achieving the stated goals.

The universe data is authored as YAML files internally. These are then post-processed into an SQLite database. Why? I am not sure exactly. I'm contemplating just removing that post-processing step.


That's the idea! Automatically building SDE exports for Singularity as soon as changes are deployed there shouldn't be much trouble either with these changes.

Not that I deal with the SDE directly to much but having everything in a single format should make dealing with it easier IMHO. Yaml is a good format for doing other conversions, from what I've seen and depending on the app it might be usable directly too. I would think Steve's conversion tool would also be easier to maintain and automate with just the one format.

YAML as the one true format sounds great, doing conversions yourself is too painful otherwise.

I like less YAML but having entire SDE in single format is fine.

I can only speak for myself but so far, the YAML files have been an inconvenience more than anything; a pre-built SQL database is much easier for me to manipulate and use. I would guess that many applications will want to build a database out of the YAML files anyway, just to be able to create complex views out of many files/tables.

Also, having to deal with it on a daily basis, I also dislike YAML very much and I think it is The Wrong Format for pretty much everything, but that's a personal and not very popular opinion.

Now, in all practical sense, if exporting YAML files makes the data set more consistent, easier to update and simple to use and build things out of for CCP and datameister Fuzzy, that cannot be a bad change.

While we're at it, it might be a good idea to finally fix integrity problems within the SDE itself. There are relations to non-existing entities.

Having a one and only data format would certainly be a blessing. Then if we have conversions to make, we can make them internally. Although, some sort of data definition would be very much appreciated