These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
123Next pageLast page
 

Proposed Changes to the Static Data Export (SDE)

First post First post
Author
CCP Tellus
C C P
C C P Alliance
#1 - 2016-02-10 18:24:23 UTC
Hello everyone,

We know that some of you have been frustrated with late SDE releases and last-minute-changes in the past. Building and deploying the SDE has always been a manual process that had a tendency to slip through the cracks.

We want to build and ship the SDE to you all sooner, and more often alongside both Tranquility and Singularity releases. This would necessitate some changes to the SDE and its build process in order to achieve continuous delivery.

In the process of migrating static data from our legacy BSD content authoring tool to our new FSD authoring tool; we've been moving away from editing tables in a centralized SQL Server database, to YAML files tracked in revision control. This process is reflected by the SDE, having shipped both a SQL Server backup file and YAML files for some time now. We don't anticipate this migration to finish anytime soon.

One way of simplifying the build process can be achieved by naïvely converting the SQL tables to YAML files.

As an example, the last two entries of the eveUnits.yaml file would look like:
-
  unitID: 141
  unitName: "Hardpoints"
  displayName: "hardpoints"
  description: "For various counts to do with turret, launcher and rig hardpoints"
-
  unitID: 142
  unitName: Sex
  displayName: "1=Male 2=Unisex 3=Female"
  description: null


The first entry of staStations.yaml would look like:
-
  stationID: 60000004
  security: 0
  dockingCostPerVolume: 0.0
  maxShipVolumeDockable: 50000000.0
  officeRentalCost: 10000
  operationID: 26
  stationTypeID: 1531
  corporationID: 1000002
  solarSystemID: 30002780
  constellationID: 20000407
  regionID: 10000033
  stationName: "Muvolailen X - Moon 3 - CBD Corporation Storage"
  x: 1723680890880.0
  y: 256414064640.0
  z: -60755435520.0
  reprocessingEfficiency: 0.5
  reprocessingStationsTake: 0.05
  reprocessingHangarFlag: 4


In order to smooth out such a transition, we would of course ship the good o' SQL Server backup file and the SQLite universe data alongside these new YAML files for the next one or two SDE releases for Tranquility. That way you'd have time to update any tools you may be running on top of the Static Data Export.

What are your thoughts and opinions on this proposal?

Thanks!
Desmont McCallock
#2 - 2016-02-10 19:48:55 UTC
tl;dr Get done with the transition and give as one format and make it an MySQL dump one.

Since 2012, when the first YAML file appeared we are dealing with the splitting of the SDE to three (3) formats.
I'm with the impression, and correct me if I'm wrong, a developer would like to work with one format when it comes down to SDE.
And most like this format would be an MySQL dump.

From my perspective, I won't see anyone use the YAML and SQLite format in their projects, not until every table of the SQL database becomes a YAML or SQLite file. And even in that case, I bet all my ISKs on that, we will develop a tool to import those files either into the SQLite file, in order to have one format, or build a MySQL dump out of them.
Florian Lousberg
Furtherance.
#3 - 2016-02-10 20:22:21 UTC
I personally would welcome getting all the SDE tables in plain YAML on a more regular basis from an official source. For me it would be much easier to handle than waiting for FuzzySteves MySQL conversion (thanks for that, Steve! You got my vote!) and then converting that again into the custom Django database schemes I use in my projects.

Besides, the ability to just run a plain text diff against the previous version would be really nice, too.
Zifrian
The Pannion Domin
#4 - 2016-02-10 21:01:54 UTC
As much as I dislike yaml, I'd rather you just publish the entire SDE in yaml for a few cycles along with the SQL server file and then phase the SQL Server file out for good. Because what I'm going to do is convert the data into what I want to use and then take only the data I need for my app. It would be great if you provided a single DB format (SQLite like the universe data?) that we could use but I don't care as long it's all the same. If yaml works for you best, then do it in that.

If I read your post correctly, it sounds like you want to add just two more files to yaml? Many of us have been dealing with this hodgepodge system for 4 years now and I'd just like you to get on with it and do it instead of this piecemeal approach. If you are going to release a few files here and there, then that is 100xs more frustrating to me than the current system and I'd rather you do nothing instead.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder

CCP Tellus
C C P
C C P Alliance
#5 - 2016-02-10 21:10:02 UTC  |  Edited by: CCP Tellus
Zifrian wrote:
If I read your post correctly, it sounds like you want to add just two more files to yaml?

I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files.

One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour.

There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes.

This may not result in the prettiest YAML files, at least not until we migrate more data over to FSD internally. This approach is, in my opinion at least, the simplest and least intrusive method of achieving the stated goals.
Zifrian
The Pannion Domin
#6 - 2016-02-10 22:05:28 UTC
CCP Tellus wrote:
Zifrian wrote:
If I read your post correctly, it sounds like you want to add just two more files to yaml?

I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files.

One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour.

There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes.

This may not result in the prettiest YAML files, at least not until we migrate more data over to FSD internally. This approach is, in my opinion at least, the simplest and least intrusive method of achieving the stated goals.

OK that's great then. I'm glad it's going to be all at once. Many of us have wanted this for years and totally understand your reasons for doing it.

Will the universe data also be migrated from SQLite? Also, I get the impression that it would be easier to export the database prior to patch day? E.g. Citadel data before it is live?

Thanks for the update, looking forward to this.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder

CCP Tellus
C C P
C C P Alliance
#7 - 2016-02-10 22:22:30 UTC
Zifrian wrote:
Will the universe data also be migrated from SQLite?

The universe data is authored as YAML files internally. These are then post-processed into an SQLite database. Why? I am not sure exactly. I'm contemplating just removing that post-processing step.

Zifrian wrote:
Also, I get the impression that it would be easier to export the database prior to patch day? E.g. Citadel data before it is live?

That's the idea! Automatically building SDE exports for Singularity as soon as changes are deployed there shouldn't be much trouble either with these changes.
Max Kolonko
Caldari Provisions
Caldari State
#8 - 2016-02-10 22:51:17 UTC
As long as some wizard like steve will provide his service of converting them to something old dog like me can understand I'm fine :)
Dragonaire
Here there be Dragons
#9 - 2016-02-11 00:20:55 UTC
Not that I deal with the SDE directly to much but having everything in a single format should make dealing with it easier IMHO. Yaml is a good format for doing other conversions, from what I've seen and depending on the app it might be usable directly too. I would think Steve's conversion tool would also be easier to maintain and automate with just the one format.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#10 - 2016-02-11 00:57:13 UTC
Go for it.

My conversion will be a lot simpler.

And easily transferable to other people. (i.e., if you have a fairly basic python install, you'll be able to import it. Into any database that sqlalchemy supports.)


Granted, it's going to be one big import file (probably) and it'll take a while to import, but it's a one off job. And I can do it directly on the server.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Masao Kurata
Perkone
Caldari State
#11 - 2016-02-11 01:03:45 UTC
YAML as the one true format sounds great, doing conversions yourself is too painful otherwise.
Pete Butcher
The Scope
Gallente Federation
#12 - 2016-02-11 05:28:20 UTC
Do you plan to convert mapDenormalize too? That would be one big ass YAMLSmile

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool

Mr Mac
Dark Goliath
#13 - 2016-02-11 06:46:33 UTC
I like less YAML but having entire SDE in single format is fine.
Desmont McCallock
#14 - 2016-02-11 08:22:19 UTC
I agree with Steve. Our tools would have more meaning (to the linux savvies out there) if we could get rid of the SQL Server.
In that case I could even make my tool target .NET Core and have it running on every OS.

On another note, as Pete already mentioned, what about the mapDenormalize table? This will produce the largest YAML file. Do we have a size number so we know approximately how big it's going to be?
Evanova Android
Traquenard Labs
The Ancients.
#15 - 2016-02-11 13:13:05 UTC
I can only speak for myself but so far, the YAML files have been an inconvenience more than anything; a pre-built SQL database is much easier for me to manipulate and use. I would guess that many applications will want to build a database out of the YAML files anyway, just to be able to create complex views out of many files/tables.

Also, having to deal with it on a daily basis, I also dislike YAML very much and I think it is The Wrong Format for pretty much everything, but that's a personal and not very popular opinion.

Now, in all practical sense, if exporting YAML files makes the data set more consistent, easier to update and simple to use and build things out of for CCP and datameister Fuzzy, that cannot be a bad change.

Evanova - The Android App for Eve Online

BigSako
Aliastra
Gallente Federation
#16 - 2016-02-11 13:17:19 UTC  |  Edited by: BigSako
Why do we need yet another format? The SDE should be in the same format as CREST, and that is JSON.

JSON is easy to parse and you can directly use it in MANY applications, making (manual) converting unnecessary!

edit: I would also like to see an index of the SDE, showing two things per "table" (file):
Last Update + Update History / Diff
Pete Butcher
The Scope
Gallente Federation
#17 - 2016-02-11 14:13:35 UTC
While we're at it, it might be a good idea to finally fix integrity problems within the SDE itself. There are relations to non-existing entities.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool

Shala Raan
Trillionaire.pro
#18 - 2016-02-11 14:41:38 UTC
This looks nice. I use Steve's converions so if it helps him I'm all in Smile
Erick1111
Quebec's Underdog League
Quebec United Legions
#19 - 2016-02-11 18:43:22 UTC  |  Edited by: Erick1111
Having a one and only data format would certainly be a blessing. Then if we have conversions to make, we can make them internally. Although, some sort of data definition would be very much appreciated Lol
Matt Faithbringer
Tribal Liberation Force
Minmatar Republic
#20 - 2016-02-16 16:18:05 UTC
Desmont McCallock wrote:
tl;dr Get done with the transition and give as one format and make it an MySQL dump one.

Since 2012, when the first YAML file appeared we are dealing with the splitting of the SDE to three (3) formats.
I'm with the impression, and correct me if I'm wrong, a developer would like to work with one format when it comes down to SDE.
And most like this format would be an MySQL dump.

From my perspective, I won't see anyone use the YAML and SQLite format in their projects, not until every table of the SQL database becomes a YAML or SQLite file. And even in that case, I bet all my ISKs on that, we will develop a tool to import those files either into the SQLite file, in order to have one format, or build a MySQL dump out of them.


I don't agree here. Not everyone uses MySQL. Some people use sqlite, some people use postgres. IMO it's bad idea to go with one format. Just migrate the whole thing to YAML, everyone can read that.
123Next pageLast page