These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Proposed Changes to the Static Data Export (SDE)

First post First post
Author
Previous Topic Next Topic
Matt Faithbringer
YOLO so no taxes please
#21 - 2016-02-16 16:19:17 UTC
CCP Tellus wrote:
Zifrian wrote:
If I read your post correctly, it sounds like you want to add just two more files to yaml?

I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files.

One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour.

There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes.

This may not result in the prettiest YAML files, at least not until we migrate more data over to FSD internally. This approach is, in my opinion at least, the simplest and least intrusive method of achieving the stated goals.


I'm not big fan of YAML, but having everything in one format would be awesome, even when that format is YAML.
Golden Gnu
Lobach Inc.
#22 - 2016-02-22 15:46:15 UTC
For me it doesn't matter directly.
I will probably always use the conversion and/or tools by Desmont McCallock or Steve Ronuken.
However, if you make their job easier, you're doing a good job.


tl:dr
I support whatever Desmont McCallock and Steve Ronuken says Lol

Creator of jEveAssets - the asset manager

"Download is the meaning of life, upload is the meaning of intelligent life"
Mr Mac
Dark Goliath
#23 - 2016-04-25 15:34:44 UTC
Patch notes 2016-04-25
Quote:
Static Data Export (SDE):

  • The universe data and the SQL Server Backup file have been converted to YAML. The SDE now consists solely of YAML files.
    • Desmont McCallock
    #24 - 2016-04-25 15:39:33 UTC
    Mr Mac wrote:
    Patch notes 2016-04-25
    Quote:
    Static Data Export (SDE):

  • The universe data and the SQL Server Backup file have been converted to YAML. The SDE now consists solely of YAML files.

    • This is HUGE news. We need a pre-release asap.
    Pete Butcher
    The Scope
    Gallente Federation
    #25 - 2016-04-25 18:20:51 UTC
    mapDenormalize too? Holy crap, this must be a 1GB of yaml.

    http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
    Steve Ronuken
    Fuzzwork Enterprises
    Vote Steve Ronuken for CSM
    #26 - 2016-04-25 19:19:41 UTC
    https://cdn1.eveonline.com/data/sde/singularity/sde-20160416-2016-DANIELLE.zip is the prerelease version which has been floating round Slack. (be aware there's a couple of dodgy numbers in the blueprint data. it's not an export problem, just a problem with the source, which should now be fixed)

    I've got everything but the universe data loaded easily. Universe data shouldn't be too hard, but it'll need a chunk of processing, to make it look like the old data.

    (Fun story: mapDenormalize isn't how they store it at CCP. This is a cut from their versioning system.)

    Biggest differences so far:

    All the tables which were in the sql server export are now in single yaml files. loading them is just a case of iterating through the array and loading them back into a table. some tables are a touch troublesome due to the number of entries (invNames, for example), but most load just fine. ( https://github.com/fuzzysteve/yamlloader )

    Woo! CSM XI!

    Fuzzwork Enterprises

    Twitter: @fuzzysteve on Twitter
    Desmont McCallock
    #27 - 2016-04-25 21:36:22 UTC  |  Edited by: Desmont McCallock
    Judging from Steve's linked SDE, universe data is split down to each individual solarsystem, which makes mapDernomalize table creation a process 3rd party tools have to undertake.

    BTW, thank you Steve for the link.
    CCP FoxFour
    C C P
    C C P Alliance
    #28 - 2016-04-25 21:48:44 UTC
    Just for clarity, we plan on doing both the old and new style SDE for a few releases since we know it will take time for people to convert. This is why we didn't worry to much about putting out a pre-release version since we fully intended on doing both anyways for a bit.

    @CCP_FoxFour // Technical Designer // Team Tech Co

    Third-party developer? Check out the official developers site for dev blogs, resources, and more.
    Desmont McCallock
    #29 - 2016-04-25 22:11:43 UTC
    CCP FoxFour wrote:
    Just for clarity, we plan on doing both the old and new style SDE for a few releases since we know it will take time for people to convert. This is why we didn't worry to much about putting out a pre-release version since we fully intended on doing both anyways for a bit.
    Feeewww. You got me worried there for some time.
    CCP Tellus
    C C P
    C C P Alliance
    #30 - 2016-04-26 13:07:47 UTC
    Desmont McCallock
    #31 - 2016-04-26 14:23:10 UTC  |  Edited by: Desmont McCallock
    @CCP Tellus
    Questions:
    1. Will this be the naming format for the SDE filename from now on?
    2. Where do we find the 'translationTables' table? Do we have to create it on our own?
    3. Would you mind fixing the duplicate entries in blueprints.yaml file? (blueprintTypeID: 41590 has typeID: 38 for manufacturing specified two (2) times with different quantity)
    CCP Tellus
    C C P
    C C P Alliance
    #32 - 2016-04-26 14:45:14 UTC
    Desmont McCallock wrote:
    1. Will this be the naming format for the SDE filename from now on?

    Is there any particular reason you're relying on the filename format remaining consistent? It may or may not change in the future.

    Desmont McCallock wrote:
    2. Where do we find the 'translationTables' table? Do we have to create it on our own?

    The 'translationTables' table was an internal look-up table used to construct the 'trnTranslations' and 'trnTranslationColumns' tables. It had not gotten deleted after the SDE export script ran. Do you know of a use-case for that table?

    Desmont McCallock wrote:
    3. Would you mind fixing the duplicate entries in blueprints.yaml file? (Can provide exact info if needed)

    I can take a look at blueprints.yaml. Can you send me the details?
    Desmont McCallock
    #33 - 2016-04-26 14:53:18 UTC  |  Edited by: Desmont McCallock
    CCP Tellus wrote:

    Is there any particular reason you're relying on the filename format remaining consistent? It may or may not change in the future.
    In my tool I have an algorithm that tries to detect if the user has dropped in the specific folder the zipped file or (s)he has extracted the content of the zipped file. If a zipped file is detected the tool unzips the content automatically. The algorithm does a match search depending on the file name (we don't want to unzip any zip file but only the SDE one)
    CCP Tellus wrote:

    The 'translationTables' table was an internal look-up table used to construct the 'trnTranslations' and 'trnTranslationColumns' tables. It had not gotten deleted after the SDE export script ran. Do you know of a use-case for that table?
    Mainly for backwards compatibility. I haven't ran into a situation nor seen anyone talking about that table, but you know how things goes. Remove something and someone will complain about it.
    CCP Tellus wrote:

    I can take a look at blueprints.yaml. Can you send me the details?
    Details already provided via editing my OP.
    Salvoxia
    Free Carpenters Union
    #34 - 2016-04-26 14:55:12 UTC  |  Edited by: Salvoxia
    Hi,

    previous SDE releases always contained some kind of version or build number (for the last one it was 117575 for example).
    I always used it to reference the exact SDE release. Will there be something similar for the new release format or will you just use the release date?

    Best Regards,
    Salvoxia
    CCP Tellus
    C C P
    C C P Alliance
    #35 - 2016-04-26 15:55:08 UTC
    Desmont McCallock wrote:
    In my tool I have an algorithm that tries to detect if the user has dropped in the specific folder the zipped file or (s)he has extracted the content of the zipped file. If a zipped file is detected the tool unzips the content automatically. The algorithm does a match search depending on the file name (we don't want to unzip any zip file but only the SDE one)
    .

    You can use the following regular expression:
    sde-(.+).zip

    We will probably not change the file naming scheme, but we may change where we host static data exports in the future.

    Salvoxia wrote:
    previous SDE releases always contained some kind of version or build number (for the last one it was 117575 for example).
    I always used it to reference the exact SDE release. Will there be something similar for the new release format or will you just use the release date?

    Use the date and the branch name.
    Hel O'Ween
    Men On A Mission
    #36 - 2016-04-26 16:05:28 UTC  |  Edited by: Hel O'Ween
    CCP Tellus wrote:
    There's a new SDE release available on our developes' site:

    https://cdn1.eveonline.com/data/sde/tranquility/sde-20160426-TRANQUILITY-legacy.zip


    Is this upposed to be the official SDE release for the Citadel expansion. If so, ramTypeRequirements seems to have some PK issues:


    Reason was: Violation of PRIMARY KEY constraint 'ramTypeRequirements_PK'.
    A duplicate key can't be inserted into the dbo.ramTypeRequirements object.
    The duplicate key value is (41590, 1, 38).

    EVEWalletAware - an offline wallet manager.
    CCP Tellus
    C C P
    C C P Alliance
    #37 - 2016-04-26 16:11:45 UTC
    Hel O'Ween wrote:
    Is this upposed to be the official SDE release for the Citadel expansion. If so, ramTypeRequirements seems to have some PK issues:

    Am I missing something? I am not able to find a table called ramTypeRequirements.
    Steve Ronuken
    Fuzzwork Enterprises
    Vote Steve Ronuken for CSM
    #38 - 2016-04-26 16:17:39 UTC
    ramTypeRequirements is an old table. And it's probable that the 'bug' there is due to the conversion from yaml to the sqlserver db, with Desmont's code?

    Woo! CSM XI!

    Fuzzwork Enterprises

    Twitter: @fuzzysteve on Twitter
    Desmont McCallock
    #39 - 2016-04-26 16:37:47 UTC
    Yeah guys, it has to do with the duplicate entry in blueprints.yaml file I have already reported in this thread.
    Desmont McCallock
    #40 - 2016-04-26 16:43:39 UTC  |  Edited by: Desmont McCallock
    CCP Tellus wrote:
    Desmont McCallock wrote:
    In my tool I have an algorithm that tries to detect if the user has dropped in the specific folder the zipped file or (s)he has extracted the content of the zipped file. If a zipped file is detected the tool unzips the content automatically. The algorithm does a match search depending on the file name (we don't want to unzip any zip file but only the SDE one)
    .

    You can use the following regular expression:
    sde-(.+).zip

    We will probably not change the file naming scheme, but we may change where we host static data exports in the future.

    I would rather avoid adding another regex search but you're the Boss.

    Edit: Just for educational purpose here the regex expression I use to match the filename of all SDE releases so far.
    ".*_\d+\.\d+[\.\d+]*_\d+[_db]*\.zip|sde-\d+-\w+[-legacy]*\.zip"