These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Hyperion 1.0 SDE is out.

First post
Author
Laendra
University of Caille
Gallente Federation
#41 - 2014-08-09 23:25:33 UTC
Steve Ronuken wrote:
Why is data being put out in yaml files?

Because that's how they're storing it in their versioning system. Because it allows them easy diffing, to see when something was changed, and how it changed. As someone who supports a development team, I can see that's a good reason.


As for why they put it out as a backup file, well, that's just easy for them. Anything else would be extra effort.

As for 'standard SQL', heh. heh heh heh. hehehehehehehehehehehehehehehe. *keels over* Sure, there are SQL standards. The vast majority of databases only kind of stick to them.


Yes, but they are providing the SDE as a convenience for 3rd party developers...YAML isn't a convenience at all. It's pure crap, to be honest.

I am the technical lead developer of my team, and understand the value of having version control for what gets migrated between environments, but if you are going to provide data for other people to use, make it in a format that is easily used by a MAJORITY of the people that are going to want to use it. SQL was fine when it was in there, but there are a lot more standardized ways to share data...XML is old, but reliable enough for this....however, JSON is what is winning out across our LAN.

I wouldn't care if they removed the SQL database part of it, I create JSON data files for what I need out of it anyway....but something a little bit more standardized for public dissemination would sure go a long way to making a lot more people happy..(you know, the ones that haven't illegally hacked into the program/data to get their exports early)
Aineko Macx
#42 - 2014-08-10 06:52:27 UTC  |  Edited by: Aineko Macx
Steve Ronuken wrote:
As for 'standard SQL', heh. heh heh heh. hehehehehehehehehehehehehehehe. *keels over* Sure, there are SQL standards. The vast majority of databases only kind of stick to them.

Sure, databases have their own extensions and SQL dialects, but exports can be made quite ANSI compliant.

To place an analogy: It's akin to CCP releasing eve in icelandic and leaving the translation up to the players. Instead, the correctly recognize that icelandic is not the lingua franca of the internet.
The dumps of SQL Server are nowhere near any standard or common norm. If you need to pull the effort card as justification, it just shows once more the priority that 3rd party developer support has at CCP...
Codeguard
Codes Corp
#43 - 2014-08-10 16:44:35 UTC  |  Edited by: Codeguard
According to the database,
"Civilian Miner" (typeID=3651) contains -2566 Trtanium (negative value!)

select * from invtypematerials
inner join invTypes on (invtypematerials.typeID = invTypes.typeID)
where invtypematerials.quantity < 0

Funny thing, you can see the same in-game. It also has some funny secret China blueprint.
Paka-Tegat Birshiri
Paragon Material Extraction and Processing
#44 - 2014-08-12 16:19:01 UTC
Thanks for the conversion to MySQL, Steve. Question: Is there a set of release notes indicating which YAML files you rolled into non-standard tables?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#45 - 2014-08-12 18:40:10 UTC
Paka-Tegat Birshiri wrote:
Thanks for the conversion to MySQL, Steve. Question: Is there a set of release notes indicating which YAML files you rolled into non-standard tables?



umm, no. I should probably write something, but haven't.


blueprints.yaml is in the industryActivity tables

certificates.yaml is in the cert* tables

iconIDs.yaml is in eveIcons

typeIDs.yaml has:
the mastery information broken out into certMasteries
the trait information broken out into invTraits

There's additional information in there I've not broken out, as I've not needed it yet.




I've not touched graphicIDs.yaml yet. When/if I do, it'll probably go into a new table. Probably eveGraphics, to match the eveIcons table (which was a recreation of the table that it used to be in)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

salacious necrosis
Garoun Investment Bank
Gallente Federation
#46 - 2014-08-13 02:25:47 UTC
Steve Ronuken wrote:
Paka-Tegat Birshiri wrote:
Thanks for the conversion to MySQL, Steve. Question: Is there a set of release notes indicating which YAML files you rolled into non-standard tables?



umm, no. I should probably write something, but haven't.


blueprints.yaml is in the industryActivity tables

certificates.yaml is in the cert* tables

iconIDs.yaml is in eveIcons

typeIDs.yaml has:
the mastery information broken out into certMasteries
the trait information broken out into invTraits

There's additional information in there I've not broken out, as I've not needed it yet.




I've not touched graphicIDs.yaml yet. When/if I do, it'll probably go into a new table. Probably eveGraphics, to match the eveIcons table (which was a recreation of the table that it used to be in)


re: eveGraphics, that's the route I went, I just recreated the table. However, there are some new fields between versions which occasionally need to be added. I wrote a loader for the yaml which barfs when it finds a new field, at which point I go in and add the field and re-run the loader, etc. The graphics data is one of the easier files to handle.

Slightly off topic, but I have a handy little script that helps me get organized around migrating between DB versions. My script compares two DB versions (hosted on a MySQL server) and reports new tables, missing tables (i.e. tables removed in the latest version), and schema changes for common tables.

I've made the code available on GitHub for those interested.

Use EveKit ! - Tools for EVE Online 3rd party development

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#47 - 2014-08-13 12:01:05 UTC
salacious necrosis wrote:

Slightly off topic, but I have a handy little script that helps me get organized around migrating between DB versions. My script compares two DB versions (hosted on a MySQL server) and reports new tables, missing tables (i.e. tables removed in the latest version), and schema changes for common tables.


In case people are looking for another tool to do this kind of thing, the mysql utilities package has a bunch of useful tools in it, including one to compare DB schemas.

https://dev.mysql.com/downloads/utilities/

To compare schemas.
https://dev.mysql.com/doc/mysql-utilities/1.4/en/mysqldiff.html


To compare schemas, including at the data level.
https://dev.mysql.com/doc/mysql-utilities/1.4/en/mysqldbcompare.html

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

salacious necrosis
Garoun Investment Bank
Gallente Federation
#48 - 2014-08-13 13:35:51 UTC
Steve Ronuken wrote:
salacious necrosis wrote:

Slightly off topic, but I have a handy little script that helps me get organized around migrating between DB versions. My script compares two DB versions (hosted on a MySQL server) and reports new tables, missing tables (i.e. tables removed in the latest version), and schema changes for common tables.


In case people are looking for another tool to do this kind of thing, the mysql utilities package has a bunch of useful tools in it, including one to compare DB schemas.

https://dev.mysql.com/downloads/utilities/

To compare schemas.
https://dev.mysql.com/doc/mysql-utilities/1.4/en/mysqldiff.html


To compare schemas, including at the data level.
https://dev.mysql.com/doc/mysql-utilities/1.4/en/mysqldbcompare.html


NICE! Can't believe I missed those. Way less hacky than what I put together.

Use EveKit ! - Tools for EVE Online 3rd party development

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#49 - 2014-08-19 00:00:43 UTC
1.9 is out (oops. Missed this being released for around a week.)

Official release is on:
http://community.eveonline.com/community/fansites/toolkit/



Conversions:
Mysql:
https://www.fuzzwork.co.uk/dump/mysql56-crius-1.9-100804.tbz2

Sqlite:
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/eve.db.bz2

Postgres
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/postgres-crius-1.9-100804.dmp.bz2

Postgres dump is now done in postgres format, for restoration with pg_restore. excludes privileges.




Everything else:
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Derrick Miles
Death Rabbit Ky Oneida
#50 - 2014-08-20 07:05:26 UTC
Steve Ronuken wrote:
1.9 is out (oops. Missed this being released for around a week.)

You're slipping! Big smile

Thanks for the conversions.
Zifrian
The Frog Pond
Ribbit.
#51 - 2014-08-20 11:09:05 UTC
Desmont McCallock wrote:
Guys, let's face it, the entire SDE will be moved to YAML files. It was announces (for CCP standards) ages ago.

But when they changes all the universe data, they put it out as an SQLite file...not to mention that post about yaml was several years ago and we've seen little in the movement to it until Cirus.

I think expecting whatever hodgepodge system they want to use that day will be what we get. 3rd party dev support is a collateral anyway, we can't expect much when it's not their primary goal. We just have to be flexible.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder

Desmont McCallock
#52 - 2014-08-21 09:49:54 UTC
Zifrian wrote:
Desmont McCallock wrote:
Guys, let's face it, the entire SDE will be moved to YAML files. It was announces (for CCP standards) ages ago.

But when they changes all the universe data, they put it out as an SQLite file...not to mention that post about yaml was several years ago and we've seen little in the movement to it until Cirus.

I think expecting whatever hodgepodge system they want to use that day will be what we get. 3rd party dev support is a collateral anyway, we can't expect much when it's not their primary goal. We just have to be flexible.
Universe data where a little massive to extract to a YAML file and that's why they chose Sqllite. From my point of view they should had gone with Sqlite for the entire SDE, as it's another DB format anyway which wouldn't be much of a hassle to adjust our tools and apps to it.
salacious necrosis
Garoun Investment Bank
Gallente Federation
#53 - 2014-08-23 14:27:08 UTC
Steve Ronuken wrote:
1.9 is out (oops. Missed this being released for around a week.)

Official release is on:
http://community.eveonline.com/community/fansites/toolkit/



Conversions:
Mysql:
https://www.fuzzwork.co.uk/dump/mysql56-crius-1.9-100804.tbz2

Sqlite:
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/eve.db.bz2

Postgres
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/postgres-crius-1.9-100804.dmp.bz2

Postgres dump is now done in postgres format, for restoration with pg_restore. excludes privileges.




Everything else:
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/


Hi Steve,

Looks like you have dups in industryActivityRaces. Two for every row it seems. If that was on purpose then I don't understand your devious scheme Big smile

Cheers,
sal

Use EveKit ! - Tools for EVE Online 3rd party development

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#54 - 2014-08-23 14:40:07 UTC
salacious necrosis wrote:
Steve Ronuken wrote:
1.9 is out (oops. Missed this being released for around a week.)

Official release is on:
http://community.eveonline.com/community/fansites/toolkit/



Conversions:
Mysql:
https://www.fuzzwork.co.uk/dump/mysql56-crius-1.9-100804.tbz2

Sqlite:
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/eve.db.bz2

Postgres
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/postgres-crius-1.9-100804.dmp.bz2

Postgres dump is now done in postgres format, for restoration with pg_restore. excludes privileges.




Everything else:
https://www.fuzzwork.co.uk/dump/crius-1.9-100804/


Hi Steve,

Looks like you have dups in industryActivityRaces. Two for every row it seems. If that was on purpose then I don't understand your devious scheme Big smile

Cheers,
sal



Umm. I know exactly why that happened. oops.

(I used my intermediate schema as a place to create that table, and forgot to dump it before the most recent load. so it was loaded into twice)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Desmont McCallock
#55 - 2014-08-26 14:27:38 UTC
Hyperion 1.0 SDE is out. Let the conversions begin!
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#56 - 2014-08-26 14:54:02 UTC  |  Edited by: Steve Ronuken
Desmont McCallock
#57 - 2014-08-26 15:28:35 UTC
Just a heads up.
There are new categories added in dgmAttributeCategories.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#58 - 2014-08-26 15:41:02 UTC
Desmont McCallock wrote:
Just a heads up.
There are new categories added in dgmAttributeCategories.



Really? I'm only seeing 28 of them, same as before?

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Desmont McCallock
#59 - 2014-08-26 17:04:13 UTC  |  Edited by: Desmont McCallock
Well in that case, some attributes changed category ID cause it broke EVEMon data files generator.

Edit: Yeap, they moved some attributes to category 17 (Speed).
Golden Gnu
The Golden Gnu Corp
#60 - 2014-08-29 13:35:09 UTC
@Steve Ronuken
Thank you for all the work you do for all us 3rd party eve developers and indirectly all the users of 3rd party eve software. :)

Creator of jEveAssets - the asset manager

"Download is the meaning of life, upload is the meaning of intelligent life"