These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Proposed Changes to the Static Data Export (SDE)

First post First post
Author
Previous Topic Next Topic
Carbon Alabel
Gemini Talon
The Curatores Veritatis Auxiliary
#81 - 2016-05-01 18:50:14 UTC  |  Edited by: Carbon Alabel
Desmont McCallock wrote:
Universe data need to be independent as they always were.

As far as I know, the SDE was always distributed in the form of a single archive file, so how exactly was it independent?
Desmont McCallock
#82 - 2016-05-01 19:40:52 UTC
Carbon Alabel wrote:
Desmont McCallock wrote:
Universe data need to be independent as they always were.

As far as I know, the SDE was always distributed in the form of a single archive file, so how exactly was it independent?
In the form that you didn't need to do a lookup on another table. itemName in mapDenormalize was always provided. You didn't had to do a lookup on another table to get the data.
Desmont McCallock
#83 - 2016-05-03 10:02:14 UTC
Something interesting I found while working with re-importing the new SDE files back into a DB is that the 'mapSolarSystemJumps', 'mapConstellationJumps', 'mapRegionJumps' tables can be easily created with a query on 'mapJumps' and 'mapDenormalize' tables.

You may say "Pfff, no big deal...", but here they are for future reference.

/* mapSolarSystemJumps */
SELECT 
    denorm1.regionID as fromRegionID,
    denorm1.constellationID as fromConstellationID,
    denorm1.solarSystemID as fromSolarSystemID,
    denorm2.solarSystemID as toSolarSystemID,
    denorm2.constellationID as toConstellationID,
    denorm2.regionID as toRegionID
FROM mapJumps as jumps
JOIN mapDenormalize as denorm1 on jumps.stargateID = denorm1.itemID
JOIN mapDenormalize as denorm2 on jumps.celestialID = denorm2.itemID
ORDER BY denorm1.solarSystemID, denorm2.solarSystemID

/* mapConstellationJumps */
SELECT 
    denorm1.regionID as fromRegionID,
    denorm1.constellationID as fromConstellationID,
    denorm2.constellationID as toConstellationID,
    denorm2.regionID as toRegionID
FROM mapJumps as jumps
JOIN mapDenormalize as denorm1 on jumps.stargateID = denorm1.itemID
JOIN mapDenormalize as denorm2 on jumps.celestialID = denorm2.itemID
WHERE denorm1.constellationID != denorm2.constellationID
GROUP BY denorm1.regionID, denorm1.constellationID, denorm2.constellationID, denorm2.regionID
ORDER BY denorm1.constellationID, denorm2.constellationID

/* mapRegionJumps */
SELECT 
    denorm1.regionID as fromRegionID,
    denorm2.regionID as toRegionID
FROM mapJumps as jumps
JOIN mapDenormalize as denorm1 on jumps.stargateID = denorm1.itemID
JOIN mapDenormalize as denorm2 on jumps.celestialID = denorm2.itemID
WHERE denorm1.regionID != denorm2.regionID
GROUP BY denorm1.regionID, denorm2.regionID
ORDER BY denorm1.regionID, denorm2.regionID
Desmont McCallock
#84 - 2016-05-06 14:19:22 UTC  |  Edited by: Desmont McCallock
Steve Ronuken wrote:
All names are in invNames, so you can look them up, and insert them. This is for asteroid belts, moons, planets, stars, stations.
Just out of curiosity I wanted to give this a try, so I did some checking via

select 
 invNames.itemID,
 invNames.itemName as InvNames_itemName,
 mapDenormalize.itemName as MapDenormalize_itemName
from dbo.invNames as invNames 
join dbo.mapDenormalize as mapDenormalize on invNames.itemID = mapDenormalize.itemID
where invNames.itemName <> mapDenormalize.itemName

The result comes back with 122 rows, proving another data inconsistency in SDE.

This is getting nowhere.
Desmont McCallock
#85 - 2016-05-08 09:14:07 UTC
Yet another data inconsistency with names.
select 
 invNames.itemID,
 invNames.itemName as InvNames_itemName,
 invUniqueNames.itemName as InvUniqueNames_itemName
from dbo.invNames as invNames 
join dbo.invUniqueNames as invUniqueNames on invNames.itemID = invUniqueNames.itemID
where invNames.itemName <> invUniqueNames.itemName collate Latin1_General_BIN


This time the query returns 6994 rows.
Hurt Kion
Hurt Kion's Inc.
#86 - 2016-05-14 23:00:03 UTC
Some blueprints materials in SDK (blueprints.yaml normal and legacy) do not match Tranquility.

For example:
CONCORD Capital Shield Extender Blueprint
in SDK:

41603:
activities:
manufacturing:
materials:
- quantity: 1
typeID: 40354 [ Capital Shield Extender I ]
- quantity: 8
typeID: 41267 [ Compact Compounds ]
- quantity: 12
typeID: 41308 [ Restrained Compounds ]
- quantity: 41
typeID: 41268 [ Compact Conductors ]
- quantity: 44
typeID: 41266 [ Compact Electronics ]
- quantity: 47
typeID: 41309 [ Restrained Conductors ]
- quantity: 50
typeID: 41307 [ Restrained Electronics ]
products:
- quantity: 1
typeID: 41459
skills:
- level: 1
typeID: 3380
- level: 1
typeID: 22242
time: 36000
blueprintTypeID: 41603
maxProductionLimit: 5

in game:

1 x Capital Shield Extender II
13 x Broadcast Node
20 x Sterile Conduits
10 x Wetware Mainframe


Same with:
CONCORD 25000mm Steel Plates Blueprint
CONCORD Capital Armor Repairer Blueprint

maybe others...
Jai Blaze
Honor Forge
Joint Operation Involving Nobodys
#87 - 2016-05-17 04:21:48 UTC
Please more JSON files!! I've tried converting typeIDs.yaml into JSON and it refuses to validate. It's the single most important file for my own use, blueprints converted nicely and a lot of the others are small enough I can excel the crap out of it and reform it into a JSON file manually, but typeIDs is too big and when I try to convert manually it crashes. And nothing I've found so far will accept it and return a JSON file.


MOAR JSON!


please :)
Myme Temet'Nosce
Caldari Provisions
Caldari State
#88 - 2016-05-22 12:51:36 UTC
Can someone explain clearly why getting rid of a standardized, very fast and very efficient format as sql to use yaml, which is not as good by far ?

What did I miss ?
Pete Butcher
The Scope
Gallente Federation
#89 - 2016-05-22 13:09:40 UTC  |  Edited by: Pete Butcher
Myme Temet'Nosce wrote:
Can someone explain clearly why getting rid of a standardized, very fast and very efficient format as sql to use yaml, which is not as good by far ?

What did I miss ?


Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#90 - 2016-05-22 14:50:55 UTC
Myme Temet'Nosce wrote:
Can someone explain clearly why getting rid of a standardized, very fast and very efficient format as sql to use yaml, which is not as good by far ?

What did I miss ?



Try version controlling a database. Then you'll realise why. Total PITA. And what works in one database, doesn't always work in another. (And Eve uses at least three. SQL Server on the server, sqlite and a custom format on the client.)

What we're getting now, is a the output from their version control system, rather than some multi step process, which is a pain in the ass for them to manage, and a pain in the ass for people to convert to other database formats.

For me, having it all moved into yaml is a godsend, as I can write a single script, which takes the yaml, and dumps in to whatever database I want it to. Rather than the multi step process which I needed to run through before. It's significantly easier, requiring a lot less attention.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#91 - 2016-05-22 15:21:04 UTC
Jai Blaze wrote:
Please more JSON files!! I've tried converting typeIDs.yaml into JSON and it refuses to validate. It's the single most important file for my own use, blueprints converted nicely and a lot of the others are small enough I can excel the crap out of it and reform it into a JSON file manually, but typeIDs is too big and when I try to convert manually it crashes. And nothing I've found so far will accept it and return a JSON file.


MOAR JSON!


please :)


https://www.fuzzwork.co.uk/dump/sde-20160429-TRANQUILITY/typeid.json.bz2 ?


Converted with:
https://github.com/fuzzysteve/yamlloader/blob/master/TypesToJson.py

Works in python 2.7 (it'd work in 3.5, except I'm not doing the prints right for that) needs pyyaml, with the libyaml library installed. Change the loader to loader, rather than cloader to work (slower) without libyaml. May need to be 64 bit, for memory usage on parsing.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Mathliney Kagrel
Caldari Provisions
Caldari State
#92 - 2016-05-22 15:27:58 UTC  |  Edited by: Mathliney Kagrel
Steve Ronuken wrote:
Try version controlling a database. Then you'll realise why. Total PITA. And what works in one database, doesn't always work in another. (And Eve uses at least three. SQL Server on the server, sqlite and a custom format on the client.)


Ok, versioning is not easy but is it really required ?

Quote:
What we're getting now, is a the output from their version control system, rather than some multi step process, which is a pain in the ass for them to manage, and a pain in the ass for people to convert to other database formats.


I can understand the need to unify but why chosing YAML ? What is the advantage beside versioning, which is not really important imho ?

Quote:
For me, having it all moved into yaml is a godsend, as I can write a single script, which takes the yaml, and dumps in to whatever database I want it to. Rather than the multi step process which I needed to run through before. It's significantly easier, requiring a lot less attention.


Yes, thanx to you about this btw. Now what we are left with is waiting for you to export SDE in SQL format or build our own tool to convert data.



Pete Butcher wrote:
Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there.


Check wiki about standard, it is clearly. About efficiency and speed, I hope you are not comparing with YAML, are you ?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#93 - 2016-05-22 15:36:28 UTC
Versioning isn't important. Really? Are you really trying to say that? Are you kidding me?

Yes, it might not be as important for us, as consumers. But for CCP, when they have multiple streams, which need to be merged together, you'd better believe it's important.


As for 'waiting for me to export it' : https://github.com/fuzzysteve/yamlloader go knock yourself out. It's been available since a couple of days after the SDE was properly released.



As for SQL being standardized, yes, there are SQL standards. Which does your database conform to? Does it implement all of it? (no, it doesn't.) Do you use double quotes round identifiers? back ticks? Are identifiers case sensitive? Do you put square brackets round your table names?


Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Pete Butcher
The Scope
Gallente Federation
#94 - 2016-05-22 15:37:42 UTC
Mathliney Kagrel wrote:
Pete Butcher wrote:
Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there.


Check wiki about standard, it is clearly. About efficiency and speed, I hope you are not comparing with YAML, are you ?


Yeah, the wiki... Run SELECT TOP 1 in MySQL and, please, tell me again how it's standardized.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Mathliney Kagrel
Caldari Provisions
Caldari State
#95 - 2016-05-22 16:09:03 UTC
Pete Butcher wrote:
Mathliney Kagrel wrote:
Pete Butcher wrote:
Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there.


Check wiki about standard, it is clearly. About efficiency and speed, I hope you are not comparing with YAML, are you ?


Yeah, the wiki... Run SELECT TOP 1 in MySQL and, please, tell me again how it's standardized.


The fact that your MySQL database doesn't respect standards doesn't imply SQL isn't standardized.
Mathliney Kagrel
Caldari Provisions
Caldari State
#96 - 2016-05-22 16:14:09 UTC
Steve Ronuken wrote:
Versioning isn't important. Really? Are you really trying to say that? Are you kidding me?

Yes, it might not be as important for us, as consumers. But for CCP, when they have multiple streams, which need to be merged together, you'd better believe it's important.


As for 'waiting for me to export it' : https://github.com/fuzzysteve/yamlloader go knock yourself out. It's been available since a couple of days after the SDE was properly released.



As for SQL being standardized, yes, there are SQL standards. Which does your database conform to? Does it implement all of it? (no, it doesn't.) Do you use double quotes round identifiers? back ticks? Are identifiers case sensitive? Do you put square brackets round your table names?




It is an export, I never implied CCP should work with SQL and not YAML, we are talking about SDE here.

I know about your script, I don't use python but could give a try, I can also create my own tool, this is not the point. My point was : why not using something that is more than proven for ages and with very rich frameworks to handle it, no matter the language / IDE you are using (not to mention the Entity Framework for example) to use YAML ? I am just asking, I didn't say it was a bad choice, I'm just curious, I don't think the majority of ppl creating tools for EVE are using YAML for something else that config files or tiny serialization needs. Now, maybe I'm wrong.

And yes, of course I follow SQL standards, are they that big constraint ?


Won't reply further on this topic, I have my answer as you stated something I read before : it is easier and more convenient for them and since some tools exist to update our own products, I'll deal with it.

Thank you for your replies (both of you).
Pete Butcher
The Scope
Gallente Federation
#97 - 2016-05-22 16:19:08 UTC
Mathliney Kagrel wrote:
Pete Butcher wrote:
Mathliney Kagrel wrote:
Pete Butcher wrote:
Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there.


Check wiki about standard, it is clearly. About efficiency and speed, I hope you are not comparing with YAML, are you ?


Yeah, the wiki... Run SELECT TOP 1 in MySQL and, please, tell me again how it's standardized.


The fact that your MySQL database doesn't respect standards doesn't imply SQL isn't standardized.


Then run SELECT ... LIMIT 1 on Oracle or SELECT ... WHERE ROWNUM < 2 on SQL Server. Or use [] in MySQL. Or the same case-insensitive query in Oracle as In MySql. Have fun.

To put this in short - SQL is very poorly standardized. So poorly that you can say each RDBMS has its own. If you want to dispute this further, take some time and do research first, or get some experience with different databses.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#98 - 2016-05-22 16:35:35 UTC
Mathliney Kagrel wrote:
Steve Ronuken wrote:
Versioning isn't important. Really? Are you really trying to say that? Are you kidding me?

Yes, it might not be as important for us, as consumers. But for CCP, when they have multiple streams, which need to be merged together, you'd better believe it's important.


As for 'waiting for me to export it' : https://github.com/fuzzysteve/yamlloader go knock yourself out. It's been available since a couple of days after the SDE was properly released.



As for SQL being standardized, yes, there are SQL standards. Which does your database conform to? Does it implement all of it? (no, it doesn't.) Do you use double quotes round identifiers? back ticks? Are identifiers case sensitive? Do you put square brackets round your table names?




It is an export, I never implied CCP should work with SQL and not YAML, we are talking about SDE here.

I know about your script, I don't use python but could give a try, I can also create my own tool, this is not the point. My point was : why not using something that is more than proven for ages and with very rich frameworks to handle it, no matter the language / IDE you are using (not to mention the Entity Framework for example) to use YAML ? I am just asking, I didn't say it was a bad choice, I'm just curious, I don't think the majority of ppl creating tools for EVE are using YAML for something else that config files or tiny serialization needs. Now, maybe I'm wrong.

And yes, of course I follow SQL standards, are they that big constraint ?


Won't reply further on this topic, I have my answer as you stated something I read before : it is easier and more convenient for them and since some tools exist to update our own products, I'll deal with it.

Thank you for your replies (both of you).



tbh, yaml isn't being used as a final data source in this case. It's being used to transfer data. Which works well. It's easy to convert from yaml to something else. And that's the real benefit. If they were releasing sql server database backup (which was how it used to be) it was a pain to migrate, requiring some manual steps. And very few people other than CCP were using it. And anything else would increase their costs to release the SDE. This way, they can automate it, and hopefully get more releases done. including, for example, exports from SiSi. It's also far less likely to break (which happened sometimes, leading to a delay in getting it out.)


Yaml is far from perfect, but it fits this use case well.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Mathliney Kagrel
Caldari Provisions
Caldari State
#99 - 2016-05-22 17:48:57 UTC
Pete Butcher wrote:
Then run SELECT ... LIMIT 1 on Oracle or SELECT ... WHERE ROWNUM < 2 on SQL Server. Or use [] in MySQL. Or the same case-insensitive query in Oracle as In MySql. Have fun.

To put this in short - SQL is very poorly standardized. So poorly that you can say each RDBMS has its own. If you want to dispute this further, take some time and do research first, or get some experience with different databses.


Once again, you are out of topic here : there IS a SQL standard, you are only stating that this standard isn't enforced / respected by every engine. Great, there is a standard nonetheless.

Now, could you calm down your need to feel superior and speak with others like they are kids / noobs ? You have no clue who I am nor in what domain I'm an expert, you don't know what experience I have with databse, you only use a side case to make your point.

This is not the first time you reply with this tone, you have a very high opinion of yourself, we don't all share it.
Pete Butcher
The Scope
Gallente Federation
#100 - 2016-05-22 17:58:42 UTC
Mathliney Kagrel wrote:
Pete Butcher wrote:
Then run SELECT ... LIMIT 1 on Oracle or SELECT ... WHERE ROWNUM < 2 on SQL Server. Or use [] in MySQL. Or the same case-insensitive query in Oracle as In MySql. Have fun.

To put this in short - SQL is very poorly standardized. So poorly that you can say each RDBMS has its own. If you want to dispute this further, take some time and do research first, or get some experience with different databses.


Once again, you are out of topic here : there IS a SQL standard, you are only stating that this standard isn't enforced / respected by every engine. Great, there is a standard nonetheless.

Now, could you calm down your need to feel superior and speak with others like they are kids / noobs ? You have no clue who I am nor in what domain I'm an expert, you don't know what experience I have with databse, you only use a side case to make your point.

This is not the first time you reply with this tone, you have a very high opinion of yourself, we don't all share it.


Non-enforced standard isn't a standard. It's one of the standards. I don't wish to sound superior, but I don't want misinformation to spread. You made three claims, all of which are false.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool