These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Rubicon SDE is out.

First post
Author
Previous Topic Next Topic
Darkblad
Doomheim
#61 - 2014-01-27 22:12:13 UTC
Desmont McCallock wrote:
This also.

This one did actually help, as I was able to download it while the other link is dead for now. thanks for that!

NPE – ISD – RIP
Captain Thunk
Explode. Now. Please.
Alliance. Now. Please.
#62 - 2014-01-29 08:21:00 UTC
Rob Crowley wrote:
Desmont McCallock wrote:
The description for the values can be found in the 'dgmAttributeTypes' table, column "displayName" (ship bonuses are with categoryID = 7) although some lack a proper 'displayName' (e.g. eliteBonusHeavyInterdictors1, eliteBonusElectronicAttackShip1, eliteBonusBlackOps1 etc.)
The vast majority of ship bonuses in dgmTypeAttributes have generic attributes from category 7 or 9 that don't have a display name and don't reference the type of bonus at all. What would be required would be a displayName for the ship bonus effect in dgmTypeEffects.

And there's also the problem of linking the bonus values in dgmTypeAttributes with the appropriate entries in dgmTypeEffects. E.g. for a Merlin we have "shipBonusCF = -4" and "shipBonusCF2 = 5", but in dgmTypeEffects there's "shipShield*ResistanceCF2" and "shipHybridDamageBonusCF2". So I don't really see a way to link the right numbers to the corresponding effects.

Kid Delicious wrote:
Using the Nestor as an example I grepped Steve's dump for "Amarr Battleship bonuses" which appears in the traits tab for that ship and got nothing.
Well, the "Amarr Battleship" part is apparently coded as "AB" which would still be somewhat usable, what I'm missing more is the text description of the bonus' effect.


The data is incomplete, the static DB released by CCP is missing a key table, called dgmExpressions iirc. This table contains the necessary information to link the Attributes and Effects. It also uses python executable code which makes it difficult to work with unless your app is also in python.

To get this table you need Entity's Reverence and a script to extract it from the client.
Desmont McCallock
#63 - 2014-01-29 20:20:45 UTC
Captain Thunk wrote:
Rob Crowley wrote:
Desmont McCallock wrote:
The description for the values can be found in the 'dgmAttributeTypes' table, column "displayName" (ship bonuses are with categoryID = 7) although some lack a proper 'displayName' (e.g. eliteBonusHeavyInterdictors1, eliteBonusElectronicAttackShip1, eliteBonusBlackOps1 etc.)
The vast majority of ship bonuses in dgmTypeAttributes have generic attributes from category 7 or 9 that don't have a display name and don't reference the type of bonus at all. What would be required would be a displayName for the ship bonus effect in dgmTypeEffects.

And there's also the problem of linking the bonus values in dgmTypeAttributes with the appropriate entries in dgmTypeEffects. E.g. for a Merlin we have "shipBonusCF = -4" and "shipBonusCF2 = 5", but in dgmTypeEffects there's "shipShield*ResistanceCF2" and "shipHybridDamageBonusCF2". So I don't really see a way to link the right numbers to the corresponding effects.

Kid Delicious wrote:
Using the Nestor as an example I grepped Steve's dump for "Amarr Battleship bonuses" which appears in the traits tab for that ship and got nothing.
Well, the "Amarr Battleship" part is apparently coded as "AB" which would still be somewhat usable, what I'm missing more is the text description of the bonus' effect.


The data is incomplete, the static DB released by CCP is missing a key table, called dgmExpressions iirc. This table contains the necessary information to link the Attributes and Effects. It also uses python executable code which makes it difficult to work with unless your app is also in python.

To get this table you need Entity's Reverence and a script to extract it from the client.
Ah yes, you are describing the perfect solution, scrubbing the bulk data from EVE cache folder.

P.S. I'm not mocking you Captain Thunk, merely CCP for an incomplete SDE.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#64 - 2014-01-30 11:23:45 UTC
https://www.fuzzwork.co.uk/dump/rubicon-1.1-94321/invTraits.xls.bz2
https://www.fuzzwork.co.uk/dump/rubicon-1.1-94321/invTraits.sql.bz2


I've not done a new complete conversion, as I'm unsure if 1.1.2 has any other updates in it. If it does, I'll do a full conversion. otherwise, I'll leave it for now.

loader is https://github.com/fuzzysteve/yamlloader/tree/master/traits

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
CCP Creber Cattus
C C P
C C P Alliance
#65 - 2014-01-30 11:28:12 UTC
Desmont McCallock wrote:
Captain Thunk wrote:
Rob Crowley wrote:
Desmont McCallock wrote:
The description for the values can be found in the 'dgmAttributeTypes' table, column "displayName" (ship bonuses are with categoryID = 7) although some lack a proper 'displayName' (e.g. eliteBonusHeavyInterdictors1, eliteBonusElectronicAttackShip1, eliteBonusBlackOps1 etc.)
The vast majority of ship bonuses in dgmTypeAttributes have generic attributes from category 7 or 9 that don't have a display name and don't reference the type of bonus at all. What would be required would be a displayName for the ship bonus effect in dgmTypeEffects.

And there's also the problem of linking the bonus values in dgmTypeAttributes with the appropriate entries in dgmTypeEffects. E.g. for a Merlin we have "shipBonusCF = -4" and "shipBonusCF2 = 5", but in dgmTypeEffects there's "shipShield*ResistanceCF2" and "shipHybridDamageBonusCF2". So I don't really see a way to link the right numbers to the corresponding effects.

Kid Delicious wrote:
Using the Nestor as an example I grepped Steve's dump for "Amarr Battleship bonuses" which appears in the traits tab for that ship and got nothing.
Well, the "Amarr Battleship" part is apparently coded as "AB" which would still be somewhat usable, what I'm missing more is the text description of the bonus' effect.


The data is incomplete, the static DB released by CCP is missing a key table, called dgmExpressions iirc. This table contains the necessary information to link the Attributes and Effects. It also uses python executable code which makes it difficult to work with unless your app is also in python.

To get this table you need Entity's Reverence and a script to extract it from the client.
Ah yes, you are describing the perfect solution, scrubbing the bulk data from EVE cache folder.

P.S. I'm not mocking you Captain Thunk, merely CCP for an incomplete SDE.



Smile
I have not checked if we changed anything in Rubicon 1.1.2 that we ship with the SDE (except adding the traits), but I THINK not.. :p
Darkblad
Doomheim
#66 - 2014-01-30 11:58:05 UTC
Darkblad wrote:
Desmont McCallock wrote:
This also.

This one did actually help, as I was able to download it while the other link is dead for now. thanks for that!

Appears to work with the 1.1.2 Dump as well.
As a sidenote: initial run of the tool didn't import factionID values into invTypes (didn't neither work with 1.1.)
Used a dirty workaround to fix that for me:
typeIDs.yaml:

  • replace iconID with tempID
  • replace factionID with iconID

run the import, then In the database

  • rename iconID column to factionID

Import again, with the unmodified yaml files

NPE – ISD – RIP
Berluth Luthian
Sebiestor Tribe
Minmatar Republic
#67 - 2014-01-30 16:02:00 UTC
Does this have the new universe data?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#68 - 2014-01-30 16:19:44 UTC
Berluth Luthian wrote:
Does this have the new universe data?


New universe data?

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Desmont McCallock
#69 - 2014-01-30 18:47:20 UTC  |  Edited by: Desmont McCallock
Darkblad wrote:
Darkblad wrote:
Desmont McCallock wrote:
This also.

This one did actually help, as I was able to download it while the other link is dead for now. thanks for that!

Appears to work with the 1.1.2 Dump as well.
As a sidenote: initial run of the tool didn't import factionID values into invTypes (didn't neither work with 1.1.)
Used a dirty workaround to fix that for me:
typeIDs.yaml:

  • replace iconID with tempID
  • replace factionID with iconID

run the import, then In the database

  • rename iconID column to factionID

Import again, with the unmodified yaml files

Didn't noticed the factionID, I'll make sure to include that and the masteries.

Edit: And the Traits.
Desmont McCallock
#70 - 2014-02-01 20:25:36 UTC  |  Edited by: Desmont McCallock
Updated EVEMonYamlToSql tools. It now imports all fields that the yaml files include into an MSSQL database (masteries, factionID, trais included).

In case you find any mistake pls post on this thread.
Rob Crowley
State War Academy
#71 - 2014-02-15 01:13:18 UTC
I finally had some time to play with the new traits info and found that a couple bonuses have some precision/rounding issues:

typeIDs.yaml wrote:
bonus: 99.650000000000006
bonusText: reduction in Torpedo Launcher powergrid requirement

bonus: 33.329999999999998
bonusText: reduction in Ice Harvester duration and activation cost

The ones I found were limited to stealth bombers and mining barges/exhumers. Maybe you could do something about those and make them correct and pretty.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#72 - 2014-02-15 01:18:45 UTC
New conversions.

https://www.fuzzwork.co.uk/dump/sqlite-latest.sqlite.bz2
https://www.fuzzwork.co.uk/dump/postgres-latest.sql.bz2
https://www.fuzzwork.co.uk/dump/mysql-latest.tar.bz2

https://www.fuzzwork.co.uk/dump/rubicon-1.2-94438/


For mysql, you'll find the universe data in a separate file to import. the others have it embedded.

I'm /pretty/ sure it'll work, though the conversion tool I used had some data type issues. Seriously considering picking up a commercial tool for this. Smile

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Darkblad
Doomheim
#73 - 2014-02-15 10:20:32 UTC
I'm a bit lost here and maybe some of you can help me out. With the universe data now separated to an sqlite file, I've lost the option to do external queries to those tables from (you may start to hate me now:) Excel.
Is there any option for me to get those tables back into the original SDE (mssql)? I've already got myself the sqlite odbc driver, but start to realize that I'm too much of an amateur for this challenge (I'm happy to put together some basic sql queries)

I feel tempted to pick the mysql conversion Steve created to merge them with the mssql SDE. Would that work?

I'd appreciate some help with that - feel free to call me names Blink

NPE – ISD – RIP
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#74 - 2014-02-15 18:53:00 UTC  |  Edited by: Steve Ronuken
Darkblad wrote:
I'm a bit lost here and maybe some of you can help me out. With the universe data now separated to an sqlite file, I've lost the option to do external queries to those tables from (you may start to hate me now:) Excel.
Is there any option for me to get those tables back into the original SDE (mssql)? I've already got myself the sqlite odbc driver, but start to realize that I'm too much of an amateur for this challenge (I'm happy to put together some basic sql queries)

I feel tempted to pick the mysql conversion Steve created to merge them with the mssql SDE. Would that work?

I'd appreciate some help with that - feel free to call me names Blink



What might work better for you, is loading data into the tables that are there, from CSV dumps from sqlite? (or from my site.)

http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/

http://stackoverflow.com/questions/15655380/import-csv-into-sql-server-including-automatic-table-creation

http://stackoverflow.com/questions/75675/how-do-i-dump-the-data-of-some-sqlite3-tables

https://www.sqlite.org/download.html

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Darkblad
Doomheim
#75 - 2014-02-15 23:00:54 UTC
I'll try that, thanks!

NPE – ISD – RIP
Desmont McCallock
#76 - 2014-02-16 09:53:36 UTC
Darkblad wrote:
I'm a bit lost here and maybe some of you can help me out. With the universe data now separated to an sqlite file, I've lost the option to do external queries to those tables from (you may start to hate me now:) Excel.
Is there any option for me to get those tables back into the original SDE (mssql)? I've already got myself the sqlite odbc driver, but start to realize that I'm too much of an amateur for this challenge (I'm happy to put together some basic sql queries)

I feel tempted to pick the mysql conversion Steve created to merge them with the mssql SDE. Would that work?

I'd appreciate some help with that - feel free to call me names Blink
If you give me some time I may be able to help you as I'm already in the works for creating a tool that imports all separately data back into the MSSQL DB (as we need that for generating the EVEMon data files).
Darkblad
Doomheim
#77 - 2014-02-16 18:47:44 UTC
Desmont McCallock wrote:
If you give me some time I may be able to help you as I'm already in the works for creating a tool that imports all separately data back into the MSSQL DB (as we need that for generating the EVEMon data files).
That's just great \o/
And there's a pile of other projects that'll keep me busy for quite some time (compiling a Mining Guide /w the current state, I start to love it Twisted)

NPE – ISD – RIP
Jess Technite
Almost Absolute
#78 - 2014-02-19 13:38:31 UTC
Steve Ronuken wrote:
https://www.fuzzwork.co.uk/dump/rubicon-1.0-93082/
https://www.fuzzwork.co.uk/dump/sqlite-latest.sqlite.bz2 (look ma! It's a symlink I'll try to keep pointing at the latest version!)
https://www.fuzzwork.co.uk/dump/mysql56-rubicon-1.0-93082.tar.bz2
https://www.fuzzwork.co.uk/dump/rubicon-1.0-93082/postgres-rubicon-1.0-93082.sql.bz2 (floats converted to numerics)


After this and your post about CSM candidature you can count with my vote, clearly.
Hel O'Ween
Men On A Mission
#79 - 2014-02-20 10:25:49 UTC
Desmont McCallock wrote:
If you give me some time I may be able to help you as I'm already in the works for creating a tool that imports all separately data back into the MSSQL DB (as we need that for generating the EVEMon data files).


That would be a nice little helper, as I'm also relying on creating EWA's data from MS SQL.

EVEWalletAware - an offline wallet manager.
Desmont McCallock
#80 - 2014-02-22 12:17:54 UTC  |  Edited by: Desmont McCallock
I'm almost done with the tool that imports the ' from now on extracted to another format' data back into MSSQL, but...

@CCP Creber Cattus
First, I have stumbled on a issue with the data types. In mapCelestialStatistics table the 'fragmented' and 'locked' columns are defined as "int" although they should be defined as "bit". This difference is causing me troubles when casting the type.

Also "pressure", "radius" and "mass" are defined as 'int' although they should be defined as "real" which results to incorrect data (data are different in previous SDE versions).

Is it possible to switch the data type to the correct type?

Second, in mapLandmarks table there is no 'radius' and 'importance' columns which leads to data loss as previous SDE versions contained data in those columns.
Is there an explanation to this?