These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
123Next pageLast page
 

SDEExternalsToSql Tool (Phoebe 1.0+ compatible)

First post First post
Author
Previous Topic Next Topic
Desmont McCallock
#1 - 2014-02-23 15:28:36 UTC  |  Edited by: Desmont McCallock
This version is deprecated. Use version 2.

With the latest changes on how CCP distributes the SDE from now on, by exporting the data to yaml files, sqlite db files and an MSSQL dump, the necessity for a tool that can import all external data back into an MSSQL DB arose.

So I'm presenting to you the SDEExternalsToSql tool. A tool that we use also in EVEMon.

You can find the source code at EVEMon's repository as the tool is embedded in it (look into the Tools folder).

The compiled version can be found for downloading here.

Direct dl link: EVEMonSDEExternalsToSql.zip

Notice: Yaml files and SQLite db files are not distributed along with the tool. You will have to get them from the community toolkit page

Yaml files mapping:

blueprints.yaml -> invBlueprintTypes, ramTypeRequirements
categoryIDs.yaml -> invCategories, translationTables, trnTranslationColumns, trnTranslations
certificates.yaml -> crtClasses, crtCertificates, crtRecommendations, crtRelationships
graphicIDs.yaml -> eveGraphics
groupIDs.yaml -> invGroups, translationTables, trnTranslationColumns, trnTranslations
iconIDs.yaml -> eveIcons
skinLicenses.yaml -> sknLicenses
skinMaterials.yaml -> sknMaterials
skins.yaml -> sknSkins
tournamentRuleSets.yaml -> ignored
typeIDs.yaml -> invTypes, dgmMasteries, dgmTypeMasteries, dgmTraits, dgmTypeTraits, translationTables, trnTranslationColumns, trnTranslations

Sqlite files mapping:

universeDataDx.db -> mapCelestialStatistics, mapConstellationJumps, mapConstellations, mapDenormalize, mapJumps, mapLandmarks, mapLocationScenes, mapLocationWormholeClasses, mapRegionJumps, mapRegions, mapSolarSystemJumps, mapSolarSystems


Known Issues:

  1. In SkinMaterials.yaml material has been renamed to materialSetID which currently doesn't get mapped. (Resolved in version 2)
  2. Translations for traits bonus text is currently not mapped. (Resolved in version 2)
  3. Some traits for non market ships are missing. (Resolved in version 2)


Enjoy responsibly.
Hel O'Ween
Men On A Mission
#2 - 2014-02-24 17:57:14 UTC
Thanks a bunch, mate!

EVEWalletAware - an offline wallet manager.
Desmont McCallock
#3 - 2014-02-24 20:09:57 UTC
Hel O'Ween wrote:
Thanks a bunch, mate!

No problem.
Darkblad
Doomheim
#4 - 2014-03-11 08:48:50 UTC
Cold be only me, but the tool crashes while attempting to import mapDenormalized from the Rubicon 1.3 SDE. My bet is that CCP fiddled with table formats in the sqlite?
(will provide the error message soon)

NPE – ISD – RIP
Darkblad
Doomheim
#5 - 2014-03-11 10:22:06 UTC  |  Edited by: Darkblad
Here's what I get on importing data: unfortunately as a screenshot only and, to make it harder, from a German language Windows installation :/
The path given (e:\Projects\...) in the error message is not given in the .config or any of the .xml files

NPE – ISD – RIP
CCP FoxFour
C C P
C C P Alliance
#6 - 2014-03-11 10:44:39 UTC
Awesome stuff! Thank you for putting the time into this, I imagine it will make lots of lives a lot easier. :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Rob Crowley
State War Academy
#7 - 2014-03-11 10:45:01 UTC
Darkblad wrote:
Cold be only me, but the tool crashes while attempting to import mapDenormalized from the Rubicon 1.3 SDE. My bet is that CCP fiddled with table formats in the sqlite?
(will provide the error message soon)

It's not just you, I fixed this quick and dirty by

MapDenormalizeTable.cs wrote:
@@ -72,7 +72,14 @@
parameters["y"] = mDenorm.y.GetValueOrDefaultString();
parameters["z"] = mDenorm.z.GetValueOrDefaultString();
parameters["radius"] = mDenorm.radius.GetValueOrDefaultString();
- parameters["itemName"] = String.Format("'{0}'", mDenorm.itemName.Replace("'", Database.StringEmpty));
+ if (mDenorm.itemName != null) {
+ parameters["itemName"] = String.Format("'{0}'",
+ mDenorm.itemName.Replace("'", Database.StringEmpty));
+ }
+ else
+ {
+ parameters["itemName"] = "Null";
+ }
parameters["security"] = mDenorm.security.GetValueOrDefaultString();
parameters["celestialIndex"] = mDenorm.celestialIndex.GetValueOrDefaultString();
parameters["orbitIndex"] = mDenorm.orbitIndex.GetValueOrDefaultString();

It ran through like that, but Desmont will know best how to do this properly.
Risingson
#8 - 2014-03-11 14:46:40 UTC
omg thank you for saving me from horror time !
Desmont McCallock
#9 - 2014-03-11 16:09:24 UTC
I'll get on it as soon as I finish with work. Expect a new version shortly.
Desmont McCallock
#10 - 2014-03-11 19:42:42 UTC
Issues have been fixed. Code was improved. New release have been issued. Same link. Enjoy.
Darkblad
Doomheim
#11 - 2014-03-11 20:00:33 UTC
Desmont McCallock wrote:
Issues have been fixed. Code was improved. New release have been issued. Same link. Enjoy.
Worked like a charm, thanks a lot! Big smile

NPE – ISD – RIP
Hel O'Ween
Men On A Mission
#12 - 2014-03-11 23:24:54 UTC  |  Edited by: Hel O'Ween
New Eden might be a cold, harsh place, but this Tech Lab forum feels like the lone warm and cozy inn in that dark void. Big smile

Thanks, Desmont.

EVEWalletAware - an offline wallet manager.
Sentenced 1989
#13 - 2014-03-17 15:05:44 UTC
Desmont McCallock wrote:
With the latest changes on how CCP distributes the SDE from now on, by exporting the data to yaml files, sqlite db files and an MSSQL dump, the necessity for a tool that can import all external data back into an MSSQL DB arose.


Nice of you to provide the tool, downloading now and thank you very much

The Incursion Guild

Epic Arc Guide
Hel O'Ween
Men On A Mission
#14 - 2014-03-20 11:41:17 UTC
Quick question, because I don't "speak" a word C# and therefore don't know where to look for this in the source: does the tool empty/drop existing tables/data before importing?

Or put another way: when an import failed (for whatever reason), do I need to remove the tables create by it before attempting another import?

EVEWalletAware - an offline wallet manager.
Desmont McCallock
#15 - 2014-03-20 20:08:57 UTC  |  Edited by: Desmont McCallock
Hel O'Ween wrote:
Quick question, because I don't "speak" a word C# and therefore don't know where to look for this in the source: does the tool empty/drop existing tables/data before importing?

Or put another way: when an import failed (for whatever reason), do I need to remove the tables create by it before attempting another import?
The tool drops and recreates the tables, except the invTypes table where it creates the columns (if they don't exist) and updates them.
Hel O'Ween
Men On A Mission
#16 - 2014-03-21 12:05:12 UTC
Awesome!

So it's basically a "no-brainer" tool. Exactly what the doctor prescribed ... Big smile

EVEWalletAware - an offline wallet manager.
Desmont McCallock
#17 - 2014-03-21 19:08:42 UTC
Hel O'Ween wrote:
Awesome!

So it's basically a "no-brainer" tool. Exactly what the doctor prescribed ... Big smile
Yeap, I always try to make "idiot proof" apps and tools.
T'rixie
Criminally Incompetent
#18 - 2014-05-03 23:17:22 UTC
Desmont McCallock wrote:
With the latest changes on how CCP distributes the SDE from now on, by exporting the data to yaml files, sqlite db files and an MSSQL dump, the necessity for a tool that can import all external data back into an MSSQL DB arose.


I've spent hours in the last few days installing & configuring MS SQL, getting my software to interface with it, working through the tables to find where things were only to discover the map data was missing, so then hunting through the interwebs and these forums to finally find my way here to this lovely thread.

Desmont - your tool ran the first time out of the box and worked beautifully. Thank you!
Desmont McCallock
#19 - 2014-05-04 08:01:11 UTC
T'rixie wrote:
Desmont McCallock wrote:
With the latest changes on how CCP distributes the SDE from now on, by exporting the data to yaml files, sqlite db files and an MSSQL dump, the necessity for a tool that can import all external data back into an MSSQL DB arose.


I've spent hours in the last few days installing & configuring MS SQL, getting my software to interface with it, working through the tables to find where things were only to discover the map data was missing, so then hunting through the interwebs and these forums to finally find my way here to this lovely thread.

Desmont - your tool ran the first time out of the box and worked beautifully. Thank you!


Glad you found it useful. :)
Darkblad
Doomheim
#20 - 2014-05-15 23:03:21 UTC
Just a short note:

Rubicon 1.4 SDE just got released
http://cdn1.eveonline.com/data/Rubicon_1.4_97069_db.zip

And the tool handled the externals w/o error, which lets me assume that it worked just fine Smile

NPE – ISD – RIP
123Next pageLast page