EVE Technology Lab

Hi Desmont,

I'm working on a kind of similar tool right now, for other reasons. What I aim to do is to provide the ability for custom mapping by using a XML file to provide the desired mapping.

Using ADO, it holds the connection strings for both the source and the target database (which should already exist). And the mapping is simply defined as

Tables
   Table Name ="invCategories"
      Columns
         Column Name="categorieID" TypeTarget="(ADO DataType Enum value for source column) Size="" Precision=""
         Column Name="categorieName" TypeTarget="(ADO DataType Enum value for source column) Size="100" Precision=""
etc.

Maybe that's an approach that removes the need for your request: "If this is the case, please be so kind and help me out by providing at least the basic specs (like what engine should be used for MySQL etc)."

This is something CCP needs to do.

Reason being different people do different flavours of conversion and so parts can change meaning if you get too used to 1 guy doing a conversion, when he quits and he will, then you will have to start using the conversion whoever takes his place. Which will likely mean rewriting your apps to accommodate the changes.

It needs to be a single consistent conversion and CCP is by far best placed to do this.

Is it possible to have a generic SQL exporter? I would imagine that there are data types that are common with the same keywords across the available RDBMS platforms. If you have CREATE TABLE and INSERT statements which are generic enough, then I imagine it would make things a lot easier for import.

Dragonaire, thank you very much for your comments. I will surely need your insights and knowledge, especially with MySQL which I believe you know best.

lol Captain. CCP never says NO, but they never say YES. Remember the famous Soon™.

Heh. I started because the person I was depending on was slow.

They stopped, I kept going.

Still need to look into writing a python script to do the migration of the sql server data over to something else.

Once that's done, the set of scripts should handle pretty much any system that sqlalchemy understands.

I guess, after that, I could look into wrapping it up with QT so it just does everything, with a nice gui.

Mainly it's not about the languages we,as devs, use to write any tool that deals with the SDE, as it's about the schema in use.

At this point let me explain the reason behind me using the schema I use.

When CCP started exporting tables to yaml and dropping them from the SDE data dump, the first logical think I thought of doing was to re-create the tables as they were before, keeping the schema as it was before the drop.
This way the DB could keep a consistence and backwards compatibility for systems that already used that schema.
I just then re-insert the data in the appropriate columns and in case there are new data, that can fit in the same table, I just add additional columns.
In case the additional data needed to be on their own separate table, I create a new table but keeping in mind the way CCP was organizing the schema (if the data represented CCP dogma then the data went to a table named dgm[TableName]).

All the above seemed the appropriate way of doing the re-importation, the "CCP style".

Steve chose to go down another path and I totally have no objections on that. And to show you that there is no rivalry between us, I'll let you in into a little secret, as I'm also contributing to Steve's tool, either by advise or actual code (@Steve I haven't given up on the tool, it's just that RL and personal projects come first).

If I go forward and extend the tool to provide the ability for it to create dumps in other SQL formats, it will be totally up the user of the tool to create the conversion of their choice, at any time they chose to do it. The only obligation from the tool's part would be to keep a steady schema and compensate on any change CCP will do to the SDE.
And as the tool is open sourced, I don't believe that there will be any problem in the future, even if I quit at some point, as there are more than enough .NET/C# devs out there, playing EVE.

Yes it is and the default on newer versions. Just as a suggestion might look at some of the things I did in Yapeal to handle updates and schema changes. 'Console/Command/DatabaseUpdater.php' and 'Console/Command/DatabaseInitialer.php' are the main files to look at. I use SQL directly with some find and replace but you could probably do the same thing in yaml if that made more sense as well for this tool. Main ideas from it would be using a diff for anyone just doing an update or the full thing for anyone that is installing fresh etc.

Just been thinking about the schema and how yours vs Steve's is different (Haven't really looked at them side by side yet) and what you might look at is SQL views. I looked it up and it seems SQLite supports them so it might be possible to come up with a common one you both like and just use some view to bridge any gaps from legacy stuff. That would allow everyone awhile to transition over without forcing everything to be done at once. The only limiting thing about them you might run into is most DB engines either don't let you or limit you about writing to the views vs what you can do with tables. This probably isn't really a problem in this case since everything should be read-only except for updating when CCP changes things where you'd just make changes to the base tables to start with.

Just to make sure I understand how your tool work currently to do the translations you go from the DATADUMPXXX.bak file after importing it into MS Server Express and importing all the yaml as well then extract everything back out for SQLite etc? Or do you go about it another way. I've grabbed your code from git but only had a little time to look at it today so a little background on the overall work flow and what to look at would be helpful and probably save me and others some time to understand the code as it is now

OP updated to keep you in the loop of project progress.