EVE Technology Lab

I was working on getting the YAML import done for EVE IPH and basically ended up writing a new program anyway. So, I added a few features and expanded it to build different types of databases and now I have an app I’m sharing with you all.

EVE SDE Database Builder is a Windows app that basically lets anyone run the SDE builds for 6 different Database types whenever they want and they can customize the import by language type and selecting the data they want or do not want (does anyone use charBloodlines?). Users just need to download the SDE from the dev site and save the folders for use.

I know there are other uploaders out there but I figured I’d make my own and share it. I’d appreciate some testing to make sure it’s working properly and feedback on its value.

Main links for the application:

• Screenshot of the program
• Binaries for installing the program
• Github for the code
• Schema I’m using in msSQL (other database types use similar types as much as possible and the table schema shouldn’t be any huge surprises)

Key features:

• Imports all YAML files in about 30 minutes (the universe files take about 23 minutes alone).
• Allows the selection of specific YAML files so users don’t have to import or build the entire database each time.
• Imports the SDE YAML files into Microsoft Access, SQLite, and CSV files with the option to save CSV to SSV – Semi-colon separated values with European decimal format (10.000,00) for use in Excel by non-US decimal format users.
• Imports the SDE YAML files into local servers for Microsoft SQL Server, PostgreSQL, and MySQL. To use these import types, you need to have a local server installed on your machine to connect to.
• Settings to import tables with translated fields for English, French, German, Japanese, Chinese, and Russian. When a field is translatable, the program will import the selected language.
• Threaded processing allows for increasing import times. Users can select to use maximum threads (no limit) or a number of 1-24 threads depending on their system. Users can set the threads in the File menu.
• Contains an updater function to update the program when changes are uploaded to GitHub.

Basic functions:

• Download the SDE files under ‘Static Data Export’ here and extract to a folder on your computer.
• Select the SDE folder (the ‘sde’ folder part of whatever folder you extracted the data into).
• If you plan to make an SQLite, Access, or CSV/SSV database, select the final folder where the database will be saved (CSV ‘database’ will be a new folder.
• Enter the name of the database in the text box provided and enter other information to connect to your particular database type as necessary.
• Select import language
• Save Settings – you should see a list of yaml files in the file list. If not, you didn’t save the correct yaml SDE file folder with the base ‘sde’ folder as the root.
• Select the files you want to import (save settings will save what you checked as well)
• Press ‘Build Database’ to begin.

Database table updates:

• mapDisallowedAnchorCategories (new) and mapDisallowedAnchorGroups (new) will contain the solarSystemID and the categoryID and groupID respectively.
• mapItemEffectBeacons table (new) contains the itemID and the effectBeaconTypeID associated with the itemID (used for secondary suns in wormholes).
• invTypes includes the field [packagedVolumes], which for ships and containers is the packaged volume of the item. For all other items, it is the volume. Information from this thread.
• The mapDenormalized table no longer has the groupID field (because it’s not in the YAML and it’s a pain to look up for duplicate data). The groupID field can be looked up by linking it to the typeID in invTypes.
• The program may import translation tables for internal use in importing tables with translation data however, they will not be imported into the final database unless checked.

I’d appreciate feedback from testing or looking through the code for improvements. I’m keeping this open source (MIT license) and I’m happy to let the community support it if it is valuable. I know it’s in VB.net, which many people loathe, but it’s simple and updates should be relatively easy when necessary. If you have any questions or bug reports, please post them here.

Thanks

Looks good so far. A couple of points:

There doesn't seem to be a vbproj file in EVE-SDE-Database-Builder-master\EVE-SDE-Database-Builder-master\EVE SDE Database Builder so the sln file can't load the project.

Any chance of enabling username / password for MSSQL?

OK I added the file. I think I hacked it on another project and caused it to not update since I set it to my local settings for debug folders, etc. If you use it, it will fail on first run until you change it. I'm sure git has a way to allow for this but I'm not sure how to do it. I ignore it in another app but not sure why it still works.

On username and pass, I can probably do that. I didn't have a server with user/pass to connect to to try it. Let me see if I can get it updated tomorrow to test.


I basically handle the blueprint information like you do with one small change. I don't have a separate table for probabilities. I just save that information in industryActivityProducts - if it's activityID 1, then the probability is one. All the other tables match yours. I don't use ramTypeRequirements.

One thing I was wondering about is if we could combine these tables a bit more so it's not all confusing. In the past I combined the skills and the materials into one column and put a 'consume' flag with 1 for all mats, and 0 for skills, which is similar to what it was in the past. Although maybe people want skills and they've used your conversions for a while probably. I'm open to whatever though, as long as we are consistent.

That's true, but the data we are talking about in blueprints.yaml had no pre-defined tables except in the old schema. CCP removed a lot of fields from that in the new yaml file. So Steve and I are creating what we think works best to store the data from the yaml file. I don't think it's necessary to recreate invBlueprints and ramTypeRequirements but having consistency in what data we combine or not combine to make it easier for 3rd party devs to understand is.

The lack of a consistent database schema has always been a lingering issue with the SDE after they started the switch to yaml. There is data duplication all over the place and there isn't really a change log (that I can find anyway) for the yaml files. So we make due with what we figure out here in the tech forum.

So i downloaded the source code and took a look at how you connect to SQL Server:

Public Sub New(ByVal DatabaseName As String, ByVal InstanceName As String, ByRef Success As Boolean)

Conn = New SqlConnection(String.Format("Server={0}\{1};Trusted_Connection=True; Initial Catalog=master; Integrated Security=True;Connection Timeout=60;",

Can I assume that Server={0}\{1} = Server=DatabaseName\InstanceName

Does {0} = the passed in DatabaseName variable
Does {1} = the passed in InstanceName variable

If so, i'm surprised this works for anyone :)

I thought Server= should only be the InstanceName, or possibly different if it's a named instance, in which case it would be Servername\InstanceName

for a test, I changed it to be: Server=localhost;

And that seemed to work - it created the DB at least

Sadly I'm getting wife aggro and no more time to test until later this evening, but will report back with findings :)

Awesome. Thanks for testing that out.

I was curious if people could connect since it's a bit funky. The way I tried to connect it is to use the "server name" format from the SQL Management Studio, and for me it's HostName\InstanceName.

So perhaps I'm just labeling it wrong. The Environment.MachineName works for the HostName and then the instance is just what you put in the box. Is there another way to do it? Did you try putting in just the instance name? I'll change the name of the label if that's the case. I can see how it's confusing since you need to strip the host from the connection string but it says "server".

I've started work on an Access database for Eve industry as a class project. For the project, I'm just using CSV imports, but after it's turned in I was planning on developing it into something for release. Would you mind if I get in touch with you after I've submitted it?

Hi,

I'm trying to use this tool to create a mysql database, but running into problems.

i downloaded the sde files and the tool binaries.

I'm on windows 7.

i selected the mysql option, got the sde folder.

then i hit " build database" and it says no files selected for import.

I figured out i needed to click on save settings. Then a list of checkbox appears, and i can chose what tables i want to convert.

i picked categoryID for a test, then i hit "build database"

i then get a non-critical error : " The Path cannot be all whitespace or empty"

then the programs loads translations.yaml, and proceeds to make the mySQL database.

The table is created but is empty of any data.

at the end, i get the following error :

An Unhandled Exception has occured and the program will now close.
Copy the data below and send to developer.

Source: Microsoft.VisualBasic
Message: File 'C:\xampp\mysql\data\eve\invCategories.csv' not found (Errcode: 2 "No such file or directory")
Raw Error Text: MySql.Data.MySqlClient.MySqlException (0x80004005): File 'C:\xampp\mysql\data\eve\invCategories.csv' not found (Errcode: 2 "No such file or directory")
at Microsoft.VisualBasic.CompilerServices.Symbols.Container.InvokeMethod(Method TargetProcedure, Object[] Arguments, Boolean[] CopyBack, BindingFlags Flags)
at Microsoft.VisualBasic.CompilerServices.NewLateBinding.CallMethod(Container BaseReference, String MethodName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack, BindingFlags InvocationFlags, Boolean ReportErrors, ResolutionFailure& Failure)
at Microsoft.VisualBasic.CompilerServices.NewLateBinding.ObjectLateCall(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack, Boolean IgnoreReturn)
at Microsoft.VisualBasic.CompilerServices.NewLateBinding.LateCall(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack, Boolean IgnoreReturn)
at EVE_SDE_Database_Builder.frmMain.BuildEVEDatabase(Object UpdateDatabase, DatabaseType DatabaseType)
at EVE_SDE_Database_Builder.frmMain.btnBuildDatabase_Click(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Thank you for sharing this very useful tool. Everything works great except for when i try to import "Universe Data (multiple files)". I receive an "Object reference not set to an instance of an object" error. I am using the 2016-12-13 SDE and all the files are located in the input directory. It does work for all the other yaml data just not the universe data.



Any suggestion would be greatly appreciated.

Thanks again!

Added this option, if the user name/id is blank, it will use the windows auth to log in. If the user name/id is there, then it will use the entry with password to try and log in. Let me know if you have issues.


Thanks, this has been fixed.