These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

Odyssey 1.0.7 89602 SDE Conversions - Mysql, XLS/CSV, Postgres, sqlite
Author
Previous Topic Next Topic
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#1 - 2013-06-20 14:47:13 UTC
Amizo Hamma
Imperial Shipment
Amarr Empire
#2 - 2013-06-21 00:13:06 UTC
Bravo. I enjoy seeing community contributions like this.

I do have a question, however. How do you do the postgres dump? I assume you do some sort of mssql connector that can import to postgres? Then dump that db. I've played with it before, but I never got it working correctly.

Thanks!
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#3 - 2013-06-21 00:56:42 UTC
I've got an old copy of the Mysql Migration Toolkit, which I use to dump to a pair of files, one with the creation sql, one with the insertion. After a little judicious editing of tinyints, I create the tables, then split up the insertion SQL (as it was too big for my mysql client input buffers)

Then I export it with mysqldump in postgres compatible mode, run it through a postgres converter to make it actually compatible (it's a python one I found a while ago) and import into postgres. Finally, I re-export it, and zip it :)

A little involved, but I'm doing the mysql one for myself anyway, so it's only 3 more steps, which are easily scripted.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#4 - 2013-06-23 03:34:27 UTC
Just to note, the toolkit page now says 1.0.9

It's the same file. They've just renamed it.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Rob Crowley
State War Academy
#5 - 2013-06-23 08:10:57 UTC
Steve Ronuken wrote:
Just to note, the toolkit page now says 1.0.9

It's the same file. They've just renamed it.

Nope, they fixed the iconIDs.yaml file.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#6 - 2013-06-23 16:47:12 UTC
Rob Crowley wrote:
Steve Ronuken wrote:
Just to note, the toolkit page now says 1.0.9

It's the same file. They've just renamed it.

Nope, they fixed the iconIDs.yaml file.



ahh. Same database though. Smile

So I'm not going to do another conversion.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Nari Neya
Vhero' Multipurpose Corp
#7 - 2013-06-25 02:01:52 UTC
Steve Ronuken wrote:
Rob Crowley wrote:
Steve Ronuken wrote:
Just to note, the toolkit page now says 1.0.9

It's the same file. They've just renamed it.

Nope, they fixed the iconIDs.yaml file.


So I'm not going to do another conversion.


Same issue as with previous dump (I wrongly attributed it to Jercy first time).
The dump is double-encoded, all rows in trnTranslations are mangled.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#8 - 2013-06-25 09:30:28 UTC
Nari Neya wrote:
Steve Ronuken wrote:
Rob Crowley wrote:
Steve Ronuken wrote:
Just to note, the toolkit page now says 1.0.9

It's the same file. They've just renamed it.

Nope, they fixed the iconIDs.yaml file.


So I'm not going to do another conversion.


Same issue as with previous dump (I wrongly attributed it to Jercy first time).
The dump is double-encoded, all rows in trnTranslations are mangled.



I just tried restoring the trnTranslations table, just using: bzcat trnTranslations.sql.bz2|mysql databasename

It, while I don't read japanese, appears to be showing up correctly?

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Nari Neya
Vhero' Multipurpose Corp
#9 - 2013-06-25 12:14:42 UTC
I don't read Japaneze either, but I can open the dump file in hex viewer and clearly see, that the bytes in multibyte Cyrillic strings are not formed with 0xD0 prefix, but with 0xC3 0x90, consistently repeating after each two other octets, which clearly indicating that the string is double-encoded into UTF-8.
Also, I didn't mean separate tables export. But the mysql55-odyssey-1.07-89602.tbz2 (full package).
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#10 - 2013-06-25 12:34:22 UTC  |  Edited by: Steve Ronuken
Weird. I'm not doing anything unusual in the export. mysqldump should be exporting files, by default, as utf8. And I'm not passing any directives to change that.

Hmm. It's probably the shell causing trouble. On the next time round, I'll try avoiding using any shell redirections.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#11 - 2013-06-25 13:50:54 UTC
Steve Ronuken wrote:
Weird. I'm not doing anything unusual in the export. mysqldump should be exporting files, by default, as utf8. And I'm not passing any directives to change that.

Hmm. It's probably the shell causing trouble. On the next time round, I'll try avoiding using any shell redirections.



http://www.fuzzwork.co.uk/dump/mysql55-odyssey-1.09-89602.tbz2

/Should/ have single encoding.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Nari Neya
Vhero' Multipurpose Corp
#12 - 2013-06-25 19:05:06 UTC
Curiouser and curiouser.
Files are slightly different, but the difference is in range of a few bytes.
And the double encoding issues are still there.
Mind if I ask, how do you construct the file?
Nari Neya
Vhero' Multipurpose Corp
#13 - 2013-06-25 19:29:11 UTC
P.S.
I doubt it is redirect. More like wrong database locale settings cause additional conversion.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#14 - 2013-06-25 22:39:13 UTC
The export is done with:

mysqldump -u root -p --default-character-set=utf8 --databases sdeodyssey107 -r mysql55-odyssey-1.09-89602.sql

The database is set with a utf8 default character set, and so are the individual tables.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Nari Neya
Vhero' Multipurpose Corp
#15 - 2013-06-26 07:23:24 UTC
Export is fine. But how do you import it?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#16 - 2013-06-26 09:24:18 UTC
Nari Neya wrote:
Export is fine. But how do you import it?



Well, to reimport that export, I'm just using

mysql databasename < filename.sql


If you're talking about the whole process, look up a few posts.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Nari Neya
Vhero' Multipurpose Corp
#17 - 2013-06-26 09:47:06 UTC
Ok, let me rephrase it. Does the database before export contains properly encoded strings?
Golden Gnu
The Golden Gnu Corp
#18 - 2013-06-26 10:23:25 UTC  |  Edited by: Golden Gnu
You should update your post (or create a new one).
Odyssey 1.0.9 is out and you already converted it. Blink
******* awesome as always - best source for data in the known universe! o7

EDIT:
It was giving me some trouble, because the tar file did not have an file extension

Creator of jEveAssets - the asset manager

"Download is the meaning of life, upload is the meaning of intelligent life"
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#19 - 2013-06-26 10:44:01 UTC
Golden Gnu wrote:
You should update your post (or create a new one).
Odyssey 1.0.9 is out and you already converted it. Blink
******* awesome as always - best source for data in the known universe! o7

EDIT:
It was giving me some trouble, because the tar file did not have an file extension



1.0.9 is the same database as 1.0.7.

The only change was some of the yaml files.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Nari Neya
Vhero' Multipurpose Corp
#20 - 2013-06-26 21:01:44 UTC
Golden Gnu wrote:
You should update your post (or create a new one).
Odyssey 1.0.9 is out and you already converted it. Blink
******* awesome as always - best source for data in the known universe! o7

EDIT:
It was giving me some trouble, because the tar file did not have an file extension

1.0.9 was GZIP'd, rather than BZIP'd. Rename it to ".tgz" or run tar -xzf instead of -xjf
12Next page