These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

SDE and IEC for Inferno 1.0
Author
Previous Topic Next Topic
Desmont McCallock
#1 - 2012-05-22 19:39:41 UTC  |  Edited by: Desmont McCallock
Static Data Export

IEC Types
IEC Renders
IEC Icons

Change Log:
-- invTypes.marketGroupID changed to int
-- invMarketGroups.marketGroupID changed to int
-- invMarketGroups.parentGroupID changed to int
Kronus Heilgar
Science and Trade Institute
Caldari State
#2 - 2012-05-22 21:45:39 UTC
MySQL port? Anyone? Anyone?

This is not the signature you're looking for.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#3 - 2012-05-22 22:27:55 UTC  |  Edited by: Steve Ronuken
Kronus Heilgar wrote:
MySQL port? Anyone? Anyone?



Inferno 1.0 dump.
http://www.fuzzwork.co.uk/dump/

The xls and separate sql dumps are done

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#4 - 2012-05-23 10:26:36 UTC
I've heard that some people are getting really poor import performance with the export above. so I've also taken the one below, which might run faster for you. It's using the extended insert option, which can improve performance (With some side effects on compatibility and resumability. Probably won't affect you.)

http://www.fuzzwork.co.uk/dump/inferno10/mysql55-inferno10-extended.sql.bz2

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#5 - 2012-05-23 11:02:10 UTC
Blerg. apparently some folks are having trouble downloading from my site. So, two alternates:

http://dl.eve-files.com/media/1205/mysql55-inferno10-extended.7z
https://www.dropbox.com/s/lmvbu40k1ftb5jt/mysql55-inferno10-extended.sql.bz2

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Khorkrak
KarmaFleet
Goonswarm Federation
#6 - 2012-05-23 18:10:39 UTC
Thanks Steve - I'll be applying this tonight and appreciate the faster queries. This form would be even faster though for MySQL.

load data local infile 'factions.dat' into table factions
fields terminated by '\t' lines terminated by '\n';

I use that for my project's trimmed down SDE tables for MySQL along with the corresponding equivalent for PostgreSQL. It's a tad annoying though that there's no way to do it inline in one file with MySQL though but that's par for the course.

COPY factions (faction_id, faction_name) FROM stdin;
500001 Caldari State
500002 Minmatar Republic
500003 Amarr Empire
500004 Gallente Federation

Developer of http://www.decloaked.com and http://sourceforge.net/projects/pykb/
Khorkrak
KarmaFleet
Goonswarm Federation
#7 - 2012-05-24 00:29:32 UTC
Just loaded the latest dump and wow - way faster than previously - thanks so much.

Developer of http://www.decloaked.com and http://sourceforge.net/projects/pykb/
diabeteman
Diabete Studios
#8 - 2012-05-30 01:47:16 UTC
Hey guys :)

I'm desperate for sqlite & postgresql conversions of the datadump.

I usually get them from here : http://zofu.no-ip.de/ , but the guy who maintains them appears to be on vacation :D

I tried to restore the mssql dump and export it to a SQL file. Then follow the steps described in http://zofu.no-ip.de/esc10/work/DBHOWTO But this doesn't work. The export format I make seems not to be compliant to what the perl scripts want.

Could someone help ?

thx in advance <3

-diab

Initiator and CTO of Eve Corp. Management
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#9 - 2012-05-30 02:12:53 UTC  |  Edited by: Steve Ronuken
I've got the mysql conversion done.

So I've taken what should be a postgres compatible export from mysql. (just mysqldump with the --compatible=postgresql option)

http://www.fuzzwork.co.uk/dump/inferno10/pgsql/pgsqlcompat-inferno1.0.sql.bz2


with the following, it may work for you. Specifically the Convert and Import section.
http://en.wikibooks.org/wiki/Converting_MySQL_to_PostgreSQL

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
diabeteman
Diabete Studios
#10 - 2012-05-30 08:31:17 UTC
Thanks mate :)

I'll try to convert it tonight and if i'm successful I'll make it available on the net.

Initiator and CTO of Eve Corp. Management
Tonto Auri
Vhero' Multipurpose Corp
#11 - 2012-05-30 08:39:28 UTC
Khorkrak wrote:
Thanks Steve - I'll be applying this tonight and appreciate the faster queries. This form would be even faster though for MySQL.

load data local infile 'factions.dat' into table factions
fields terminated by '\t' lines terminated by '\n';

LOAD DATA LOCAL is even further from compatibility with various environments, than extended inserts.
If you want performance evaluation for different insertion methods, you may dig up old SDD threads on archived forum.

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Tonto Auri
Vhero' Multipurpose Corp
#12 - 2012-05-30 08:54:01 UTC
Dug it up for you.
http://community.eveonline.com/ingameboard.asp?a=topic&threadID=1376938&page=1#16

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
diabeteman
Diabete Studios
#13 - 2012-05-30 12:20:15 UTC  |  Edited by: diabeteman
YAY! It worked!

On the wiki page you linked, I found a ruby utility that does the translation between mysql and postgres :)

I made conversions for both postgresql and sqlite3: http://releases.eve-corp-management.org/eve_sde/ You will find the scripts I used to do this (instructions are in the 'config' file).

I hope others will have some use of this Big smile

Initiator and CTO of Eve Corp. Management
Desmont McCallock
#14 - 2012-05-30 12:36:47 UTC
diabeteman
Diabete Studios
#15 - 2012-05-30 12:37:31 UTC
Desmont McCallock wrote:
Doesn't this suffice? http://zofu.no-ip.de/esc10/esc10-pgsql-v1.sql.bz2

this is not inferno. Read my first post ^^

Initiator and CTO of Eve Corp. Management
Desmont McCallock
#16 - 2012-05-30 12:39:34 UTC
My bad.
diabeteman
Diabete Studios
#17 - 2012-05-31 23:14:57 UTC
The conversions I did had encoding problems, I created new clean ones

http://releases.eve-corp-management.org/eve_sde/inferno10/inferno10-psql-2.sql.bz2
http://releases.eve-corp-management.org/eve_sde/inferno10/inferno10-sqlite-2.db.bz2

Please tell me if there is anything wrong :)

Initiator and CTO of Eve Corp. Management
Coreth Oake
Bacon Chip Cookie Ice Cream Sammich
#18 - 2012-06-03 09:53:58 UTC
All the identifiers in the psql conversion are quoted, which makes them case sensitive. This is a problem because to reference case sensitive identifiers you have to use quotes everywhere, for example, SELECT * FROM invTypes doesn't work, you would have to use SELECT * FROM "invTypes" instead.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#19 - 2012-06-03 18:15:31 UTC
The next SDE what comes out, I'll see about a PostGres conversion too. No promises, but it's likely enough to annoy me into doing it.



Anyway, for the sensitivity issue, why don't you just:

ALTER TABLE "invTypes" RENAME TO invtypes;

?
There should be an easy way to list all the tables, and if it's a select statement, then you can use that statement to generate all the sql you need to run.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Coreth Oake
Bacon Chip Cookie Ice Cream Sammich
#20 - 2012-06-04 14:49:45 UTC
Steve Ronuken wrote:

Anyway, for the sensitivity issue, why don't you just:

ALTER TABLE "invTypes" RENAME TO invtypes;


The problem is it affects all the identifiers, including column names too. I am sure it would be possible to write some script to fix it, but at that point I may as well do the conversion from scratch for a similar amount of *effort*.
12Next page