These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Proteus SDE conversions

First post First post
Author
Previous Topic Next Topic
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#1 - 2015-01-13 15:50:19 UTC  |  Edited by: CCP Phantom
Slivo
Beehive Surveillance
X877.
#2 - 2015-01-13 17:58:11 UTC
Thanks for your work

I'd like discuss about it if you got a moment
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#3 - 2015-01-13 21:03:52 UTC
Public or private? Smile

Feel free to ask whatever you want.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Slivo
Beehive Surveillance
X877.
#4 - 2015-01-13 21:15:01 UTC
We are thinking about reworking the EveHQ SDE import to use directly the SDE from CCP.
I'd like to know how you are going from the SDE to your .db file and see if we can do something (Or at least understand it better)

On the other hands, we spotted few strange things in the SDE, and i'd like to talk about it to someone that may help me to undestand
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#5 - 2015-01-13 21:23:40 UTC  |  Edited by: Steve Ronuken
The process has, thankfully, simplified somewhat. (it used to be more involved. Mostly because the migration toolkit choked)

I import the backup into a copy of sql server express, then use the migration toolkit in the Mysql Workbench to convert both that and the sqlite file into 2 mysql databases.

I export them from the windows box I did that on, then import them into the same mysql database on a linux box.

I then process the yaml files with some php. After that, I use a script to convert the database into the sqlite version.

A proper ETL tool would do it faster and better, but I'm not really wanting to spend the money on one, when I have a working toolchain that doesn't take that long, or that much attention. Right now it's mostly 'kick a stage off, then leave it'

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Slivo
Beehive Surveillance
X877.
#6 - 2015-01-13 21:34:11 UTC
It sounds painfull. I guess we will think about it twice x)

Did you notice the "????YC117???" entries in the proteus SDE ?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#7 - 2015-01-13 22:04:39 UTC
Slivo wrote:
It sounds painfull. I guess we will think about it twice x)

Did you notice the "????YC117???" entries in the proteus SDE ?



Yup. They're actually chinese characters. I suspect for Serenity. When used with a utf8 compatible server, they should work. (if not, it's a conversion problem from mysql to sqlite.)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Slivo
Beehive Surveillance
X877.
#8 - 2015-01-13 22:53:33 UTC  |  Edited by: Slivo
Oh, I see.

It gave us a bad heartbeat. We used a awfull workaround to remove them and make the data import work, until we know what it really was. Is it normal those data are included in our SDE ?
Zifrian
Federal Defense Union
Gallente Federation
#9 - 2015-01-14 01:18:13 UTC
Not sure if you are interested in SQLite, but that's what I use and thought I'd offer up my experience.

I wrote a DB builder in VB.net for IPH. I import the data into MSSQL and then run the yaml updates to that database. After that I build what I need into a SQLite file. I really like SQLite over any other DB I've used. It's really fast and easy to export for use. I also found a fantastic management tool that is free for SQLite.

Let me know if you want to know more.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Zifrian
Federal Defense Union
Gallente Federation
#10 - 2015-01-14 04:11:09 UTC  |  Edited by: Zifrian
Steve Ronuken wrote:
Slivo wrote:
It sounds painfull. I guess we will think about it twice x)

Did you notice the "????YC117???" entries in the proteus SDE ?



Yup. They're actually chinese characters. I suspect for Serenity. When used with a utf8 compatible server, they should work. (if not, it's a conversion problem from mysql to sqlite.)

Curious why they are in there actually.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Slivo
Beehive Surveillance
X877.
#11 - 2015-01-14 11:36:54 UTC
Zifrian wrote:
Not sure if you are interested in SQLite, but that's what I use and thought I'd offer up my experience.

I wrote a DB builder in VB.net for IPH. I import the data into MSSQL and then run the yaml updates to that database. After that I build what I need into a SQLite file. I really like SQLite over any other DB I've used. It's really fast and easy to export for use. I also found a fantastic management tool that is free for SQLite.

Let me know if you want to know more.


It could be really interesting if we want to be able to work from the CCP SDE directly !
We need to discuss about it :)
CCP FoxFour
C C P
C C P Alliance
#12 - 2015-01-14 12:23:37 UTC
Zifrian wrote:
Steve Ronuken wrote:
Slivo wrote:
It sounds painfull. I guess we will think about it twice x)

Did you notice the "????YC117???" entries in the proteus SDE ?



Yup. They're actually chinese characters. I suspect for Serenity. When used with a utf8 compatible server, they should work. (if not, it's a conversion problem from mysql to sqlite.)

Curious why they are in there actually.


Everything that gets authored in EVE has to be done here in Reykjavik, submitted, and then deployed to TQ. If the Serenity server wants something specific created for their server it still has to be done here, submitted, deployed to TQ, and then when deployed to Serenity later it will be activated.

In this case there are some special ships that are for the Serenity server only. To help avoid confusion on TQ we made their English translation be the same as the Chinese translation. That way if someone looks up the item on TQ it will be pretty clear it doesn't belong there.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Slivo
Beehive Surveillance
X877.
#13 - 2015-01-14 15:15:57 UTC
Is it the first time chinese characters are added to the SDE ?

Is it also possible for the SDE to be inaccurate on some point ?

Some value in the eve.db released by Steve Ronuken seems wrong, and some entry contains multiples values (both int and float)
Snitch Ashor
Republic Military School
Minmatar Republic
#14 - 2015-01-14 17:37:42 UTC
Thanks for your work. I'm currently using your sql dump to provide updated versions of the ships and modules mysql tables to be used by the eveapi mod for phpbb by Cyerus, however right now I still need to use the yaml files to figure out the ships exact factions (e.g. gurista, caldari navy...) is there any link to this information in the sql that I'm just missing?
Zifrian
Federal Defense Union
Gallente Federation
#15 - 2015-01-14 18:06:02 UTC
Snitch Ashor wrote:
Thanks for your work. I'm currently using your sql dump to provide updated versions of the ships and modules mysql tables to be used by the eveapi mod for phpbb by Cyerus, however right now I still need to use the yaml files to figure out the ships exact factions (e.g. gurista, caldari navy...) is there any link to this information in the sql that I'm just missing?

I do an incredibly hacked and messy bit of processing to set the race id. I think the meta types can help you. I'll look at my code a bit later and send what I have, unless someone has a simpler answer.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Cryten Jones
Advantage Inc
#16 - 2015-01-17 00:33:29 UTC
Can anyone recommend a VBA library for reading the yaml files included in the SDE?

Thanks
SJ Astralana
Syncore
#17 - 2015-01-18 18:12:44 UTC
Zifrian wrote:

I think the meta types can help you. I'll look at my code a bit later and send what I have, unless someone has a simpler answer.


I got this far:

select * from invTypes
join invMetaTypes on invTypes.typeID = invMetaTypes.typeID
join invMetaGroups on invMetaTypes.metaGroupID = invMetaGroups.metaGroupID
where metaGroupName = 'Faction'

Hyperdrive your production business: Eve Production Manager

Snitch Ashor
Republic Military School
Minmatar Republic
#18 - 2015-01-26 11:32:19 UTC
SJ Astralana wrote:

I got this far:

select * from invTypes
join invMetaTypes on invTypes.typeID = invMetaTypes.typeID
join invMetaGroups on invMetaTypes.metaGroupID = invMetaGroups.metaGroupID
where metaGroupName = 'Faction'


Thanks, that's something i was already trying but this wont tell me the difference between let's say Caldary navy faction ships and Guristas.

I think the connection to the factionID is missing.
Hel O'Ween
Men On A Mission
#19 - 2015-02-14 11:08:28 UTC
@CCP: with Tiamat on the horizon already - any idea when the SDE for it will be available to us?

EVEWalletAware - an offline wallet manager.