These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Information Portal

 
  • Topic is locked indefinitely.
Previous page123
 

Dev blog: EVE Universe Static Data Export

First post
Author
Previous Topic Next Topic
Pyllantra
Corporexu
#41 - 2014-02-15 17:16:08 UTC
Salpun wrote:
Noticed a change in the training queue all partial skills are yellow now, all the time, even if that sorting option is off.

Is this an intended change without patch notes its hard to tell?

I would keep the change, I like it but it does jump out at you if your not expecting it, but if it was not on purpose its a bug which might have other ramifications.

In game bug report not written. I'll submit an out of game one if needed.


Report it!
JP Nakamura
Union of Intergalactic Miners and Nano Assemblers
#42 - 2014-02-15 22:17:42 UTC
CCP Nobody wrote:

...
We use a wide array of files for the Client (and Server), depending on what we need at the time (f.ex cpickles and homebrewed binary format).
And with our build system we can pretty easily create additional files for the static data export, that will get built every time someone changes the underlying data. So there is actually a tiny bit of work that goes into generating the build script for the SDE files, but once that is done, it is all autogenerated.

We decided to provide this dump in sqlite format just because we thought that it would be easy for the 3rd party dev community to utilize that in clever ways, since you guys already use sqlite.


Ah, thanks, I was wondering about this, and I appreciate you being able to "pull back the curtain a little".

The SQL lite files make it much easier for those not on Windows to access and play with the data (I play on OSX and develop on both OSX and Linux).

So now that you've migrated this part of the SDE to this mechanism, would it be safe to assume that eventually (priorities and resources permitting) the rest of the SDE will also migrate over to SQL lite also?

CCP Nobody wrote:

...
Steve Ronuken wrote:

Adding celestials, removing celestials. Changing the links between systems. Adding systems (well, with other changes too.)

This will have more effect on the development process, than on actual deployments, as it'll let them handle the changes in a properly staged fashion, with the changes in data being very easy to document.

Change management in databases is a pain in the ass.

You sir, certanly know your stuff Smile


He certainly does! One of the reasons I use his tools, enjoy chatting with him on IRC, and why he has my support for CSM.

I also urge anyone who's involved in Industry at all to consider supporting him.

CCP: 10+ years of Harvesting players Tears  (latest efforts being Source Limited Edition, and Alliance Logo Revised Policies)
Salpun
Global Telstar Federation Offices
Masters of Flying Objects
#43 - 2014-02-16 16:44:58 UTC
Pyllantra wrote:
Salpun wrote:
Noticed a change in the training queue all partial skills are yellow now, all the time, even if that sorting option is off.

Is this an intended change without patch notes its hard to tell?

I would keep the change, I like it but it does jump out at you if your not expecting it, but if it was not on purpose its a bug which might have other ramifications.

In game bug report not written. I'll submit an out of game one if needed.


Report it!

Sometime posting here gets a dev response fasterBig smile.

bug report submitted.

If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide

See you around the universe.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#44 - 2014-02-17 03:55:43 UTC  |  Edited by: Steve Ronuken
I've just spent a little more time looking into the data. It turns out there's some data which no longer exists in there, probably because it's tightly tied to location information.

dbo.agtAgents
dbo.agtResearchAgents
dbo.chrBloodlines
dbo.chrfactions
dbo.crpNPCCorporationDivisions
dbo.crpNPCCorporationResearchFields
dbo.crpNPCCorporationTrades
dbo.crpNPCCorporations
dbo.ramAssemblyLineStations
dbo.ramAssemblyLines
dbo.staStations


Any chance of getting this data again?

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
CCP Nobody
C C P
C C P Alliance
#45 - 2014-02-17 09:24:58 UTC
Steve Ronuken wrote:
I've just spent a little more time looking into the data. It turns out there's some data which no longer exists in there, probably because it's tightly tied to location information.

dbo.agtAgents
dbo.agtResearchAgents
dbo.chrBloodlines
dbo.chrfactions
dbo.crpNPCCorporationDivisions
dbo.crpNPCCorporationResearchFields
dbo.crpNPCCorporationTrades
dbo.crpNPCCorporations
dbo.ramAssemblyLineStations
dbo.ramAssemblyLines
dbo.staStations


Any chance of getting this data again?

Yes, this was an error on my part.
We´ve fixed it and it should be up on the Toolkit website later today.

Thank you for the heads up!
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#46 - 2014-02-17 09:49:53 UTC  |  Edited by: Steve Ronuken
CCP Nobody wrote:
Steve Ronuken wrote:
I've just spent a little more time looking into the data. It turns out there's some data which no longer exists in there, probably because it's tightly tied to location information.

dbo.agtAgents
dbo.agtResearchAgents
dbo.chrBloodlines
dbo.chrfactions
dbo.crpNPCCorporationDivisions
dbo.crpNPCCorporationResearchFields
dbo.crpNPCCorporationTrades
dbo.crpNPCCorporations
dbo.ramAssemblyLineStations
dbo.ramAssemblyLines
dbo.staStations


Any chance of getting this data again?

Yes, this was an error on my part.
We´ve fixed it and it should be up on the Toolkit website later today.

Thank you for the heads up!



Thanks Big smile

(as a complete aside, is there a reason you're in a newb corp?)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
CCP Nobody
C C P
C C P Alliance
#47 - 2014-02-17 10:25:58 UTC
Steve Ronuken wrote:

Thanks Big smile

(as a complete aside, is there a reason you're in a newb corp?)

After enduring the long and extremely painful hazing ritual, I´ve now been accepted as a member of the CCP corp.
Lukas Rox
Aideron Technologies
#48 - 2014-02-17 15:46:58 UTC  |  Edited by: Lukas Rox
Steve Ronuken wrote:
/me buries his head in his hands. More stuff to convert.

Wonder if anyone would notice if I just sneak in the old tables Blink

Edit:

On reading:
\o/

All looks good to me. I'll probably do conversions to mysql and postgres for it, and possibly a combined sqlite DB.

I'll get onto that conversion later today, when I'm at home.


I don't mind having to convert the data myself, but IMHO maintaining SDE in three different formats is stretching it. If Steve and a bunch of others make conversions and everyone has a slightly different approach to these additional (yaml, sqlite) tables, we (the third party devs) get a fragmented community, because devs A,B,C will use Steve's conversion, D and E will use someone elses and F will use mine.

Any chance this could be converted in-house by CCP to at least one common format? It could even be MSSQL 2012 (although SQLite would be preferred). We (the third party devs) will do the rest. It's ok to have people convert the data to as many formats as required, as long as there is one common db schema.

Especially that according to CCP Nobody it's all automated.

Proud developer of LMeve: Industry Contribution and Mass Production Tracker: https://github.com/roxlukas/lmeve | Blogging about EVE on http://pozniak.pl/wp/
Drailen
Doomheim
#49 - 2014-02-22 22:28:13 UTC
CCP Nobody wrote:
Steve Ronuken wrote:
I've just spent a little more time looking into the data. It turns out there's some data which no longer exists in there, probably because it's tightly tied to location information.

dbo.agtAgents
dbo.agtResearchAgents
dbo.chrBloodlines
dbo.chrfactions
dbo.crpNPCCorporationDivisions
dbo.crpNPCCorporationResearchFields
dbo.crpNPCCorporationTrades
dbo.crpNPCCorporations
dbo.ramAssemblyLineStations
dbo.ramAssemblyLines
dbo.staStations


Any chance of getting this data again?

Yes, this was an error on my part.
We´ve fixed it and it should be up on the Toolkit website later today.

Thank you for the heads up!


Any news on this fixed export? The toolkit still points to the 1.1.2 version and the link in this thread is the broke version.
Zifrian
The Frog Pond
Ribbit.
#50 - 2014-03-04 22:40:14 UTC  |  Edited by: Zifrian
Lukas Rox wrote:
Steve Ronuken wrote:
/me buries his head in his hands. More stuff to convert.

Wonder if anyone would notice if I just sneak in the old tables Blink

Edit:

On reading:
\o/

All looks good to me. I'll probably do conversions to mysql and postgres for it, and possibly a combined sqlite DB.

I'll get onto that conversion later today, when I'm at home.


I don't mind having to convert the data myself, but IMHO maintaining SDE in three different formats is stretching it. If Steve and a bunch of others make conversions and everyone has a slightly different approach to these additional (yaml, sqlite) tables, we (the third party devs) get a fragmented community, because devs A,B,C will use Steve's conversion, D and E will use someone elses and F will use mine.

Any chance this could be converted in-house by CCP to at least one common format? It could even be MSSQL 2012 (although SQLite would be preferred). We (the third party devs) will do the rest. It's ok to have people convert the data to as many formats as required, as long as there is one common db schema.

Especially that according to CCP Nobody it's all automated.

I'd like to second and third this. I use the ms SQL in my import program and when you started talking about using ymal, I got annoyed. Now you are making a separate export for universe data? SQLite is easy, free, and afaik can work across multiple operating systems. I'd rather not have to convert data into table form and just be able to learn one universal db system and go from there. All of us are developing this stuff for free and in our free time. While no one is forcing us to do that, constantly changing up the db system for export adds a lot of tedious work and sucks happiness.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Alicina Goodbody
Posique Corp
#51 - 2014-03-17 15:27:09 UTC
Is this the change that reduced our in game transaction logs to 30 days?
Previous page123