These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Can we get the SDE in SQLite and forget YAML?

First post
Author
Previous Topic Next Topic
Zifrian
The Frog Pond
Ribbit.
#1 - 2015-07-18 01:44:19 UTC  |  Edited by: Zifrian
I was looking to see if the newest DB version was updated and noticed something...

Quote:
YAML

This data is in the YAML standard and is generally the new way data is created at CCP for EVE. This data includes such things as blueprints, certificates, graphicIDs, iconIDs, typeIDs.

SQLite

We also include an SQLite DB file for universe data. Internally this data is in YAML but considering we package this into an SQLite file for our own use it was decided that was a better format than the thousands of YAML files. Specifically this is the universeDataDx database file.

YES, it IS a better format. Considering it doesn't look like you'll release the universe data in YAML anyway, then why don't you convert the YAML (and other tables) to SQLite for the SDE and call it a day? There are free SQLite editors out there that can run in minutes instead of a 20 minute install of MSSQL Server and no one needs to find a YAML library, learn how to import it, deal with the quirks in the data, and then after all that finally get to use it in a database.

This change to move to YAML was 3 years ago (http://community.eveonline.com/news/dev-blogs/changes-to-toolkit-exported-data/) and we really haven't progressed much. I know there are benefits for Linux and other non-MS developers but it's taken 3 years to get a few tables in YAML. Considering the "thousands" of files in the Universe database, it doesn't look like this is going to be a reality anyway?

When this was released we all kind of groaned a bit because it's another hoop we have to jump through to get the basic static data we all use. I was a bit annoyed when I read this comment in that devblog thread by CCP Solomon when he replied to someone saying they were going to convert the data into something usable:
Quote:
Yes, we absolutely encourage this and anticipated it to a certain extent, the 3rd party developer community are a resourceful bunch.
Yeah, and we all have RL jobs and other stuff too. After three years of deal with this hodgepodge system, is there any plan to consolidate and finalize the SDE into something we don't need to be "resourceful" to do?

Every few months of updating apps to deal with changes and missing data with the SDE is getting really frustrating. I really appreciate having it, but the days of just having it all in the MSSQL file were great compared to now. It seems you already have the capability, so can we switch for good?

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Dragonaire
Here there be Dragons
#2 - 2015-07-18 22:34:10 UTC
Or maybe you just need to spend 5 minutes on this forum to find where someone already has a tool you can use or his all ready done conversions to every other common DB format including SQLite. As to going to something other then Yaml I wouldn't expect that to happen since CCP is internally switch everything over to Yaml on their servers as well and they start releasing it to the third party developers that way when they make to switch themselves. BTW since you don't seem to have the time IRL to take the 5 minutes it appears to find this I took an addition 15 secs out of mine to find the link to the conversions for you https://forums.eveonline.com/default.aspx?g=posts&t=433747&find=unread enjoy.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Zifrian
The Frog Pond
Ribbit.
#3 - 2015-07-19 00:15:59 UTC  |  Edited by: Zifrian
Dragonaire wrote:
Or maybe you just need to spend 5 minutes on this forum to find where someone already has a tool you can use or his all ready done conversions to every other common DB format including SQLite. As to going to something other then Yaml I wouldn't expect that to happen since CCP is internally switch everything over to Yaml on their servers as well and they start releasing it to the third party developers that way when they make to switch themselves. BTW since you don't seem to have the time IRL to take the 5 minutes it appears to find this I took an addition 15 secs out of mine to find the link to the conversions for you https://forums.eveonline.com/default.aspx?g=posts&t=433747&find=unread enjoy.

You apparently missed my point. If it doesn't look like they aren't going to switch the universe files to yaml, then why keep going with it? If they already are converting the data, then why can't we have it from the source?

I know about the conversions, I've been doing this for several years. I don't like to rely on other players for my apps because I've lost a lot of time when someone stopped developing their product and I had to fix my app for it. Especially when I've been able to do fine without conversions for years.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Desmont McCallock
#4 - 2015-07-19 07:12:11 UTC
Zifrian wrote:
Dragonaire wrote:
Or maybe you just need to spend 5 minutes on this forum to find where someone already has a tool you can use or his all ready done conversions to every other common DB format including SQLite. As to going to something other then Yaml I wouldn't expect that to happen since CCP is internally switch everything over to Yaml on their servers as well and they start releasing it to the third party developers that way when they make to switch themselves. BTW since you don't seem to have the time IRL to take the 5 minutes it appears to find this I took an addition 15 secs out of mine to find the link to the conversions for you https://forums.eveonline.com/default.aspx?g=posts&t=433747&find=unread enjoy.

You apparently missed my point. If it doesn't look like they aren't going to switch the universe files to yaml, then why keep going with it? If they already are converting the data, then why can't we have it from the source?

I know about the conversions, I've been doing this for several years. I don't like to rely on other players for my apps because I've lost a lot of time when someone stopped developing their product and I had to fix my app for it. Especially when I've been able to do fine without conversions for years.
Zifrain, this is always the case. There will always be something that will brake your app. CCP started releasing the SDE as an MSSQL dump, when they changed that they broke your app. But someone come along and build a tool to compensate on the change so your app can continue to work without changing it. If that someone stops maintaining that tool your app will return to the braking state it was originally. Bottom line, as long as there will be changes, your app will always face the "breaking" threat.
Zifrian
The Frog Pond
Ribbit.
#5 - 2015-07-19 10:31:07 UTC  |  Edited by: Zifrian
Desmont McCallock wrote:
Zifrain, this is always the case. There will always be something that will brake your app. CCP started releasing the SDE as an MSSQL dump, when they changed that they broke your app. But someone come along and build a tool to compensate on the change so your app can continue to work without changing it. If that someone stops maintaining that tool your app will return to the braking state it was originally. Bottom line, as long as there will be changes, your app will always face the "breaking" threat.

Oh I totally get that, change is inevitable. A library that works today might not work tomorrow. I have updated my app quite a bit due to many changes (aka Crius). But with the SDE, it's pretty much a fundamental way to build 3rd party apps. Without it, you really can't do that much but the current system is relying more and more on Steve and others to produce conversions for it's use. I think that's a great service he provides but what if he stops playing? Now people that want to do 3rd party development have to find someone else to do that or he or she can't use the SDE? We all have to write/update our own programs to import the new changes every 6 weeks?

I guess my frustration boiled over a bit with this last set of changes but I honestly was just asking if there is a better way. It seems they can and will put the data into a pretty usable format that has easier access than YAML and I was suggesting/hoping we could get it that way in the future.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Desmont McCallock
#6 - 2015-07-19 10:53:51 UTC
Maybe you should drop in at this conv.
SJ Astralana
Syncore
#7 - 2015-07-19 12:28:47 UTC
I'd much prefer deserializable yaml to any other option on the table at this point. SQL isn't coming back and any anemic variety of SQL would be a step backwards. If I can paste a document into a POCO generator, I can easily deal with changes as fast as they come.

Hyperdrive your production business: Eve Production Manager

Zifrian
The Frog Pond
Ribbit.
#8 - 2015-07-19 14:11:15 UTC
Desmont McCallock wrote:
Maybe you should drop in at this conv.

Thanks for that. I didn't know those sorts of conversations happened on github.


Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#9 - 2015-07-19 17:28:18 UTC
As a side note, I'm busy working up a toolset to go from YAML to any database sqlalchemy (python thing) supports. And it's on github, so if I were to, say, get hit by a bus, people could take care of it themselves.

I'd expect to see more moving into yaml from CCP's side (maybe even the universe data.) and then people having the tools to deal with the bits they actually need.

Where an old form existed, I'm mirroring that, when it's something new, I'm creating a format which seems to match the yaml closely (like with the industry data. As the new version doesn't really fit the old form, I created a new table structure)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter