EVE Information Portal

Meh, JSON is more readable and friendlier than YAML

Many thanks for the detailed explanation, sounds like it has the potential for significant benefits, which makes it easy to accept the additional work it'll need.

Though a 3rd party community project to script this back into SQL tables would be awesome (and I suspect far better than what I will otherwise cludge together on my own!).

where do I start..
CCP.. thanks for making my day and ruining my day in the same dev blog. ;) good one CCP
/me looks through all my Stored procs I have made over the years. Shrugs and says.. i guess that I didnt need `em any way.

moving to a noSQL style is a great idea.. not sure about YAML tho.. never had it work properly. ended up chewing more resourse then it was worth.. but its good to see a decentralized approach in the future,

Question: These tables being removed or left in as well. if these tables are getting dropped could you save us OLD timers and give us a sql file with all the inserts for the table so we could use either YAML and or keep SQL that we have grown up.

will we have to write our own tools to put the data back into the SQL database?

So just to confirm, future MSSQL data exports which have had some data converted to the new structure will be missing certain data tables (as these will be provided in the YAML files)? I guess I'm trying to establish if we will continue to get a full (as in, the pre-Inferno schema) SQL export until you've finished this project or we need to start working on partial conversions now.

Dont get me wrong. Its great what you are doing.. But (and there is always a butt!) until all the data is in the new format. this method is going to be a pain/. Part data in one and part data in the other. us old timers will have to Re-import (or god forbid not update) the YAML data into MS-SQL so that or functions/Stored Procs/SQL goodness will still work.

there would be no point moving to a new data struture until ALL static is moved to YAML format. Also this will cause coding issues every time a new YAML port is released..

I would still release the complete SQL database whole until the 100% of the static data is released. that way we wont have to do code changes EVERY TIME.. only once.

I would rather spend a week converting Stored Proc's then spending a day here and there until 100% Statics data is converted to YAML.

I am not saying I dont want YAML... I am saying.. I would rather do it at one go then every month. most people who have code like mine will have to import back into SQL to keep stuff working until the eventual day when there is no SQL at all

The plan is to drop the migrated tables from the data dump with every release. We know that this is difficult but it will add a lot of overhead to insist that while moving over to a more flexible data format, that we maintain backwards compatibility to a completely separate representation form – in the end this would make us less flexible and able to take advantage of the benefits of the new format.

Unfortunately there are so many systems and so much static data in Eve that any attempt to do them all at once would be a multi-month effort that would be doomed to failure because we wouldn’t have worked through all the problems and issues while trying to apply the solution. We would also cause all feature development to stop, and break all of the tools that we use in day to day development, while likely introducing issues into every single game system. This just isn’t a practical option for us or for you.

*sigh*

I don't even know where to begin...

First...

YAML: YAML Ain't Markup Language

... come on.. really? I'm seriously, physically rolling my eyes at this.

I've very quickly just skimmed over the what the structure is about. So f'n not impressed.

Cons I see....

- First, looking at the "yaml.org" website, it looks like it was coded by a five year old with limited knowledge of anything to do with a computer, let alone design a new type of data structure. Designed in something pre-Netscape Designer. Doesn't ooze a lot of professionalism and confidence towards code base (If there is a "code base" behind specifications of a data structure) and functionality and theory behind the actual concept of the data format, really, nor does it raise any kind of confidence behind who the designers of this data format are when this has been around since 2001. (Yes it was a run-on sentence - sorta) However, I'll give credit where credit is due and note that they did use an external style sheet... which reading on down the code looks like the page was generated anyways. Makes me wonder if the site itself is read from a YAML file?

- Second, just like XML, JSON, and any other non-managed database system that doesn't rely on an index of sorts, one must read all data, or at least to the point where the data you want exists while assuming the data is sorted, from top to bottom, to get that bit (literal) of information to determine whether or not typeID 21471 is a Published object. What a waste. Don't get me wrong. Both have their place. Exchange of clear, described data to be put somewhere. I know massive XML documents float around at times, exchanging hands from one type of system to another, but that XML file isn't used as a "lookup information" source 99.995% of the time.

- The volume of data within EVE... Looking at the SQLite database conversion from Crucible, its over 200meg in size. Thats with packed data (IE: 10 character numbers in 4 bytes of data), indexing, structure definitions, page files, etc. A query to pull any data from anywhere in that file takes MILLISECONDS worth of time (Just timed it, 19ms to find out it is published). Text files? Too large to handle. I'd have to read thousands of lines to GET to that point.

- Not sure I'm too keen on the whole idea of just taking data out of the existing MS SQL backup and dropping them into text files to be re-consumed. I'd ask that all data exists in the MS SQL data and slowly roll out the new YAML files as you massage out the structure you want. Then when all tables are done, then drop MS SQL

- Some of this structure looks similar to Windows INI files.... 'Cept, headers are marked with an identifier followed by a colon instead of a [identifier] type of ordeal. I do acknowledge there are advancements in comparison to the INI format, but not much more.

Pros...

- No MSSql - Although I started off with training against MS SQL 2000 Enterprise, I've moved very far from it simply due to costs. Yes, I know its free NOW, but it wasn't like that until recently, and I've never looked back. I might have been an MSSql fanboi if there were always free versions. I'm cheap.

- Take the generalized data and put it into a proprietary structure our applications work with. MSSql, MySQL, SQLite, CSV, our own structure of data (I'm looking at you EVEMon! {wink}) or whatever we want is a GREAT bonus.

Final thoughts

Of course, all we (us?) developers are going to have to follow your lead if we're going to keep developing our tools for your game, but honestly, I've never been, never will be, a complete fan of single or multiple text files that is supposed to relay some sort of structured data. I avoid creating XML, I avoid CSV, I avoid plain text simply because repetitive reads of data slows the whole process down, ESPECIALLY when you get into thousands of lines.

With all the enhancements you ladies and gents at CCP have been putting into improving UI response times, I'm quite thrown back that you'd go to a text file to manage database worthy information, static data or not.

{30 minutes later}

... come to think of it... YAML originated in 2001 and has had pretty much NO MOVEMENT since then... and you're using it for in-house processes and implementing it as a data store in half way though 2012?!?!?!

Following up on Katrina, go play with Couchbase (not CouchDB), it's just as sexy as redis.

In other news, **** YAML. Royally. I'll convert it to a plethora of formats since, like I said - **** YAML.

So if i understood right, more and more data will be available on the client in YAML format ?

Or, will we still have to rely on the toolkit ?

Big thanks to all devs that replied with a lot of technical stuff! I can see the future now