EVE Information Portal

Hmm. Gulp.

cries a bit with confusing and worry.

Cards on table - I'm not a coder. Never have been and tried repeatedly since line numbers were required.

I enjoy designing spreadsheets and messing about with sql and then Access to get the queries and make tables to import and use as pivot tables on Excel.

Just when I've finally got Eve Industrial Organiser to the point where it's very very easy to keep it updated in a few key strokes (the hardest part is remembering how to restore the backup database each patch release as I only do it for the data dumps), you're going to change it to the point where it doesn't seem like query look up tables is going to be an option straight out of the box.

Perhaps it will make coding easier to understand? Maybe I'll lose the fear factor and just start 'getting' it - like I eventually did for excel and access.

Worst case scenario I stop updating EIO, best case scenario - I figure it all out and produce an exe version that doesn't require the use of Excel 2007 which will make more than a few potential users very happy.

They can: it's just that you're limited to either pre-defining the list length (or number of children) ahead of time, or living with a table definition which can change at runtime (which, of course, has it's own set of challenges*).

* Hint: Views are your friend if you do this, as a View can effectively maintain a "static" view of a table even if it suddenly starts adding columns to itself at runtime.


Just curious: one of the advantages of an MS-SQL dump is that it's trivial to import into Access or Excel and run some basic analytics. Does anyone know of a good YAML-to-relational database integration tool that's freeware? Even an industrial-grade integration tool like Informatica or Cast Iron would be good, so long as they were free.

Thank you CCP for making my head spin xP

Now you got me thinking on Plain Text databases, MySQL & RDMS's, anddd stuff like YAML, Reddis, and MongoDB...

There are so many things that make this way too complicated for me to understand at face value. I wish they had a index of what makes a database good for management instead of "Hey you, go write in 15+ different database wrappers so you can figure out specifically which one is best suited for your specific application" xD

So who's going to be the hero of Weaksauce Developers such as myself by converting everything back to an SQLite database?

If that can help some people : https://forums.eveonline.com/default.aspx?g=posts&m=1362009

I have to vent some anger over this too :p


I totally understand AND applaud the move away from a database dump. The MSSQL datadump means it's basiclaly only accessible to people running Windows. MOving away from that and to a "plain text" format is the right thing to do. For this same reason lots of interchange text formats are being used to make different systems communicate with eachother.



However, I do have worries about yaml.
The yaml format is now over 10 years old, and yet, it's not really seeing a whole lot of traction. The available tools for dealing with yaml are in a pretty sorry state. I tried several libs and most can't even handle the extended sample file.
With the state of things, it looks like I'll have to make my own yaml parser just to get the features/performance/memory use I need.
XML would have been my personal preference as an interchange format (much more mature, LOTS of support tools available, and this really nifty/elegant thing called XSLT), but I'm not really going to debate that issue.

What I AM worried about however is the complete lack of data description. With the DB, you at least could query the database for the table colums, type, nullability which in turn makes it easy to prepare your own internal structures. Yaml is completely form-free. And this is a HUGE disadvantage for what most people use the eve datadump for. Form-free data has it's uses, tabular data ISN'T one of them.

With how YAML is now, you need to read the ENTIRE yaml file into memory, just to be able to figure out what all the columns are in the table. YOu also need to read the entire yaml file into memory just to have a rough idea of the datatypes. If you want actual datatypes, you need to iterate over the entire table and data and infer actual types from the basic types (int, float, string) yaml provides. And even then, we only know what IS there, not what COULD be there in some future version of the data dump.

So you read a yaml file, see that one field called humptyDumpty is an interger type in yaml, and by looking at all the values you find, you see no value is larger than 255. SO you infer it's a 1 byte unsigned integer. Then in the next datadump more data is added and oops, your internal table no longer works. Is a string going to need unicode ? or will ANSI/ASCII suffice ?

You stated it yourself, if you want performance, you don't want to use the yaml "as is" in a dictionary type associative array. It is too slow to load, too memory wasteful. If you need to convert, then knowing the correct "record" format up front is a huge benefit.



So here are my 2 big questions in this issue...
1° Can we expect to get "easily computer-parsable" descriptions with decent types of each of the yaml tables ?

2° With the poor state of yaml parsers out there. Can we get a guarantee about what features of yaml you guys will be using? If you have to make your own parser (or pick one that has the features you're going to be using), then the plain "unordered key/values blocks" / associative array/dictionary thing as used in the current 2 yaml files is easy enough to handle. Some of the other yaml features are... quite complex to parse.



Semi related question: Are bugs/inconsistensies in the datadump something we should report in the normal bug forum ? Or is there another place for those ?

CREST will return data in a JSON format, so yes (sort of). It's something for the future, though.

-GK