These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Information Portal

 
  • Topic is locked indefinitely.
123Next pageLast page
 

New dev blog: Changes to Toolkit Exported Data

First post First post
Author
CCP Guard
C C P
C C P Alliance
#1 - 2012-05-03 16:41:49 UTC  |  Edited by: CCP Navigator
Our EVE 3rd party development community is simply awe inspiring and now that I've gotten that out of my system, I recommend all 3rd party developers go here and read this dev blog about some changes we're making to the EVE 3rd Party Toolkit.

CCP Guard | EVE Community Developer | @CCP_Guard

Cathrine Kenchov
Ice Cold Ellites
#2 - 2012-05-03 16:46:23 UTC
cool stuff
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#3 - 2012-05-03 16:47:02 UTC  |  Edited by: Steve Ronuken
/me cries quietly about more conversion work Blink

Yay! Time to learn something new. (that's actually a positive yay, rather than a sarcastic one)



Now to wait for the update so I can download it.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Akrasjel Lanate
Immemorial Coalescence Administration
Immemorial Coalescence
#4 - 2012-05-03 16:49:11 UTC
Cool but short

CEO of Lanate Industries

Citizen of Solitude

Packtu'sa
Nabaal Construction and Industrials Corp
Nabaal Syndicate
#5 - 2012-05-03 16:54:06 UTC
Can you elaborate on the decision to change to YAML? My understanding of the format (having used it before) is that it's appropriate when the information needs to be both human- and machine-readable. Configuration files are an obvious candidate. Static data, on the other hand, seems more appropriately stored in a database or at least a table format (like CSV). YAML is a lot of key/value pairs and arrays. Why was it chosen to represent tabular data?
Droxlyn
School of Applied Knowledge
Caldari State
#6 - 2012-05-03 16:59:57 UTC  |  Edited by: Droxlyn
I'm confused as well.
Why not keep the data in the database and export it at each release for the client to use?

For those that do not want to change, it'll just require ramming it back into the database somehow.
ctx2007
Republic Military School
Minmatar Republic
#7 - 2012-05-03 17:05:44 UTC  |  Edited by: ctx2007
sixth, I guess as module names change and more are added then changes to the toolkit are needed too. Hmm evolution.

You only realise you life has been a waste of time, when you wake up dead.

darmwand
Stay Frosty.
A Band Apart.
#8 - 2012-05-03 17:14:54 UTC
Aaah, this is awesome! Thanks a lot, this should make it much easier to deal with EVE data, using YAML is definitely the right thing to do.

"The pen is mightier than the sword if the sword is very short, and the pen is very sharp."

Droxlyn
School of Applied Knowledge
Caldari State
#9 - 2012-05-03 17:40:07 UTC
Samples for those too lazy to open it up:

graphicsIDs.yaml wrote:

12:
description: Stargate
graphicFile: res:/Model/Jumpgate/Caldari/cj2/cj2.blue
obsolete: true


typeIDs.yaml wrote:

5:
graphicID: 6
6:
graphicID: 1015


I hope they find more value in later YAMLs. These could have been done just as well with CSV files.
Antihrist Pripravnik
Cultural Enrichment and Synergy of Diversity
Stain Neurodiverse Democracy
#10 - 2012-05-03 17:59:53 UTC
Never heard of YAML, but I've heard (and worked) a lot with JSON. Why YAML and not JSON?
Thebriwan
LUX Uls Xystus
#11 - 2012-05-03 18:08:56 UTC
I truly don't understand why anyone would use this Yam-something over XML but that is not my decision to make.

Where I really see a problem is here:

There are at least two totally different uses cases for the static data:

a) Something like a stand alone application for processing manually some data.

b) A web app processing Millions of datasets and calculation on the fly.

For a) it does not really matter if there are csv oder xml or yam-something files.

But b) needs a database.

And for all the effort you are doing here there is no way to provide sql files anymore?
Two step
Aperture Harmonics
#12 - 2012-05-03 18:10:36 UTC  |  Edited by: Two step
A suggestion:

Before you guys switch over to the new system, how about you release a *full* data dump in the new format? Giving us 2 sample files, especially when it doesn't sound like there will be a 1-1 mapping of old tables to the new files isn't enough for people to build tools to handle the new data.

Edit: I just re-read the blog, sounds like you are *only* changing those two files for Inferno. My request stands for when you change over more stuff.. :)

CSM 7 Secretary CSM 6 Alternate Delegate @two_step_eve on Twitter My Blog

Packtu'sa
Nabaal Construction and Industrials Corp
Nabaal Syndicate
#13 - 2012-05-03 18:18:55 UTC
Antihrist Pripravnik wrote:
Never heard of YAML, but I've heard (and worked) a lot with JSON. Why YAML and not JSON?


YAML is a superset of JSON. That said, I don't think either are appropriate. They can handle tabular data, but that's like saying you can send someone a picture as a Word document filled with comma-separated color values. Yes, the information is there, but it's in the wrong format.

(As far as I know,) EVE static data is tabular, not flexible key/value data like YAML is meant to represent. XML is sort of in between the two.

Honestly, I'm more concerned that CCP is choosing to use YAML internally than that they're forcing us to use it. I guess I shouldn't jump to conclusions before CCP can comment on it, though.
Xarrg
Crushed Ambitions
#14 - 2012-05-03 18:43:56 UTC
How easy/hard to transport them back to sql format ?
I'm sure most of the 3rd party guys will stick with they ms/mysql, so this will be just a extra step for us.
Zagdul
Federal Navy Academy
Gallente Federation
#15 - 2012-05-03 18:50:55 UTC
Item ID please stop using names.


Dual Pane idea: Click!

CCP Please Implement

Vessper
Dark Mason Society
#16 - 2012-05-03 18:51:53 UTC
How long before we stop getting the MSSQL format?
Abdiel Kavash
Deep Core Mining Inc.
Caldari State
#17 - 2012-05-03 18:55:37 UTC
Because relational databases which have decades of research, optimalizations, and development behind them are too mainstream. Let's throw XML at the problem.

Also, we want to make life easy for application programmers, so for the forseeable future we will make you pull half of the data from a DB, and half of it from XML. You know, just to expand your horizons.
Chribba
Otherworld Enterprises
Otherworld Empire
#18 - 2012-05-03 18:56:51 UTC
Vessper wrote:
How long before we stop getting the MSSQL format?

This?

I'd probably be looking at converting it all back to db since that's what I prefer myself though.

/c

★★★ Secure 3rd party service ★★★

Visit my in-game channel 'Holy Veldspar'

Twitter @ChribbaVeldspar

CCP Solomon
C C P
C C P Alliance
#19 - 2012-05-03 19:01:31 UTC
As some have correctly noted, the reason for the split format delivery is due to an internal process change in how we manage static data, both during authoring and at run-time. This is a gradual migration effort that will see more and more portions of the static data dump delivered as YAML files. There is currently no date for when we will stop delivering the database dump, rather it will occur when there is no data left in it.

We will be delivering up to date samples of the YAML files and data base dumps ahead of each major release (assuming the data has changed) to give 3rd party developers a chance to update their tools.

These will be posted on the EVE Toolkit page.

CCP Solomon | Technical Producer | EVE Online @RoryAbbott

CCP Solomon
C C P
C C P Alliance
#20 - 2012-05-03 19:03:48 UTC
Chribba wrote:
Vessper wrote:
How long before we stop getting the MSSQL format?

This?

I'd probably be looking at converting it all back to db since that's what I prefer myself though.

/c


Yes, we absolutely encourage this and anticipated it to a certain extent, the 3rd party developer community are a resourceful bunch.

CCP Solomon | Technical Producer | EVE Online @RoryAbbott

123Next pageLast page