These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

WTF eve toolkit (YAML version) has artifacts!!

First post
Author
Previous Topic Next Topic
crener
Caldari Provisions
Caldari State
#1 - 2015-07-26 21:45:20 UTC  |  Edited by: crener
I decided to write a parser to get the information out of the YAML version of the eve toolkit (https://developers.eveonline.com/resource/static-data-export) now that im almost done with it i notice that multipul blueprints have square bracket were numbers should be.

Yes, i could write try, catch blocks but that would bloat the code significantly and for what 10 (yes 10) out of 3,288.

99.997% are the same way why would change it for 0.003%!


http://imgur.com/jme3usp

the ones that have an issue are:
2738
35684
35772
35773
53780
35782
35791
35792
35796
35797

Please fix them CCP
SJ Astralana
Syncore
#2 - 2015-07-26 21:58:54 UTC
The yaml is valid. You're going to have to suck it up and support empty arrays.

Hyperdrive your production business: Eve Production Manager

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#3 - 2015-07-26 22:10:31 UTC
parsing the YAML yourself is, generally, a mistake. Writing your own parser, for a standard format, is generally a mistake.

Use a library, which takes care of all the little twiddly bits.

YAML's a complicated format. You're just asking for pain.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
crener
Caldari Provisions
Caldari State
#4 - 2015-07-26 22:11:45 UTC
yeah, i get that but why have 10 blueprints with empty arrays when there are 3000 which just leave out the empty fields.

If all blueprints had empty arrays there would be no need for 3 - 4 paths that each blueprint could go making the code nice and slim, but having to support both types just sucks.

I get that there is nothing wrong with the format i just think that it should be adjusted to match the rest of them.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#5 - 2015-07-26 22:12:40 UTC
What language are you using?

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
crener
Caldari Provisions
Caldari State
#6 - 2015-07-26 22:15:13 UTC
Steve Ronuken wrote:
What language are you using?


C#
crener
Caldari Provisions
Caldari State
#7 - 2015-07-26 22:21:12 UTC
Steve Ronuken wrote:
parsing the YAML yourself is, generally, a mistake. Writing your own parser, for a standard format, is generally a mistake.

Use a library, which takes care of all the little twiddly bits.

YAML's a complicated format. You're just asking for pain.


Well i looked at the C++ solutions that were available and couldn't get them to work properly, then decided to learn c# and wrote the parser which is now done (except from research stuff which i don't think i'll need), it just excludes the 10 blueprints.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#8 - 2015-07-26 22:22:02 UTC
Zifrian
The Frog Pond
Ribbit.
#9 - 2015-07-27 02:49:33 UTC  |  Edited by: Zifrian
Someone brought those up in a previous thread and I think CCP said they were there for a reason. Something to do with items that only exist on serenity and do not have an English translation or something.

I couldn't get a library to work well either and wrote (aka hacked at) one too but never had any issues with those type ids. The only reason why I found them is because they violated my type name not null constraint.

But I don't fault you for writing your own, I didn't find many .Net options out there either (Python seems to have a good one though). If I have issues later I'll probably use the one that Steve posted since I didn't know the github one existed until now (couldn't get the code plex one to work either). I did find a great json library though if you need that.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
crener
Caldari Provisions
Caldari State
#10 - 2015-07-27 03:07:45 UTC
Zifrian wrote:
Someone brought those up in a previous thread and I think CCP said they were there for a reason. Something to do with items that only exist on serenity and do not have an English translation or something.

I couldn't get a library to work well either and wrote (aka hacked at) one too but never had any issues with those type ids. The only reason why I found them is because they violated my type name not null constraint.

But I don't fault you for writing your own, I didn't find many .Net options out there either (Python seems to have a good one though). If I have issues later I'll probably use the one that Steve posted since I didn't know the github one existed until now (couldn't get the code plex one to work either). I did find a great json library though if you need that.



well i spent maybe another 20 minutes changing the way that i extract the numbers and now it reports a 0 for the "[]" results. I ran some tests and it imports all blueprints now :). Downside is that i will have to make whole new way of doing it with the typeID to name yaml document, but i can live with that.
Mr Mac
Dark Goliath
#11 - 2015-07-27 07:26:39 UTC
crener wrote:
Steve Ronuken wrote:
What language are you using?


C#

Get YamlDotNet from NuGet. It will make your life easy
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#12 - 2015-07-27 12:44:18 UTC
Mr Mac wrote:
crener wrote:
Steve Ronuken wrote:
What language are you using?


C#

Get YamlDotNet from NuGet. It will make your life easy



Just to confirm, that's the same as https://github.com/aaubry/YamlDotNet Smile

(if you link through the project page, it links to there)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Desmont McCallock
#13 - 2015-07-27 13:29:59 UTC  |  Edited by: Desmont McCallock
I really don't understand Zifrian and crener for not using YamlDotNet. This is exactly the same library my tool uses and I had no problems what so ever with parsing the yaml files. Maybe you both need to take a look at the tool's source code to understand how to use it.
crener
Caldari Provisions
Caldari State
#14 - 2015-08-04 04:05:02 UTC
I decided to look into YAMLdotNet after you guys said that i was being an idiot and after a few days of figuring out how the hell it works i have an alternate parser. I found that there are less anomalies in the dotNet version but it is slower by about 2-4 times. I will probably end up using the dotNet version just because i know that i don't have to adjust my code a little every time that CCP adds a new order of items the hand coded version can anticipate.

I was about to comment about not being able to figure out quite how to extract data from YamlSequenceNodes and about to ask if there was any more documentation about YAMLdotNet, while i was about to comment i looked back at the example that are provided and actually found a solution that works for me. :)

Hmm mabie it has something to do with the time.... i always seem to have breakthoughs between 2-4am.

If anyone is interested i could do some comparisons between my dotNet implementation and my line by line scanning teqnique?
Cryten Jones
Advantage Inc
#15 - 2015-08-15 00:52:38 UTC
Hi,

I am just having a look at this for library for a similar uses as you.

Would you mind giving me a pointer as to how it's used, I can't figure it out from the documentation. I have the yaml file loading into a node but I just can't piece together what I do with it after that !

Thanks for any assistance.
Mr Mac
Dark Goliath
#16 - 2015-08-15 07:27:57 UTC
Cryten Jones wrote:
Hi,

I am just having a look at this for library for a similar uses as you.

Would you mind giving me a pointer as to how it's used, I can't figure it out from the documentation. I have the yaml file loading into a node but I just can't piece together what I do with it after that !

Thanks for any assistance.


https://dotnetfiddle.net/rrR2Bb
Pew Terror
All of it
#17 - 2015-08-20 01:33:54 UTC
Writing a lexer/parser is hard. It's basically programming endgame.

If you want to experiment with parsing your own formats without using existing implementations have a look at proper lexer/parser generators like lex/yacc/antlr. It's really a blast to get into and loads of fun.
crener
Caldari Provisions
Caldari State
#18 - 2015-08-27 14:33:50 UTC
Since i already started this thread i figured i might as well continue posting in here...

Is there a document in the data export that explains what the groupID/marketGroupIDs are (or more specifically what the related names are). There are some old posts that mention it was in the old XML static dump, the newer YAML version doesn't seem to include the info or am i missing it?