These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Retribution 1.0 SDE Issues (CCPPrismX twitt version)

First post
Author
Previous Topic Next Topic
Zifrian
The Frog Pond
Ribbit.
#41 - 2012-12-14 12:56:40 UTC
Desmont McCallock wrote:
He already stated that in his last post. On one hand he's justified. On the other hand I'm about to explode by the criticism I receive why I haven't updated EVEMon yet. I'm between a rock and a hard place.

Not to mention that be said he's not the data dump guy anymore either and he's given us more than him.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Golden Gnu
The Golden Gnu Corp
#42 - 2012-12-14 22:55:37 UTC
I hope they learn a lesson: (PrismX + API/SDE) or fail (J/K P)
Anyway, hope they improve the process for next release.
One should think it could be tested and automated pretty easy (one could of course be wrong)

if EveMon is not updated, Eve-Online is not fully updated.
Everyone knows EveMon just as much a part of the game as the eve client IMHO.

Also, I'm sorry if I'm being rude... Oops

Creator of jEveAssets - the asset manager

"Download is the meaning of life, upload is the meaning of intelligent life"
mramo
Doomheim
#43 - 2012-12-15 16:59:27 UTC  |  Edited by: mramo
Golden Gnu wrote:
if EveMon is not updated, Eve-Online is not fully updated.
Everyone knows EveMon just as much a part of the game as the eve client IMHO.


That's why CCP should build their own skilltraining / EVE-monitoring app. EveMon provides a great improvement in playing the game. It is mandatory. CCP shouldn't let some not-controllable third party handle a crucial part of the game.
Dei
Cosmic Core Industries
#44 - 2012-12-15 17:24:06 UTC
That's the opposite of what they should do.


  1. 3rd party developers are free for CCP
  2. CCP developers can focus on the game, not mini applications outside of the game
  3. If CCP developers replicate Evemon, not only have they pissed off the creator, no-one else is going to develop anything knowing that at any time CCP will make an 'official' application
  4. CCP don't have the resources to support all apps in existance, so why should EveMon be the only one?
  5. CCP stated they would support 3rd party developers with the SSD
  6. CREST is apparently somewhere around that would help EveMon
Cyerus
University of Caille
Gallente Federation
#45 - 2012-12-15 18:09:32 UTC
Version 1.0.2 is still missing the slot-layout of ships.
dgmAttributeTypes (or something) seems to be missing the information.
Dei
Cosmic Core Industries
#46 - 2012-12-15 19:43:21 UTC
In ramTypeRequirements, the data interfaces for invention do not have a damageperjob of 0. They should have a damageperjob of 0.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#47 - 2012-12-15 19:49:58 UTC
Dei wrote:
In ramTypeRequirements, the data interfaces for invention do not have a damageperjob of 0. They should have a damageperjob of 0.



This isn't a new thing. Ever since I first imported the dump it's been wrong on interfaces. (Just for reference. would still be nice to have sorted)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Dei
Cosmic Core Industries
#48 - 2012-12-15 21:27:54 UTC
Yeah, was doing work on it today and wondered what people did, and apparently you're all just manually fixing the data you strange people, you. P
Golden Gnu
The Golden Gnu Corp
#49 - 2012-12-16 09:41:08 UTC
@mramo
Yes, because CCP is known to create epic user interfaces - it's their key sell point AFAIK? P
Do not fix what is not broken. EveMon is great, no fix is needed!

@Dei
Everyone fixing it sounds like a giant waste of time. Straight
Join force FFS! Big smile And let the lovely people that convert the SDE to MySQL release it when it's done, please...?

Creator of jEveAssets - the asset manager

"Download is the meaning of life, upload is the meaning of intelligent life"
Harry Pearce
The Intel Project
#50 - 2012-12-16 16:45:28 UTC
As a warning to anyone else bitten by this, not only are ship slots missing from the newest dump, but the eveIcons table has also disappeared. Whilst the iconID column had already been removed from invTypes, the table was present in the initial retribution dump.
This means you need to be aware that trying to join on the iconID column in (at least) chrFactions, crtCertificates, dgmAttributeTypes, invCategories, invGroups, invMarketGroups and invMetaGroups won't work until you've recreated an eveIcons table suited to your needs from the YAML.

There are also some market groups[1] that have hasTypes = 1, but no corresponding rows in invTypes, and one[2] with hasTypes = 0, but no subgroups - possible worth pruning from your personal dumps if you're generating a market tree!

[1] SELECT invMarketGroups.marketGroupID, marketGroupName FROM invMarketGroups LEFT JOIN invTypes ON invMarketGroups.marketGroupID = invTypes.marketGroupID WHERE typeID IS NULL AND hasTypes != 0

[2] SELECT mg.marketGroupID, mg.marketGroupName FROM invMarketGroups mg LEFT JOIN invMarketGroups mg2 ON mg.marketGroupID = mg2.parentGroupID WHERE mg.hasTypes = 0 AND mg2.marketGroupID IS NULL
Raven Seldon
Cambridge Mountain Rescue
The Monarchy
#51 - 2012-12-17 09:27:07 UTC
A bit weird it takes this long.
Will the service be discontinued?
Nik W
Critical Mass Inc.
#52 - 2012-12-17 23:04:56 UTC
Silence is bad Sad

Can we get an update on the status of this?
CCP Prism X
C C P
C C P Alliance
#53 - 2012-12-18 09:29:52 UTC
I'm not ignoring this, I'm just busy with other things. My plan is to do my utmost to get everything I need to have done, and fix the SDE as well, before I go on vacation. Please don't hate me, I'm delicate like a lotus flower in bloom. Sad

CCP Prism X

Programmer on Team Gridlock & Expert Ranter
Callean Drevus
Perkone
Caldari State
#54 - 2012-12-18 10:43:01 UTC  |  Edited by: Callean Drevus
We don't hate you, we are just wondering why this is seemingly so low on CCP's priority list :(

Are new features / meetings really more important than fixing this? I mean, I know they are generally more fun to do than data exporting, but we (including the one whose task it is) will all become very happy when this is done.

It may have been asked before, but why is there not a standard program / script that does this whole dump every time an expansion is launched. I mean, the data structure cannot change THAT much every cycle... and this is not the first time we've had these issues. More like, it happens almost every time an expansion is launched...

Developer/Creator of EVE Marketeer
CCP Prism X
C C P
C C P Alliance
#55 - 2012-12-18 12:42:16 UTC
Can't speak for the company, but this is low on my priority list because I'm actually not responsible for this and would rather finish up stuff first for which I will be held accountable. Not an ideal situation I know, but them's the breaks.

As to why this breaks all the time:
Multiple reasons. Bad authoring + bad integration of static data between branches + bad code in the generation scripts that has never been bad before because it needed the bad authoring + bad integration to actually become an issue, was the first issue.

So I had to make code changes to shield you from broken relations, empty record names and such. And when you change code **** breaks.

And thus I'm really responsible for this. And that's why I'll be making another dump, fixed according to the issues I see here. I was going to wait a few days to sort out all the issues, as I understood from the first few posts that the issues weren't that critical anymore. Just some missing eveUnits records. Apparently that is not the case.

But other stuff comes up. I need to respond to that as well. It's no fun when what you need gets ranked on a low priority but it has to happen somewhere. Otherwise you're not prioritizing. But I was planning on doing this today, that might not work out anymore as I've lost a lot of time between the morning and noon due to unforeseen issues.

Today or tomorrow. Sorry for the delay. If someone would make a conscice bulletpointed list of all the remaining issues, I might consider hugging him next FanFest. Lol
But it's not like it's a huge issue for me to do it myself either. But then you don't get to feel like you helped!

CCP Prism X

Programmer on Team Gridlock & Expert Ranter
Sidrat Flush
School of Applied Knowledge
#56 - 2012-12-18 12:54:46 UTC
We appreciate your "free"? Time and I realise it's a labor of love mixed with a dash of hate to get the SDE to us, I want to understand how the future YAML formatted SDE will be easier to update and release, assuming it will be of course.

What about CREST does it the SDE form part of it or is crest a separate entity?

Its time to stand up against the bad empire based CEO telling falsehoods about what new characters can accomplish and pushing them towards an in game experience of drudgery and loneliness keeping them in the shadow of ignorance for at nest their own profit at worse apathy towards all the experiences that Eve has to offer.
CCP Prism X
C C P
C C P Alliance
#57 - 2012-12-18 13:02:23 UTC
I'm not much involved with the YAML files. I suspect they'll just create more issues, at least for a fair bit. They already have.

I see numerous posts here about people wondering why invTypes doesn't have an iconID, and that's because typeID->iconID and typeID->graphicID relations have been moved into YAML files. So we now have type information split between the restore data and the YAML data.

However, YAML data should be less prone to bullshit incidents such as the data we ship to you not being the 100% current data of the release branch. As it stands we have a code versioning system and a seperate data versioning system, which means the data can be out of synch with the code. Of course we've put a lot of effort into both avoiding that and handling such incidents gracefully so we don't notice this, f.e. all the unnamed types went out to TQ and broke nothing but broke lots of third party apps.

And CREST has nothing to do with the SDE. Just like the API has nothing to do with the SDE. Although obviously any application that uses the API (Legacy or, in the future, CREST) could benefit from having data from the SDE stored locally.

CCP Prism X

Programmer on Team Gridlock & Expert Ranter
Callean Drevus
Perkone
Caldari State
#58 - 2012-12-18 13:10:58 UTC
CCP Prism X wrote:
Can't speak for the company.... But then you don't get to feel like you helped!


My apologies for sounding so bitchy. It's nothing personal Blink. Thanks for both the explanation and taking the time to do this. Bad code is a *****.

As to priorities, I understand why it is not high on YOUR priority list P I'm just not sure why it's not higher on CCP's priority list.

I'll download the current dump and try to see if I find any issues.

Developer/Creator of EVE Marketeer
Muscaat
EVE Markets
#59 - 2012-12-18 13:55:43 UTC
CCP Prism X wrote:
...awesomeness


CCP Prism X, best Prism X.

Thanks for your continued efforts to help us out Smile
CCP Prism X
C C P
C C P Alliance
#60 - 2012-12-18 15:34:56 UTC
We've got a new SDE up guys, get it while it's hot!

IMPORTANT NOTICE:
READ THIS OR YOU'LL GET ANGRY WHEN I MAKE FUN OF YOU FOR NOT HAVING READ THIS!
Now that I got your attention, reading this will also make you angry.
There is no iconID or graphicID in invTypes anymore. You will never find it there again. It has been moved into the types.YAML file we ship with the SDE zip file.
If you're using a third party conversion that does not include this data in one form or another that's your converter doing something wrong.
If you are doing a conversion and want this data back into a relationial database, you're going to have to get it from the YAML file.

Changelog:
invTypes:
Problem: table is missing the iconID column.
Solution: Get them, along with the graphicIDs, from types.YAML. They will not be found in the official backup.

eveUnits:
Problem: unitID 103, 122, 129 is missing.
Solution: These units do not have a display name, do include units without a display name.


dogmaTypeAttributes, dogmaAttributeTypes:
Problem: Missing slot attributes
Solution: Fixing eve units to include the units will retain the slot data rathre than sanitizing it out as a broken relation.

invMarketGroups, invTypes
Problem: Missing records.
Solution: These are unreleased DUST records which do not belong in the SDE.


ramTypeRequirements
Problem: Data Interfaces for invention have a damage per job of 1.
Solution: This is how the data is authored and always has been. Checked the history. I can't fix things that are by design.
Interfaces are special cased in our code. Maybe one day I get to rewrite the clusterintimacy that the S&I backend is.


invMarketGroups
Problem: Market groups exist with hasTypes = 0, but still have types. Market groups exist with hasTypes = 1 but have no types.
Solution: Check these relations and sanitize out bad data. I have no clue when GD will actually fix the data but at least I can fix your SDE

CCP Prism X

Programmer on Team Gridlock & Expert Ranter