These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVESDEToSQL (v3.5.0)

Author
Tristan Agion
Viziam
Amarr Empire
#41 - 2016-04-17 17:43:58 UTC
Hi there!

Thanks for this excellent tool! First, let me say that everything is working fine for me as far as program setup and running is concerned. But I have a problem with some data I get from this.

Using YC-118-3_1.0_117575.zip with "import" followed by "export -csv", I get a bunch of CSV files. Among them is

mapSolarSystems.csv

in which I can find the following systems

10000004;20000047;30000327;D95-FQ;
10000004;20000047;30000328;ZSPJ-K;

where I have just copied the first four columns

regionID;constellationID;solarSystemID;solarSystemName;

Now, if we look at Dotlan, then we see that they are connected by a stargate

Dotlan map

I would expect to see this connection in

mapSolarSystemJumps.csv

Basically, there should be an entry

10000004;20000047;30000327;30000328;20000047;10000004;

which is just a connection from D95-FQ to ZSPJ-K according to

fromRegionID;fromConstellationID;fromSolarSystemID;toSolarSystemID;toConstellationID;toRegionID;

But in fact neither systems is listed here at all (the relevant SolarSystemIDs are not in this CSV file).

I am relatively sure that I'm reading the files right in general, since I get other connections right.

I can see three possibilities:

1. My understanding is false/incomplete, this particular connection is listed elsewhere or in a different manner.
2. CCP's SDE is corrupt concerning this, and Dotlan somehow knows better.
3. The SDE is correct, but something goes wrong in the import or export to CSV with your program.

I was hoping that you could shed some light on what is going on there. I should mention that I have basically zero clue about SQL databases, I just followed the various instructions to set stuff up (apparently successfully, everything seems to be running smoothly). I'm a Matlab kind of guy. Hence the export to CSV, which I can read into Matlab.

I've spotted this problem when I ran a Matlab script to collect all wormholes. My idea was to simply find all systems that have no static connections. That finds all the J-systems and Thera, but to my surprise I found several extras like this one...
Desmont McCallock
#42 - 2016-04-18 07:38:37 UTC  |  Edited by: Desmont McCallock
It's case 2.

From a look into the original SDE data it's missing data for the following constellations in that region also:

constellationID, constallationName
20000047 L5-H1O
20000048 J-4QD7
20000049 LC-AQD
20000050 7PUP-K
20000051 V8UW-M
20000052 T7B-VB
20000053 B-PHYN
20000054 9RW5-Z
20000055 VL-OGL
20000056 VW-JMB
20000057 HN5-RL
20000058 RO-AZT
20000059 KCR-1E
20000060 FL-TD4

So, file a bug report with CCP to have that fixed in the next release.

P.S. Here is some additional info: This issue was introduced with Rhea 1.0 SDE on Dec 2014.
Tristan Agion
Viziam
Amarr Empire
#43 - 2016-04-18 09:30:46 UTC
Thanks for the quick response. I have now submitted a bug report! Actually, I found a lot more of these falsely isolated systems, I just didn't write this up:

'WF-1LM' 'D95-FQ' 'ZSPJ-K' 'U1F-86' 'T-P7A6' 'Y-T3JJ' 'F3R-IA' '74-YTJ' '8-RS3U' 'OVFN-N' '1Q-BBM' 'WXNC-N' 'G-EA07' 'X-L6BO' 'D-PHUA' '3-HXHQ' '18A-NB' '3-J5OQ' 'GYF-GD' 'W-6TS9' 'VIG-VR' 'KX-P5C' 'N-FJBK' '2-4ZT5' 'NVN-6F' '09-8TH' 'TI0-AX' '7O-POM' 'L6Q-SX' 'BFJ-TB' 'ZZ7-L6' 'L-CHVW' 'X0LN-U' 'RQAE-M' '7CO-SA' '4G-E5A' 'A-VWK9' 'JQHP-4' '6Q5K-5' 'P-MVFP' 'J-Z1UW' 'W477-P' 'NQ1-BL' 'K7A-G8' 'HP-PMX' '6BN-K9' 'WLE-PY' 'EH-HXW' 'OS-RR3' 'V4-GZL' '4C-Z91' 'RU-97T' '1S-1V7' 'PE1-R1' 'Polaris' 'JB-007' 'USJ2-M' '7M-RAL' 'LPBU-U' 'RF-342' 'J2V-XY' 'Z-JBTR' 'S-QNXH' 'S94-X8' 'J-YQEC' '8MX-OR' '97YC-C' 'V-AMD5' 'U-JC8X' '1HH3-E' 'DUIU-Q' 'LQH0-H' 'FRW3-2' '9MX-1C' 'IED-4U' 'N-9EOQ' '6F3-TK' '2E0P-2' 'U-ITH5' 'N-4G5L' 'RB-2EA' 'ZK5-42' 'YRZ-E4' 'A3-PAT' 'H55-2R' 'P6-DBM' '9XI-0X' 'Q8T-MC' 'Z-YOJ9' '4T4B-L' 'F-JB3H' 'XBO7-F' 'FI-449' 'UA7-U4' 'PZP1-D' 'R1KE-A' 'JGDF-B' '1SR-HT' 'SQ-2XA' 'Z-FYJR' 'ZA6-9N' 'J1-6CJ' '7H-Z5R' '0RZ5-2' 'A9-NB6' 'LG1-TA' 'TNK-BQ' 'E2AX-5' 'HPE-KP' 'THS-MN' 'UBES-K' 'I-R8B0' 'QIW-TQ' 'WLL-QX' 'BJC4-8' 'PQA-9K' 'S5-U0R' 'CW-R71' 'QO-3LC' '3E-ER7' 'REZ-YZ' 'OU-AIT' 'VYX2-I' '5-P3CQ' 'M-FDTD' '54-VNO' 'IAMZ-5' 'HD3-JK' 'PBXG-A' '9-ERCP' 'KN7M-N' 'Z-D1DW' 'FO-3PJ' '6-QXE6' 'N-FKXV' 'X7-8IG' 'R-G1SF' '6-NCE7' 'WDJQ-G' 'JS3-RS' 'JX-T1W' 'CZ-CED' 'BKK4-H' 'Y-4V7U' 'L-TPN0' '3-XORH' 'G1VU-H' 'W6H6-K' '6-23NU' 'DVAR-P' 'J-JS0D' 'VR3-PS' 'LH-J8H' 'I9D-0D' 'HGB-C6' '2L5-FI' 'RS08-B' '4U-14I' 'H-EDXD' '8-ULAA' 'KF1-DU' 'W-WQM5' 'G5J-LH' 'H7OL-I' 'TO21-U' 'RN-5K9' '0M-M64' 'W5-SGC' '8RV-1L' '1C-TD6' 'YBYX-1' '2-NF2Z' '0Z-VHC' '9-BUSQ' 'LQB-TC' 'II-1B3' '6-HFD6' 'P3UD-M' 'LCN-0V' 'FX-XMW' 'G-N6MC' '7-8XK0' '90G-OA' 'DT-7EO' 'B-Y06L' 'HHQ-8L' 'Z-KPAR' '8U-RZH' '2RV-06' 'CLDT-L' 'QU7-EE' 'UC-X28' 'R79-I7' 'E-RPGP' 'ZV-KZO' 'NSE-U1' 'KER-EU' '69A-54' 'M9-OS2' '5V-YL6' '8-UWFS' 'PQWA-L' 'BWO-UU' 'SQVI-U' 'T-YWDD' 'DLY-RG' 'T-C5A0' 'UP-L3Y' 'F-KBNV' 'JL-P9P' 'FR-RCH' 'FNS3-F' '7BA-TK' 'IAWJ-X' '50-TJY' '3-CE1R' '0IRK-R'
Desmont McCallock
#44 - 2016-04-18 11:00:33 UTC
Yeah, you're posting the solar systems. I posted the constellations.
Mr Mac
Dark Goliath
#45 - 2016-04-18 17:21:30 UTC  |  Edited by: Mr Mac
Hmm, I checked lastest SDE...
I see the constellations and solarsystem there.

Edit. I can confirm missing data in mapSolarSystemJumps table
Cryten Jones
Advantage Inc
The Matari Consortium
#46 - 2016-04-21 13:20:47 UTC  |  Edited by: Cryten Jones
Ignore
Desmont McCallock
#47 - 2016-04-26 15:16:55 UTC
@Tristan Agion While I was dealing with the new SDE format I realized that region UUA-FA is the CCP region. If you look at the EVE client map you will see that there are only a few solar systems that are connected (i.e. have stargates). Therefor, the data are correct and you should close the bug report cause your claim doesn't stand.
Tristan Agion
Viziam
Amarr Empire
#48 - 2016-04-26 16:00:43 UTC  |  Edited by: Tristan Agion
Desmont McCallock wrote:
@Tristan Agion While I was dealing with the new SDE format I realized that region UUA-FA is the CCP region. If you look at the EVE client map you will see that there are only a few solar systems that are connected (i.e. have stargates). Therefor, the data are correct and you should close the bug report cause your claim doesn't stand.

Does that follow? It still could be that the EVE client map is running on the same corrupted, compiled data that goes into the SDE, while in reality there are stargates in the system. My main comparison was against Dotlan, and Dotlan does show connections. Where does Dotlan get those from? I guess the objective test is to just fly to for example ZSPJ-K and see if I can find a stargate to D95-FQ as Dotlan says, or not, as the SDE says...

(Possibly some significance of this being "the CCP region" escapes me. I'm still pretty new to the game, after all. If this is a Singularity-like test region where things are more fluid, or if this is only accessible to CCP, do let me know.)

Edit: OK, on looking around on the web and Dotlan some more, it seems that my "objective" test will fail simply because one cannot jump into this region directly in the first place. Does anybody know whether any wormhole connections ever form to UUA-FA? Still wondering how Dotlan got its data...
Hel O'Ween
Men On A Mission
#49 - 2016-04-26 16:06:55 UTC
Hi Desmont,

using the new (now deemed "legacy") SDE linked by CCP Tellus, I got an error:

Unable to import ramTypeRequirements
Reason was: Verletzung der PRIMARY KEY-Einschränkung 'ramTypeRequirements_PK'.
Ein doppelter Schlüssel kann in das dbo.ramTypeRequirements-Objekt nicht eingefügt werden.
Der doppelte Schlüsselwert ist (41590, 1, 38).

... which translates to

Reason was: Violation of PRIMARY KEY constraint 'ramTypeRequirements_PK'.
A duplicate key can't be inserted into the dbo.ramTypeRequirements object.
The duplicate key value is (41590, 1, 38).


I've also crossposted over in the SDE changes thread: https://forums.eveonline.com/default.aspx?g=posts&m=6457455#post6457455

EVEWalletAware - an offline wallet manager.

Desmont McCallock
#50 - 2016-04-26 16:38:36 UTC  |  Edited by: Desmont McCallock
Hel O'Ween wrote:
Hi Desmont,

using the new (now deemed "legacy") SDE linked by CCP Tellus, I got an error:

Unable to import ramTypeRequirements
Reason was: Verletzung der PRIMARY KEY-Einschränkung 'ramTypeRequirements_PK'.
Ein doppelter Schlüssel kann in das dbo.ramTypeRequirements-Objekt nicht eingefügt werden.
Der doppelte Schlüsselwert ist (41590, 1, 38).

... which translates to

Reason was: Violation of PRIMARY KEY constraint 'ramTypeRequirements_PK'.
A duplicate key can't be inserted into the dbo.ramTypeRequirements object.
The duplicate key value is (41590, 1, 38).


I've also crossposted over in the SDE changes thread: https://forums.eveonline.com/default.aspx?g=posts&m=6457455#post6457455

https://forums.eveonline.com/default.aspx?g=posts&m=6457513#post6457513
Workaround:
- Open the blueprints.yaml file.
- Navigate to line 169277 (blueprintTypeID: 41590)
- Comment out lines 169277-78.
- Run the tool again.
Desmont McCallock
#51 - 2016-04-26 22:25:12 UTC  |  Edited by: Desmont McCallock
@Hel O'Ween
After talking with CCP Tellus (https://forums.eveonline.com/default.aspx?g=posts&m=6458181#post6458181) I would suggest that you correct the blueprints.yaml file manually.

1. Delete lines 169277-78
2. Correct the quantity of typeID: 38 for manufacturing from 3 to 8.
Hel O'Ween
Men On A Mission
#52 - 2016-04-27 16:15:21 UTC
Ah, didn't realize that the bluepint.yaml was culprit. Then the fix is easy for me: delete that YAML, as I don't use the BPO stuff at all.

As you explained longer ago, your tool is clever enough to skip non-existing YAMLs. All hail software that's smarter as its users (=me)! Smile

EVEWalletAware - an offline wallet manager.

Darkblad
#53 - 2016-04-28 15:35:05 UTC  |  Edited by: Darkblad
How curious

Did you also find that several tables of the legacy ms sql dump are mssing their content? dgmTypeAttributes (and its dgmAttributeTypes) is a bit annoying.

I just got the info from Fuzzsteve that there was another more recent SDE, so I got that one What?
Desmont McCallock
#54 - 2016-04-28 16:00:21 UTC
Darkblad wrote:
How curious

Did you also find that several tables of the legacy ms sql dump are mssing their content? dgmTypeAttributes (and its dgmAttributeTypes) is a bit annoying.

I just got the info from Fuzzsteve that there was another more recent SDE, so I got that one What?
Not true with Citadel 1.1 SDE.
salacious necrosis
Garoun Investment Bank
Gallente Federation
#55 - 2016-04-29 11:59:59 UTC
Hey Des,

Do you need to cut a new release of your tool with fixes for the new SDE? Or are you planning to leave it as-is for now?

Thanks! And thanks again for the great tool!

Use EveKit ! - Tools for EVE Online 3rd party development

Desmont McCallock
#56 - 2016-04-29 12:02:06 UTC
salacious necrosis wrote:
Hey Des,

Do you need to cut a new release of your tool with fixes for the new SDE? Or are you planning to leave it as-is for now?

Thanks! And thanks again for the great tool!

I'm already working on a version that can import the new and the legacy format but it will take some time.
salacious necrosis
Garoun Investment Bank
Gallente Federation
#57 - 2016-04-29 12:03:34 UTC
Desmont McCallock wrote:
salacious necrosis wrote:
Hey Des,

Do you need to cut a new release of your tool with fixes for the new SDE? Or are you planning to leave it as-is for now?

Thanks! And thanks again for the great tool!

I'm already working on a version that can import the new and the legacy format but it will take some time.


Awesome. Happy to help beta test when ready.

Use EveKit ! - Tools for EVE Online 3rd party development

Golden Gnu
Garoun Investment Bank
Gallente Federation
#58 - 2016-04-30 11:05:39 UTC
@Desmont McCallock
Thank you for the workaround for blueprints.yaml
And as always; thank you for this great program. Big smileCool

Creator of jEveAssets - the asset manager

"Download is the meaning of life, upload is the meaning of intelligent life"

Desmont McCallock
#59 - 2016-04-30 12:24:48 UTC  |  Edited by: Desmont McCallock
Tristan Agion wrote:
Desmont McCallock wrote:
@Tristan Agion While I was dealing with the new SDE format I realized that region UUA-FA is the CCP region. If you look at the EVE client map you will see that there are only a few solar systems that are connected (i.e. have stargates). Therefor, the data are correct and you should close the bug report cause your claim doesn't stand.

Does that follow? It still could be that the EVE client map is running on the same corrupted, compiled data that goes into the SDE, while in reality there are stargates in the system. My main comparison was against Dotlan, and Dotlan does show connections. Where does Dotlan get those from? I guess the objective test is to just fly to for example ZSPJ-K and see if I can find a stargate to D95-FQ as Dotlan says, or not, as the SDE says...

(Possibly some significance of this being "the CCP region" escapes me. I'm still pretty new to the game, after all. If this is a Singularity-like test region where things are more fluid, or if this is only accessible to CCP, do let me know.)

Edit: OK, on looking around on the web and Dotlan some more, it seems that my "objective" test will fail simply because one cannot jump into this region directly in the first place. Does anybody know whether any wormhole connections ever form to UUA-FA? Still wondering how Dotlan got its data...

Sorry for the late reply. My eye just now caught your post.
UUA-FA has always been the CCP region meaning that in that region CCP tests things in production.
No capsuleer can access that region unless (s)he has been transferred there by CCP.
Dotlan hasn't updated its Universe data, since who knows when, probably from the time Wormholes where introduced, simply because there is no reason to do so as these data are static and don't change frequently.

Now if your O.C.D. (joke) impulses you for Dotlan and SDE to match, contact Wollari and ask him to update his DB.
Desmont McCallock
#60 - 2016-05-05 16:25:45 UTC  |  Edited by: Desmont McCallock
I'm sadden to announce that EVESDEToSQL will NOT support the new SDE format until CCP responds to my post or fixes at least the data inconsistencies between the legacy and the new format (also mentioned in my post).

@CCP devs
Not answering is not a nice way to treat customers not to mention long-time customers.