These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Static DB dump, W-space anomalies

Author
Atum
Eclipse Industrials
Quantum Forge
#1 - 2013-08-05 20:56:07 UTC
Trying to find where the anomaly type lives in the database... dbo.mapLocationWormholeClasses has (as you'd expect) the w-space classes, but nothing about whether said system is a boring regular one, wolf-rayet, red giant, etc...
Kid Delicious
Caldari Provisions
Caldari State
#2 - 2013-08-06 04:50:59 UTC
"Magnetar Effect Beacon Class 1" is typeid 30847.

They're in invTypes, I ran across these a few weeks ago. Like all types, you can get the attributes from the dgm tables.
Atum
Eclipse Industrials
Quantum Forge
#3 - 2013-08-06 15:20:25 UTC
Kid Delicious wrote:
"Magnetar Effect Beacon Class 1" is typeid 30847.

They're in invTypes, I ran across these a few weeks ago. Like all types, you can get the attributes from the dgm tables.

Ahh, strange place to be. Using groupID = 920 looks to be a cleaner way to get what I need :)
Atum
Eclipse Industrials
Quantum Forge
#4 - 2013-08-06 20:58:24 UTC
Sigh... not getting there, might need a dev's help. invTypes.typeID 30844 through 30884 don't appear to link back to anything. I don't see them as a celestial anywhere in the w-space systems, nor do the w-space local stars use different mapSolarSystems.suntypeID from those in k-space.

(for what it's worth, I finally have a little bit of time to get back to my EVEMap resurrection project, and I've tracked down the navigation problem to a value not being included in the data files. While figuring that out, I noticed that right beside where that value should have been, the code was also attempting to get w-space anomaly types, but also coming up with an uninitialized value).
Kid Delicious
Caldari Provisions
Caldari State
#5 - 2013-08-06 23:15:25 UTC
It is an odd place, since inv stands for inventory I think. It probably something to do with those attributes and the dgm system.

I imagine that data not being there is by design.

There is a celestials table in /EVEINSTALL/bulkdata/mapbulk.db (sqlite) but it only has groups

Quote:

Asteroid Belt
Moon
Planet
Stargate
Sun


The wormhole websites out there are populated by users. You could scrap one (with permission), I'm guessing the anoms don't change.
Atum
Eclipse Industrials
Quantum Forge
#6 - 2013-08-07 13:46:49 UTC
Kid Delicious wrote:
It is an odd place, since inv stands for inventory I think. It probably something to do with those attributes and the dgm system.

I imagine that data not being there is by design.

There is a celestials table in /EVEINSTALL/bulkdata/mapbulk.db (sqlite) but it only has groups

Quote:

Asteroid Belt
Moon
Planet
Stargate
Sun


The wormhole websites out there are populated by users. You could scrap one (with permission), I'm guessing the anoms don't change.

Pretty sure you're right about why the effect generator "item" lives in the inv tables (the way a shield booster or afterburner would), since effects on your ship originate within the dogma system (and there is a link between invTypes.typeID and dgmEffects.effectID via the dgmTypeEffects table). Just find it strange that AcriQuo would have written a complete (and not commented out) chunk of code and made a field for the data if it weren't available as part of the data dump. One of those "we gotta be missing something" sort of feelings.
Bloemkoolsaus
Deep Core Mining Inc.
Caldari State
#7 - 2013-08-07 15:12:18 UTC
If I'm understanding your problem correctly, these might help:


SELECT t.typename, c.wormholeclassid
FROM mapdenormalize n
LEFT JOIN invtypes t ON n.typeid = t.typeid
LEFT JOIN maplocationwormholeclasses c ON n.regionid = c.locationid
WHERE n.solarsystemid = 123456789
AND n.groupid = '995'



SELECT i.typename, t.attributename, t.displayname, a.valuefloat, t.unitid
FROM dgmtypeattributes a
INNER JOIN dgmattributetypes t ON t.attributeid = a.attributeid
INNER JOIN invtypes i ON i.typeid = a.typeid
WHERE i.typename LIKE '%Red Giant%'
AND i.typename LIKE '%Class 5%'
Atum
Eclipse Industrials
Quantum Forge
#8 - 2013-08-07 17:09:49 UTC
Bloemkoolsaus wrote:
If I'm understanding your problem correctly, these might help:


SELECT t.typename, c.wormholeclassid
FROM mapdenormalize n
LEFT JOIN invtypes t ON n.typeid = t.typeid
LEFT JOIN maplocationwormholeclasses c ON n.regionid = c.locationid
WHERE n.solarsystemid = 123456789
AND n.groupid = '995'



SELECT i.typename, t.attributename, t.displayname, a.valuefloat, t.unitid
FROM dgmtypeattributes a
INNER JOIN dgmattributetypes t ON t.attributeid = a.attributeid
INNER JOIN invtypes i ON i.typeid = a.typeid
WHERE i.typename LIKE '%Red Giant%'
AND i.typename LIKE '%Class 5%'

Hot dog, it *is* in the denormalize table like I thought it'd be... just that the itemID value is so far beyond everything else in each respective solar system that I missed it entirely until your suggestion to filter stuff down led me to focus on just a single system, and bingo :)

Now I just gotta figure out how to take your code (MySQL?) and translate it to the free MSSQL studio package :)
Atum
Eclipse Industrials
Quantum Forge
#9 - 2013-08-07 19:18:25 UTC
Sigh... MSSQL doesn't like me for some reason. Here's my current SELECT statement:
Quote:
SELECT A.[solarSystemName]
,A.[solarSystemID]
,A.[constellationID]
,A.[x]
,A.[y]
,A.[z]
,A.[security]
,SUM(CASE WHEN B.[groupID] = 7 THEN 1 ELSE 0 END) as [Num_Planets]
,SUM(CASE WHEN B.[groupID] = 8 THEN 1 ELSE 0 END) as [Num_Moons]
,SUM(CASE WHEN B.[groupID] = 9 THEN 1 ELSE 0 END) as [Num_Belts]
,0 as [Zero_1]
,(CASE WHEN B.[typeID] = 30574 THEN 3
WHEN B.[typeID] = 30575 THEN 1
WHEN B.[typeID] = 30576 THEN 5
WHEN B.[typeID] = 30577 THEN 4
WHEN B.[typeID] = 30669 THEN 6
WHEN B.[typeID] = 30670 THEN 2
ELSE 0 END) as [anomaly]

FROM [dbo].[mapSolarSystems] A
JOIN [dbo].[mapDenormalize] B ON A.[solarSystemID] = B.[solarSystemID]
GROUP BY
A.[solarSystemName]
,A.[solarSystemID]
,A.[constellationID]
,A.[x]
,A.[y]
,A.[z]
,A.[security]
GO

Now before I added the underlinged segment, it was just another ",0 as [Zero_2]" and things were fine. Now that I've added that CASE statement, I get the message

Msg 8120, Level 16, State 1, Line 12
Column 'dbo.mapDenormalize.typeID' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause.

What I don't understand is why this wasn't a problem with [Num_Planets] and such. At this point it should be fairly obvious I'm learning by doing, so pointers would be appreciated :)
Fubar
Eagle Eye Inc.
#10 - 2013-08-08 01:47:59 UTC  |  Edited by: Fubar
Quote:

Now before I added the underlinged segment, it was just another ",0 as [Zero_2]" and things were fine. Now that I've added that CASE statement, I get the message

Msg 8120, Level 16, State 1, Line 12
Column 'dbo.mapDenormalize.typeID' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause.

What I don't understand is why this wasn't a problem with [Num_Planets] and such. At this point it should be fairly obvious I'm learning by doing, so pointers would be appreciated :)

Add sum or min or max, doesn't matter which one, at the beginning of the underlined segment. That will add the segment as part of the aggregate function.

The query would not work the way you wanted it to if you added b.typeID to the GROUP BY clause.
Atum
Eclipse Industrials
Quantum Forge
#11 - 2013-08-08 02:58:57 UTC  |  Edited by: Atum
Fubar wrote:
Add sum or min or max, doesn't matter which one, at the beginning of the underlined segment. That will add the segment as part of the aggregate function.

The query would not work the way you wanted it to if you added b.typeID to the GROUP BY clause.

Awesome, worked perfectly!! (I did try adding something to the GROUP BY portion earlier this afternoon as you hinted, but came up with an entirely different error that made it seem like I was trying to feed oranges to a flamingo)

Now to go hack at the last bit... system classification (mapLocationWormholeClasses), which I think is the missing link in getting route planning to work again... CCP did this kinda weird by defining entire regions as high, null, or C1-C6, then overriding low-sec system by system. (except for two constellations in Jove space which are apparently undefined classes 10 and 11?). Here's how I *think* it needs to be done, but I have no idea how to write the JOIN to make it work, since mapLocationWormholeClasses doesn't share a one-for-one match to mapSolarSystems they way mDenormalize does (C. is mLWC)...

,SUM(CASE WHEN C.[locationID] = A.[regionID] THEN C.[wormholeClassID] /*Matches highsec, nullsec, C1-C6*/
CASE WHEN C.[locationID] = A.[constellationID] THEN C.[wormholeClassID] /*special case for the Jove*/
ELSE 8 END) as [classification] /*anything that's left must be low sec*/

That's probably an ugly hack, but I'm not sure how else I could copy the values over since mLWC is missing that one-for-one that mD shares with mSS :(
Atum
Eclipse Industrials
Quantum Forge
#12 - 2013-08-08 04:12:33 UTC  |  Edited by: Atum
Figure'd I'd post what I'm working with before I crash for the night... seems as if I'm looking to perform a "full outer join" but the syntax doesn't seem quite right. The query:
Quote:
SELECT A.[solarSystemName]
,A.[solarSystemID]
,A.[constellationID]
,A.[x]
,A.[y]
,A.[z]
,A.[security]
,SUM(CASE WHEN B.[groupID] = 7 THEN 1 ELSE 0 END) as [planets]
,SUM(CASE WHEN B.[groupID] = 8 THEN 1 ELSE 0 END) as [moons]
,SUM(CASE WHEN B.[groupID] = 9 THEN 1 ELSE 0 END) as [belts]
,SUM(CASE WHEN C.[locationID] = A.[regionID] THEN C.[wormholeClassID]
WHEN C.[locationID] = A.[constellationID] THEN C.[wormholeClassID]
ELSE 8 END) as [classification]
,SUM(CASE WHEN B.[typeID] = 30574 THEN 3
WHEN B.[typeID] = 30575 THEN 1
WHEN B.[typeID] = 30576 THEN 5
WHEN B.[typeID] = 30577 THEN 4
WHEN B.[typeID] = 30669 THEN 6
WHEN B.[typeID] = 30670 THEN 2
ELSE 0 END) as [anomaly]
FROM [dbo].[mapSolarSystems] A
JOIN [dbo].[mapDenormalize] B ON A.[solarSystemID] = B.[solarSystemID]
FULL OUTER JOIN [dbo].[mapLocationWormholeClasses] C
GROUP BY
A.[solarSystemName]
,A.[solarSystemID]
,A.[constellationID]
,A.[x]
,A.[y]
,A.[z]
,A.[security]

And the error:
Quote:
Msg 156, Level 15, State 1, Line 25
Incorrect syntax near the keyword 'GROUP'.

It's worth noting that I get the same error whether I use the SUM/MIN/MAX thing, or leave it off entirely.
Bloemkoolsaus
Deep Core Mining Inc.
Caldari State
#13 - 2013-08-08 07:38:54 UTC
The mapLocationWormholeClasses table is an interesting table. The locationID refers to the regionID from the mapSolarSystems table. You've already found out this gives some interesting results for some solarsystems, especially low-sec.

The trick is, that for some systems the locationID is actually the solarSystemID.
Fubar
Eagle Eye Inc.
#14 - 2013-08-08 14:00:57 UTC
I think this query is what you are looking for.

Quote:
SELECT A.[solarSystemName]
,A.[solarSystemID]
,A.[constellationID]
,A.[x]
,A.[y]
,A.[z]
,SUM(CASE WHEN B.[groupID] = 7 THEN 1 ELSE 0 END) as [planets]
,SUM(CASE WHEN B.[groupID] = 8 THEN 1 ELSE 0 END) as [moons]
,SUM(CASE WHEN B.[groupID] = 9 THEN 1 ELSE 0 END) as [belts]
,COALESCE(D.[wormholeClassID],C.[wormholeClassID],9) as [classification]
,ROUND(B.security,1) as [security]
,(CASE
WHEN B.[typeID] = 30574 THEN 3
WHEN B.[typeID] = 30575 THEN 1
WHEN B.[typeID] = 30576 THEN 5
WHEN B.[typeID] = 30577 THEN 4
WHEN B.[typeID] = 30669 THEN 6
WHEN B.[typeID] = 30670 THEN 2
ELSE 0 END) as [anomaly]
FROM [dbo].[mapSolarSystems] A
JOIN [dbo].[mapDenormalize] B ON A.[solarSystemID] = B.[solarSystemID]
LEFT JOIN [dbo].[mapLocationWormholeClasses] C ON C.locationID = A.regionID
LEFT JOIN [dbo].[mapLocationWormholeClasses] D ON D.locationID = A.solarSystemID

GROUP BY
A.[solarSystemName]
,A.[solarSystemID]
,A.[constellationID]
,A.[x]
,A.[y]
,A.[z]
,COALESCE(D.[wormholeClassID],C.[wormholeClassID],9)
,B.security
,(CASE WHEN B.[typeID] = 30574 THEN 3
WHEN B.[typeID] = 30575 THEN 1
WHEN B.[typeID] = 30576 THEN 5
WHEN B.[typeID] = 30577 THEN 4
WHEN B.[typeID] = 30669 THEN 6
WHEN B.[typeID] = 30670 THEN 2
ELSE 0 END)
ORDER BY b.[security], a.[solarSystemName]
I have underlined the changed I made.

I remembered how to properly GROUP BY formulas.
Atum
Eclipse Industrials
Quantum Forge
#15 - 2013-08-08 14:30:33 UTC
Bloem: Yeah, pretty much. I wonder what made CCP decide to do it that way. Probably for performance, but.....?
Fubar: Let me see if I follow your code correctly...

LEFT JOIN [dbo].[mapLocationWormholeClasses] C ON C.locationID = A.regionID
LEFT JOIN [dbo].[mapLocationWormholeClasses] D ON D.locationID = A.solarSystemID
- The two LEFT JOIN statements connect the mLWC to the mSS table via either regionID or solarSystemID, rather than my method of just throwing both tables into the bucket and seeing what sloshes back out. The results are then made available via C.[x] and D.[x]. You've left out the two Jove constellations since nobody can get there anyway.

COALESCE(D.[wormholeClassID],C.[wormholeClassID],9) as [classification]
- This replaces my convoluted method of assigning a classification by iterating through the nine defined system classifications until there's a match. Again, the two Jove classes (10 and 11) were left out since nobody can legitimately go there.

ROUND(B.security,1) as [security]
- Not sure why this is being replaced, since you can pluck A.[security] and not need to do anything special

(CASE WHEN B.[typeID] = 30574 THEN 3.......
- I'm guessing the changes to the GROUP BY formula made the SUM unnecessary?

GROUP BY.........
- Really don't understand just what this does yet. The MS Technet article makes it seem like this functions rather like a PivotTable, but since there should only be one row for each solar system, I don't quite get why it's needed. Unless SQL returns only values for fields that were SELECTed, in which case I could see this collapsing 12 rows, each with only two fields populated and the rest NULL, down into one row with all 12 fields populated?
Atum
Eclipse Industrials
Quantum Forge
#16 - 2013-08-08 14:43:55 UTC
Slight tweakage to what you gave me, but it looks to be working correctly, and had then (unintended!!) side effect of sorting everything first by wormholeClassID, then by solarSystemID :)
Quote:
SELECT A.[solarSystemName]
,A.[solarSystemID]
,A.[constellationID]
,A.[x]
,A.[y]
,A.[z]
,A.[security]
,SUM(CASE WHEN B.[groupID] = 7 THEN 1 ELSE 0 END) as [planets]
,SUM(CASE WHEN B.[groupID] = 8 THEN 1 ELSE 0 END) as [moons]
,SUM(CASE WHEN B.[groupID] = 9 THEN 1 ELSE 0 END) as [belts]
,COALESCE(D.[wormholeClassID],E.[wormholeClassID],C.[wormholeClassID],11) as [classification]
,SUM(CASE WHEN B.[typeID] = 30574 THEN 3
WHEN B.[typeID] = 30575 THEN 1
WHEN B.[typeID] = 30576 THEN 5
WHEN B.[typeID] = 30577 THEN 4
WHEN B.[typeID] = 30669 THEN 6
WHEN B.[typeID] = 30670 THEN 2
ELSE 0 END) as [anomaly]
FROM [dbo].[mapSolarSystems] A
JOIN [dbo].[mapDenormalize] B ON A.[solarSystemID] = B.[solarSystemID]
LEFT JOIN [dbo].[mapLocationWormholeClasses] C ON C.locationID = A.regionID
LEFT JOIN [dbo].[mapLocationWormholeClasses] E ON E.locationID = A.constellationID
LEFT JOIN [dbo].[mapLocationWormholeClasses] D ON D.locationID = A.solarSystemID
GROUP BY A.[solarSystemName]
,A.[solarSystemID]
,A.[constellationID]
,A.[x]
,A.[y]
,A.[z]
,A.[security]
,COALESCE(D.[wormholeClassID],E.[wormholeClassID],C.[wormholeClassID],11)

Fubar
Eagle Eye Inc.
#17 - 2013-08-08 14:56:19 UTC
Atum wrote:
LEFT JOIN [dbo].[mapLocationWormholeClasses] C ON C.locationID = A.regionID
LEFT JOIN [dbo].[mapLocationWormholeClasses] D ON D.locationID = A.solarSystemID
- The two LEFT JOIN statements connect the mLWC to the mSS table via either regionID or solarSystemID, rather than my method of just throwing both tables into the bucket and seeing what sloshes back out. The results are then made available via C.[x] and D.[x]. You've left out the two Jove constellations since nobody can get there anyway.

ALL JOIN statements need an ON condition. The LEFT JOIN specifies that all records from the left table (here mapSolarSystems) and only the records from the right table (here mapLocationWormholeClasses) that match, otherwise NULL if returned.


Quote:
COALESCE(D.[wormholeClassID],C.[wormholeClassID],9) as [classification]
- This replaces my convoluted method of assigning a classification by iterating through the nine defined system classifications until there's a match. Again, the two Jove classes (10 and 11) were left out since nobody can legitimately go there.

COALESCE can be defined as a case shortcut. Basically the first value will be used if the value is not null, if it is null the second value is considered for null or not null. Finally if both of the first values are null 9 is used.


Quote:
ROUND(B.security,1) as [security]
- Not sure why this is being replaced, since you can pluck A.[security] and not need to do anything special

I used round just to verify the classification was returning the proper value. You don't have to use round if you don't want to.


Quote:
(CASE WHEN B.[typeID] = 30574 THEN 3.......
- I'm guessing the changes to the GROUP BY formula made the SUM unnecessary?

You are correct. I remembered that the full expression is used in the GROUP BY section.


Quote:
GROUP BY.........
- Really don't understand just what this does yet. The MS Technet article makes it seem like this functions rather like a PivotTable, but since there should only be one row for each solar system, I don't quite get why it's needed. Unless SQL returns only values for fields that were SELECTed, in which case I could see this collapsing 12 rows, each with only two fields populated and the rest NULL, down into one row with all 12 fields populated?

The GROUP BY section is required when any aggregate function is used (here sum was used).
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#18 - 2013-08-08 15:32:13 UTC  |  Edited by: Steve Ronuken
Group By:

You have the table:
col1,col2,col3
A,X,1
A,Y,2
A,X,3
B,X,1


select sum(col3) from table:
7

select col1,sum(col3) from table group by col1:
A,6
B,1

select col1,col2,sum(col3) from table group by col1,col2:
A,X,4
A,Y,2
B,X,1


Basically it causes the aggregate functions to work over selected subsets of the data. Sort the data, then when one of the values changes, it resets and starts sum-ing (max-ing, min-ing, etc) again.


Handy for cases like working out average salaries for departments and genders, for example. This can be expanded by use of Rollup and Cube, to add in aggregate values when you ignore one of the group by values. The syntax varies between databases. I'm using mysql here.

select col1,col2,sum(col3) from table group by col1,col2 with rollup:

A,X,4
A,Y,2
A,NULL,6
B,X,1
B,NULL,1



Cube would lead to:
A,X,4
A,Y,2
A,NULL,6
B,X,1
B,NULL,1
NULL,X,5
NULL,Y,2
NULL,NULL,7

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter