These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Proposed Changes to the Static Data Export (SDE)

First post First post
Author
Desmont McCallock
#61 - 2016-04-28 10:56:47 UTC
Bug report for data format inconsistency in invFlags table.
Hel O'Ween
Men On A Mission
#62 - 2016-04-28 15:33:48 UTC
I've skimmed through the patch notes, but didn't find it mentioned somewhere, so I ask it here:

1) Which API provides the citadel's location? Would those be included in ConquerableStationList?

2) What "magic number(s)" - if any, is necessary for calculating the citadel's solar system from an asset's locationID?

EVEWalletAware - an offline wallet manager.

Pete Butcher
The Scope
Gallente Federation
#63 - 2016-04-28 15:42:43 UTC
Hel O'Ween wrote:
I've skimmed through the patch notes, but didn't find it mentioned somewhere, so I ask it here:

1) Which API provides the citadel's location? Would those be included in ConquerableStationList?

2) What "magic number(s)" - if any, is necessary for calculating the citadel's solar system from an asset's locationID?


https://forums.eveonline.com/default.aspx?g=posts&m=6456723#post6456723

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool

Hel O'Ween
Men On A Mission
#64 - 2016-04-28 16:55:38 UTC
Yeah, saw your thread, Pete, but as it hasn't got an answer, I thought I might menion it elsewhere (here) again.

Or are you implying with your post that the locations are indeed in crest.industry.facilities, as Mr. mac suggested there? If so, that would totally break EWA ... :-(

EVEWalletAware - an offline wallet manager.

Pete Butcher
The Scope
Gallente Federation
#65 - 2016-04-28 17:02:06 UTC
Hel O'Ween wrote:
Yeah, saw your thread, Pete, but as it hasn't got an answer, I thought I might menion it elsewhere (here) again.

Or are you implying with your post that the locations are indeed in crest.industry.facilities, as Mr. mac suggested there? If so, that would totally break EWA ... :-(


Nah, I didn't know you've seen that topic. I still have no idea how to handle citadels.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool

Desmont McCallock
#66 - 2016-04-29 08:13:57 UTC
@CCP Tellus
region.staticdata for each region is missing the radius. Are we supposed to calculate that from the given info?
constallation and solarsystem staticdata files include that info however.
CCP Tellus
C C P
C C P Alliance
#67 - 2016-04-29 10:05:17 UTC
Desmont McCallock wrote:
region.staticdata for each region is missing the radius. Are we supposed to calculate that from the given info? constallation and solarsystem staticdata files include that info however.

You'll need to calculate the radius. It should be the Euclidean distance between the region's 'min' and 'max', divided by the square root of two.
Desmont McCallock
#68 - 2016-04-29 10:35:29 UTC
CCP Tellus wrote:
Desmont McCallock wrote:
region.staticdata for each region is missing the radius. Are we supposed to calculate that from the given info? constallation and solarsystem staticdata files include that info however.

You'll need to calculate the radius. It should be the Euclidean distance between the region's 'min' and 'max', divided by the square root of two.
Shouldn't the formula be radius = √((xMax - x)^2 + (yMax - y)^2 + (zMax - z)^2) ?
CCP Tellus
C C P
C C P Alliance
#69 - 2016-04-29 10:44:23 UTC
Desmont McCallock wrote:
Shouldn't the formula be radius = √((xMax - x)^2 + (yMax - y)^2 + (zMax - z)^2) ?

The distance between 'min' and 'max' is the diameter. You should divide that by two to get the radius, but someone messed it up by putting the division by two inside the square root for each field. The end result is that you divide the diameter with the square root of two instead.

This is based on my understanding of quickly looking through the source that used to build the universe SQLite file. If the numbers don't check out, I'll take another stab at it for you. :)
Desmont McCallock
#70 - 2016-04-29 10:55:27 UTC
CCP Tellus wrote:
Desmont McCallock wrote:
Shouldn't the formula be radius = √((xMax - x)^2 + (yMax - y)^2 + (zMax - z)^2) ?

The distance between 'min' and 'max' is the diameter. You should divide that by two to get the radius, but someone messed it up by putting the division by two inside the square root for each field. The end result is that you divide the diameter with the square root of two instead.

This is based on my understanding of quickly looking through the source that used to build the universe SQLite file. If the numbers don't check out, I'll take another stab at it for you. :)
OK. I assumed that a region is a sphere so I had to calculate the distance from the center, which apparently is incorrect.

Fiddling around the formula becomes radius = (√((xMax - xMin)^2 + (yMax - yMin)^2 + (zMax - zMin)^2)) / 2. This gives me fairly the same numbers as the old universe format. Still need to figure out some rounding.
Desmont McCallock
#71 - 2016-04-29 13:17:48 UTC  |  Edited by: Desmont McCallock
@CCP Tellus
I'm afraid that there is something fundamentally wrong with the formula you use to calculate the radius.

According to known maths the radius of a sphere centered at the point (x0,y0,z0) derives from the following formula.

r^2 = (x-x0)^2+(y-y0)^2+(z-z0)^2
which is equivalent to
r = √((x-x0)^2+(y-y0)^2+(z-z0)^2)

From my understanding 'min' coordinates are the coordinates of the celestial closest to the center and 'max' coordinates are the coordinates of the celestial farthest to the center.

Using
radius = (√((xMax - xMin)^2 + (yMax - yMin)^2 + (zMax - zMin)^2)) / 2
or even
radius = (√(((xMax - xMin)^2 / 2) + ((yMax - yMin)^2 / 2) + ((zMax - zMin)^2 / 2))) / √2
(which is the formula you say you use) as the formula to calculate the radius seems wrong to me in two points.

1. The used formula is incorrect (there shouldn't be any dividing)
2. The radius should be calculated from the center and not from the closest point to the center.

Am I wrong in my assumptions?

Edit: Even using the Euclidean formula for distance (where distance is actually the radius)
d(Max, 0) = √((xMax - x)^2+(yMax - y)^2+(zMax - zi)^2)
I can't seem to match the numbers you have in SDE.

Edit2: For example using the radius of a sphere from center formula, for A821-A region I get the number
5.1900850094775462E+17
but in SDE it's
5.29466097898403E+16
CCP Tellus
C C P
C C P Alliance
#72 - 2016-04-29 15:33:43 UTC  |  Edited by: CCP Tellus
Desmont McCallock wrote:
I'm afraid that there is something fundamentally wrong with the formula you use to calculate the radius.

Sorry for all the confusion. Took a stab at this and this is what I get:

From the universe data SQLite file, we have that the radius is 5.57645447335415e+16.

sqlite> select regionID, regionName, radius from mapRegions where regionName = 'A821-A';
regionID    regionName  radius             
----------  ----------  --------------------
10000019    A821-A      5.57645447335415e+16


By letting x1, y1, and z1 be the region's 'max' coordinates, and x2, y2, and z2 be the region's 'min' coordinates, we get:

>>> math.sqrt((x1 - x2)**2 + (y1 - y2)**2 + (z1 - z2)**2) / 2
5.576454473354151e+16


This matches the radius of the region in the legacy SDE. It also turns out I was wrong about dividing by the square root of two. Terribly sorry. I should test code before reaching a conclusion. :(

Hope this helps.
Desmont McCallock
#73 - 2016-04-29 15:48:50 UTC
My bad, it's indeed
5.57645447335415E+16
. My head was spinning and I posted the radius of Derelik.
Will give it a try.
Masao Kurata
Perkone
Caldari State
#74 - 2016-04-29 15:56:08 UTC
That's a really strange definition of a "radius". That's half the diagonal of a cube.
Desmont McCallock
#75 - 2016-04-29 15:56:26 UTC  |  Edited by: Desmont McCallock
Still I believe the formula is wrong.
 math.sqrt((x1 - x2)**2 + (y1 - y2)**2 + (z1 - z2)**2)
IS the Euclidean distance which IS the radius for a sphere. It shouldn't be divided by two and x2,y2,z2 should be the center coordinates and not the min ones.

Someone confused distance with diameter.

Never the less, even if it's wrong I guess it won't be fixed as it's been set to stone for years in the SDE, right?
CCP Tellus
C C P
C C P Alliance
#76 - 2016-04-29 16:22:01 UTC
Desmont McCallock wrote:
Never the less, even if it's wrong I guess it won't be fixed as it's been set to stone for years in the SDE, right?

Pretty much. Sad
Desmont McCallock
#77 - 2016-04-29 16:56:13 UTC  |  Edited by: Desmont McCallock
@ CCP Tellus

I can't match the numbers that the SDE had for the region radius. Probably because of the coordinates rounding or .NET has different implementation for math.sqrt than Python.
Is it to much to ask to include the region radius info in the universe yaml file (region.staticdata)?
It doesn't have to be this one. The next one maybe?
Desmont McCallock
#78 - 2016-05-01 14:19:42 UTC  |  Edited by: Desmont McCallock
After 5 long days battling with the new SDE format I'm finally done.
CCP Tellus I know I have become a pain in your bud but you had it coming (joking).

Here are my findings:

Negative points (errors)
- provide 'radius' info in region.staticdata for each region (already been discussed in this thread why it should be provided)
- landmarks are missing landmarkName, description (instead their ID counterparts are provided)
- constellation 'Tranquility' in Curse is named 'Tranquillity' (notice the double 'l')
- constellation 'G-C00324' in G-R00031 is named 'G-C00311' (which is correct?)
- solar system 'J170376' in 'C-R00015 > C-C00146' is named 'J140208' (which is correct?)
- provide 'orbitIndex' info in order to be able to produce the correct itemName for mapDenormalize (mainly for asteroid belts and moons)
- provide the special name of a celestial (sun, planet, moon etc) if exists (i.e. Uplingur IV (Ndoria), New Caldari I (Matigu), New Caldari III (Orieku), New Caldari V (Oniteseru))
- provide the npcStations name in order to be able to produce the correct itemName for mapDenormalize (please don't tell me that it can be found in staStations, legacy sde provided that without needing staStations). ownerID is provided, if you can provide ownerName too, it will be super.

Positive points
- Landmarks iconID can be now placed in the correct column (in legacy SDE it's falsely placed in radius column)
- mapCelestialStatistics 'pressure' info are more precise (it's truncated in legacy)
- A lot more data is available in new SDE format (but I'll stick on providing what was in legacy for EVESDEToSQL, at least for now)

Questions
- In legacy mapDenormilize table stargates and NPCStations have a number in radius. Where do we find that in the new SDE format?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#79 - 2016-05-01 18:12:13 UTC  |  Edited by: Steve Ronuken
All names are in invNames, so you can look them up, and insert them. This is for asteroid belts, moons, planets, stars, stations.

Radius and orbit index are, indeed, a bit of a pain. Though both can be calculated. (I need to work in the radius calculation)

Landmark descriptions are in trnTranslations now. (side effect of how the files are generated.)

The stargate Radius, I suspect, comes from typeIDs.yaml

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Desmont McCallock
#80 - 2016-05-01 18:27:43 UTC  |  Edited by: Desmont McCallock
Steve Ronuken wrote:
All names are in invNames, so you can look them up, and insert them. This is for asteroid belts, moons, planets, stars, stations.
Steve, sorry man but you don't get it. Universe data need to be independent as they always were.
Use case? If someone wants to import only the universe data back to a DB, he ends up with missing data. UniverseData sqlite was independent and could be used as such, so should the new format.
Steve Ronuken wrote:
Radius and orbit index are, indeed, a bit of a pain. Though both can be calculated. (I need to work in the radius calculation)
About region radius we already discussed that in length with CCP Tellus in this thread (read previous posts). Numbers only match if you use Python. As for orbitIndex, it's because I tried to calculate them, I'm asking about that info to be included. The index order isn't the same for all celestials and it gets all messed up.
Steve Ronuken wrote:
Landmark descriptions are in trnTranslations now. (side effect of how the files are generated.)
Same case as with celestial names.
Steve Ronuken wrote:
The stargate Radius, I suspect, comes from typeIDs.yaml
Still same case as celestial names. Independency.