These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

Building a ship list without 'useless' special edition ships

First post
Author
Previous Topic Next Topic
Tristan Ekain
The Scope
Gallente Federation
#1 - 2015-01-21 21:24:45 UTC
Hi all!

First of all, please don't close this immediately out of fear of a huge wall of text. I tried to make it as easy to read as possible.

I am trying to write a script that takes the CCP SDE dump and builds a list of ships, excluding all those 'useless' special edition ships that don't have differnt attributes than their 'base type'. Special Edition ships that are different should of course not be filtered out.

Let's take the Catalyst as an example. For that ships, these are the ships that I'd like to ignore:

  • Aliastra Catalyst
  • Catalyst Serpentis Edition
  • Inner Zone Shipping Catalyst
  • Intaki Syndicate Catalyst
  • InterBus Catalyst
  • Quafe Catalyst


So far I couldn't find a single condition that does the job 100%. Instead, I need 3 conditions, and I'm not really happy with this. Maybe someone else has a better idea.

So let me show you what I've done so far. To make it easier to follow, I have created a Google Spreadsheet Document with all the data that I'm using.

The first few columns are directly from the SDE dump:

  • typeName: from the invTables table
  • typeID: from the invTypes table
  • metaLevel: attribute #633
  • techLevel: attribute #422
  • metaGroup: invMetaGroups.metaGroupdName (via joins invTypes.typeID -> invMetaTypes.typeID, invMetaTypes.metaGroupID -> invMetaGroups.metaGroupID)
  • baseType: the hull on which this ships is based (e.g. Wreathe for the Prowler); invMetaTypes.parentTypeID (via join invTypes.typeID -> invMetaTypes.typeID)
  • marketGroup: the ship's market group; invTypes.marketGroupID


The other columns are calculated:

  • ignore: TRUE, when any of the following columns are TRUE; FALSE otherwise
  • ignore_no_market_group: TRUE when the ship does not have a market group
  • ignore_ends_with_edition: TRUE when the ship's name ends with the word 'Edition'
  • ignore_faction_meta_0: TRUE when the ship is a faction ship, but at the same time has a meta level of 0


To see which ships are filtered and which not, you can just use the filtering feature for the 'ignore' column in the Google Spreadsheet.

So far, it seems to me that the 'ignore' column is correct. But I'm absolutely hot happy with this, because

  • tt's terribly complicated, and
  • I'm not 100% certain that the 'ignore' column is always correct,
  • it seems fragile, as CCP might add the next special edition ship in a way that none of the conditions match.


So if someone (maybe even a CCP dev) has a better idea, or if someone finds an error in my table, please tell me. There's still some hope left in me that a better solution exists.



- Tristan
CCP FoxFour
C C P
C C P Alliance
#2 - 2015-01-21 22:28:07 UTC
Eh, wait a few months until we have the new ship skinning system and these types will all go away. :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Tristan Ekain
The Scope
Gallente Federation
#3 - 2015-01-21 23:05:59 UTC
CCP FoxFour wrote:
Eh, wait a few months until we have the new ship skinning system and these types will all go away. :)

Well, that's good news :-)

So what should I do until then? Do the conditions I'm using make any sense to you?

Maybe I can find someone who's more knowledgable in ship types to verify my list.
If the new ship skinning engine removes the need for new special edition ships, I could even live with a hardcoded list until then.
Warde Guildencrantz
Caldari Provisions
Caldari State
#4 - 2015-01-22 01:49:09 UTC
CCP FoxFour wrote:
Eh, wait a few months until we have the new ship skinning system and these types will all go away. :)


And for a small amount of aurum, you can buy each one back!

(I really hope that's not the case)

TunDraGon ~ Low sec piracy since 2003 ~ Youtube ~ Join Us
Sir Substance
Sebiestor Tribe
Minmatar Republic
#5 - 2015-01-22 01:54:00 UTC  |  Edited by: Sir Substance
My pragmatic solution would be a hardcoded exclude list.

As FoxFour says, in a few months the skinning system will hopefully see all these variants removed.

In the mean time, we have at least one new non-special ship coming out (the svipul), but to my knowledge no special edition ships.

I'd hard-code a list of special editions to ignore, that'll solve your issue for the next few months.

If you want a more dynamic solution, I would say that unless you need this to happen in real time, I would just brute force it, probably with regex.

For each item in the ship list, do a regex match to find other items that contain the same full name string, but have extra crap in the name. Cull the list as you go for minor optimization points.

Inelegant, hacky, inefficient, but it'll work, and especially if this is a batch process you do once per new SDE, it'll do the job without you tearing your hair out trying to find a fancy solution.

The beatings will continue until posting improves. -Magnus Cortex

Official Eve Online changelist: Togglable PvP. - Jordanna Bauer
Tristan Ekain
The Scope
Gallente Federation
#6 - 2015-01-22 08:49:09 UTC
Sir Substance wrote:
My pragmatic solution would be a hardcoded exclude list.

As FoxFour says, in a few months the skinning system will hopefully see all these variants removed.

...

For each item in the ship list, do a regex match to find other items that contain the same full name string, but have extra crap in the name. Cull the list as you go for minor optimization points.

...


Yeah, I don't mind hard coding stuff; a pragmatic solution is better than none at all. Just using string macthing won't do the job however, as for example the "Vexor Navy Issue" might get excluded then, but this ship is definitely not just a repainted Vexor ;-)

I think I'll just got with the conditions I currently have (including some string matching); that should get the job done until the new skinning engine is ready. Unless of course someone finds an error in that list... ;-)
Sir Substance
Sebiestor Tribe
Minmatar Republic
#7 - 2015-01-22 10:22:42 UTC
Tristan Ekain wrote:

Yeah, I don't mind hard coding stuff; a pragmatic solution is better than none at all. Just using string macthing won't do the job however, as for example the "Vexor Navy Issue" might get excluded then, but this ship is definitely not just a repainted Vexor ;-)

Bah, true, I didn't think of that one. That brings us back to silly lists of multiple rules. Better to go with an exceptions list.

The beatings will continue until posting improves. -Magnus Cortex

Official Eve Online changelist: Togglable PvP. - Jordanna Bauer
Tristan Ekain
The Scope
Gallente Federation
#8 - 2015-01-22 11:18:07 UTC
Sir Substance wrote:
Tristan Ekain wrote:

Yeah, I don't mind hard coding stuff; a pragmatic solution is better than none at all. Just using string macthing won't do the job however, as for example the "Vexor Navy Issue" might get excluded then, but this ship is definitely not just a repainted Vexor ;-)

Bah, true, I didn't think of that one. That brings us back to silly lists of multiple rules. Better to go with an exceptions list.


The only problem with the exception list is that verifying it is quite time consuming. Some ships which, judging by their name, could be 'useless' special editions, are indeed a bit different from their base ships.

But as the new ship skinning engine will solve that once and for all in a while, I'm not even going to attempt building a 100% correct list. I already have my spreadsheet that seems to work pretty well, and will just use that for the time being.

At least it was interesting digging around in the database dump. I'm just glad that I don't have to work with the dogma tables... ;-)
Sir Substance
Sebiestor Tribe
Minmatar Republic
#9 - 2015-01-23 02:25:39 UTC
Tristan Ekain wrote:

The only problem with the exception list is that verifying it is quite time consuming. Some ships which, judging by their name, could be 'useless' special editions, are indeed a bit different from their base ships.


Post your list here once you're done, I'll double check it for you.

The beatings will continue until posting improves. -Magnus Cortex

Official Eve Online changelist: Togglable PvP. - Jordanna Bauer
Tristan Ekain
The Scope
Gallente Federation
#10 - 2015-01-23 07:30:58 UTC
Sir Substance wrote:
Tristan Ekain wrote:

The only problem with the exception list is that verifying it is quite time consuming. Some ships which, judging by their name, could be 'useless' special editions, are indeed a bit different from their base ships.


Post your list here once you're done, I'll double check it for you.

Thanks for the offer. I think I'll finish it tomorrow afternoon; then I'll post it here.
Tristan Ekain
The Scope
Gallente Federation
#11 - 2015-01-23 10:13:07 UTC
Allright, I've finished my script a bit earlier than I thought, and here are the results.
I have to split this up into two posts because of length limits.

Distinct ships:

  • Abaddon
  • Absolution
  • Adrestia
  • Aeon
  • Algos
  • Amarr Shuttle
  • Anathema
  • Anshar
  • Apocalypse
  • Apocalypse Imperial Issue
  • Apocalypse Navy Issue
  • Apotheosis
  • Arazu
  • Arbitrator
  • Archon
  • Ares
  • Ark
  • Armageddon
  • Armageddon Imperial Issue
  • Armageddon Navy Issue
  • Ashimmu
  • Astarte
  • Astero
  • Atron
  • Augoror
  • Augoror Navy Issue
  • Avatar
  • Badger
  • Bantam
  • Barghest
  • Basilisk
  • Bellicose
  • Bestower
  • Bhaalgorn
  • Blackbird
  • Bowhead
  • Breacher
  • Broadsword
  • Brutix
  • Brutix Navy Issue
  • Burst
  • Bustard
  • Buzzard
  • Caldari Navy Hookbill
  • Caldari Shuttle
  • Cambion
  • Caracal
  • Caracal Navy Issue
  • Catalyst
  • Celestis
  • Cerberus
  • Chameleon
  • Charon
  • Cheetah
  • Chimera
  • Chremoas
  • Claw
  • Claymore
  • Coercer
  • Condor
  • Confessor
  • Corax
  • Cormorant
  • Covetor
  • Crane
  • Crow
  • Crucifier
  • Cruor
  • Crusader
  • Curse
  • Cyclone
  • Cynabal
  • Damnation
  • Daredevil
  • Deimos
  • Devoter
  • Dominix
  • Dominix Navy Issue
  • Dragoon
  • Drake
  • Drake Navy Issue
  • Dramiel
  • Eagle
  • Echelon
  • Echo
  • Enyo
  • Eos
  • Epithal
  • Erebus
  • Eris
  • Etana
  • Executioner
  • Exequror
  • Exequror Navy Issue
  • Falcon
  • Federation Navy Comet
  • Fenrir
  • Ferox
  • Flycatcher
  • Freki
  • Gallente Shuttle
  • Garmur
  • Gila
  • Gnosis
  • Gold Magnate
  • Golem
  • Goru's Shuttle
  • Griffin
  • Guardian
  • Guardian-Vexor
  • Guristas Shuttle
  • Harbinger
  • Harbinger Navy Issue
  • Harpy
  • Hawk
  • Hel
  • Helios
  • Hematos
  • Heretic
  • Heron
  • Hoarder
  • Hound
  • Huginn
  • Hulk
  • Hurricane
  • Hurricane Fleet Issue
  • Hyena
  • Hyperion
  • Ibis
  • Imicus
  • Immolator
  • Impairor
  • Impel
  • Imperial Navy Slicer
  • Incursus
  • Inquisitor
  • Interbus Shuttle
  • Ishkur
  • Ishtar
  • Iteron Mark V
  • Jaguar
  • Keres
  • Kestrel
  • Kitsune
  • Kronos
  • Kryos
  • Lachesis
  • Legion
  • Leopard
  • Leviathan
  • Loki
  • Machariel
  • Mackinaw
  • Maelstrom
  • Magnate
  • Malediction
  • Malice
  • Maller
  • Mammoth
  • Manticore
  • Mastodon
  • Maulus
  • Megathron
  • Megathron Federate Issue
  • Megathron Navy Issue
  • Merlin
  • Miasmos
  • Mimir
  • Minmatar Shuttle
  • Moa
  • Moracha
  • Moros
  • Muninn
  • Myrmidon
  • Naga
  • Naglfar
  • Navitas
  • Nemesis
  • Nereus
  • Nestor
  • Nidhoggur
  • Nighthawk
  • Nightmare
  • Noctis
  • Nomad
  • Nyx
  • Obelisk
  • Occator
  • Omen
  • Omen Navy Issue
  • Oneiros
  • Onyx
  • Oracle
  • Orca
  • Orthrus
  • Osprey
  • Osprey Navy Issue
  • Paladin
  • Panther
  • Phantasm
  • Phobos
  • Phoenix
  • Pilgrim
  • Police Pursuit Comet
  • Primae
  • Probe
  • Procurer
  • Prophecy
  • Prorator
  • Prospect
  • Proteus
  • Providence
  • Prowler
  • Punisher
  • Purifier
  • Ragnarok
  • Rapier
  • Raptor
  • Rattlesnake
  • Raven
  • Raven Navy Issue
  • Raven State Issue
  • Reaper
  • Redeemer
  • Republic Fleet Firetail
  • Retribution
  • Retriever
  • Revelation
  • Revenant
  • Rhea
  • Rifter
  • Rokh
  • Rook
  • Rorqual
  • Rupture
  • Sabre
  • Sacrilege
  • Scimitar
  • Scorpion
  • Scorpion Ishukone Watch
  • Scorpion Navy Issue
  • Scythe
  • Scythe Fleet Issue
  • Sentinel
  • Sigil
  • Silver Magnate
  • Sin
  • Skiff
  • Slasher
  • Sleipnir
  • Stabber
  • Stabber Fleet Issue
  • Stiletto
  • Stratios
  • Succubus
  • Taipan
  • Talos
  • Talwar
  • Taranis
  • Tayra
  • Tempest
  • Tempest Fleet Issue
  • Tempest Tribal Issue
  • Tengu
  • Thanatos
  • Thorax
  • Thrasher
  • Tormentor
  • Tornado
  • Tristan
  • Typhoon
  • Typhoon Fleet Issue
  • Utu
  • Vagabond
  • Vangel
  • Vargur
  • Velator
  • Vengeance
  • Venture
  • Vexor
  • Vexor Navy Issue
  • Viator
  • Vigil
  • Vigilant
  • Vindicator
  • Violator
  • Vulture
  • Whiptail
  • Widow
  • Wolf
  • Worm
  • Wreathe
  • Wyvern
  • Zealot
  • Zephyr

Tristan Ekain
The Scope
Gallente Federation
#12 - 2015-01-23 10:13:56 UTC
"Skinned", special edition ships with no difference to their base ship except looks:

  • Abaddon Kador Edition
  • Abaddon Tash-Murkon Edition
  • Aliastra Catalyst
  • Apocalypse Blood Raider Edition
  • Apocalypse Kador Edition
  • Apocalypse Tash-Murkon Edition
  • Bestower Tash-Murkon Edition
  • Brutix Serpentis Edition
  • Caracal Nugoeihuvi Edition
  • Caracal Wiyrkomi Edition
  • Catalyst Serpentis Edition
  • Coercer Blood Raiders Edition
  • Cormorant Guristas Edition
  • Cyclone Thukker Tribe Edition
  • Dominix Quafe Edition
  • Ferox Guristas Edition
  • Golem Guristas Edition
  • Golem Kaalakiota Edition
  • Golem Nugoeihuvi Edition
  • Hyperion Aliastra Edition
  • Hyperion Inner Zone Shipping Edition
  • Incursus Aliastra Edition
  • Incursus Inner Zone Shipping Edition
  • Inner Zone Shipping Catalyst
  • Inner Zone Shipping Imicus
  • Intaki Syndicate Catalyst
  • InterBus Catalyst
  • Iteron Inner Zone Shipping Edition
  • Kronos Inner Zone Shipping Edition
  • Kronos Police Edition
  • Kronos Quafe Edition
  • Mackinaw ORE Development Edition
  • Maelstrom Krusual Edition
  • Maelstrom Nefantar Edition
  • Mammoth Nefantar Edition
  • Megathron Inner Zone Shipping Edition
  • Megathron Police Edition
  • Megathron Quafe Edition
  • Merlin Nugoeihuvi Edition
  • Merlin Wiyrkomi Edition
  • Miasmos Amastris Edition
  • Miasmos Quafe Ultra Edition
  • Miasmos Quafe Ultramarine Edition
  • Moros Interbus Edition
  • Naglfar Justice Edition
  • Nefantar Thrasher
  • Omen Kador Edition
  • Omen Tash-Murkon Edition
  • Opux Luxury Yacht
  • Orca ORE Development Edition
  • Paladin Blood Raider Edition
  • Paladin Kador Edition
  • Paladin Tash-Murkon Edition
  • Phoenix Wiyrkomi Edition
  • Prophecy Blood Raiders Edition
  • Punisher Kador Edition
  • Punisher Tash-Murkon Edition
  • Quafe Catalyst
  • Rattlesnake Victory Edition
  • Raven Guristas Edition
  • Raven Kaalakiota Edition
  • Raven Nugoeihuvi Edition
  • Revelation Sarum Edition
  • Rifter Krusual Edition
  • Rifter Nefantar Edition
  • Rokh Nugoeihuvi Edition
  • Rokh Wiyrkomi Edition
  • Rorqual ORE Development Edition
  • Sarum Magnate
  • Stabber Krusual Edition
  • Stabber Nefantar Edition
  • Stratios Emergency Responder
  • Sukuuvestaa Heron
  • Tash-Murkon Magnate
  • Tayra Wiyrkomi Edition
  • Tempest Justice Edition
  • Tempest Krusual Edition
  • Tempest Nefantar Edition
  • Thorax Aliastra Edition
  • Thorax Inner Zone Shipping Edition
  • Thrasher Thukker Tribe Edition
  • Tristan Quafe Edition
  • Vargur Justice Edition
  • Vargur Krusual Edition
  • Vargur Nefantar Edition
  • Vexor Quafe Edition
  • Vherokior Probe


Feedback on those lists is more than welcome.
The spreadsheet I've linked in my first post is also updated, and contains the same data.
Sir Substance
Sebiestor Tribe
Minmatar Republic
#13 - 2015-01-23 13:07:54 UTC
The Opux Luxuary Yacht is a distinct ship, not a skin.
The Police Pursuit Comet is a skin, not a distinct ship.
The Stratios Emergency Responder is a distinct ship, not a skin.
Goru's Shuttle, although not generally considered a "skinned" ship, is identical to the Caldari shuttle in stats.

Other than that, I think that list is accurate. However, I don't know if it is comprehensive. I can't be certain that every published ship is on that list.

The beatings will continue until posting improves. -Magnus Cortex

Official Eve Online changelist: Togglable PvP. - Jordanna Bauer
Tristan Ekain
The Scope
Gallente Federation
#14 - 2015-01-23 13:50:32 UTC
Sir Substance wrote:
The Opux Luxuary Yacht is a distinct ship, not a skin.
The Police Pursuit Comet is a skin, not a distinct ship.
The Stratios Emergency Responder is a distinct ship, not a skin.
Goru's Shuttle, although not generally considered a "skinned" ship, is identical to the Caldari shuttle in stats.

Other than that, I think that list is accurate. However, I don't know if it is comprehensive. I can't be certain that every published ship is on that list.


Wow, that was quick Smile
I have corrected the entries for these four ships in the Google spreadsheet.

Now that the list is complete, I can start with what I really want to achive: generating flashcards out of this list.
Sir Substance
Sebiestor Tribe
Minmatar Republic
#15 - 2015-01-23 14:14:41 UTC
Neat, drop a link here if you choose to release it?

The beatings will continue until posting improves. -Magnus Cortex

Official Eve Online changelist: Togglable PvP. - Jordanna Bauer
Tristan Ekain
The Scope
Gallente Federation
#16 - 2015-01-23 14:29:42 UTC
Sir Substance wrote:
Neat, drop a link here if you choose to release it?


Yes, I will release it.
The basic import is already working. Now I need to do some more fine-tuning and optimizing (e.g. image sizes), and find out how to handle future updates.
The software I'm using for this is Anki.
Maybe I'll upload something at the end of the weekend; that sounds quite realistic.
Tristan Ekain
The Scope
Gallente Federation
#17 - 2015-01-24 11:59:24 UTC
I have just released a first version of my flash card deck. See this forum thread: EVE Ships Flash Cards
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#18 - 2015-01-24 22:07:10 UTC
The pyfa team had similar issues when we wanted to reduce the number of skinned ships that show up on the ship browser. We also had to hardcode it. Any fit that might be imported is automaically assigned to it's base hull as determined by this conversion map:

https://github.com/DarkFenX/Pyfa/blob/master/service/conversions/skinnedShips.py

We try to keep this up to date with every expansion, so if anyone would like to use it to base their own mapping on feel free. Hopefully new skinning system will make this unneeded
Tristan Ekain
The Scope
Gallente Federation
#19 - 2015-01-24 22:13:23 UTC
Sable Blitzmann wrote:
The pyfa team had similar issues when we wanted to reduce the number of skinned ships that show up on the ship browser. We also had to hardcode it. Any fit that might be imported is automaically assigned to it's base hull as determined by this conversion map:

https://github.com/DarkFenX/Pyfa/blob/master/service/conversions/skinnedShips.py

We try to keep this up to date with every expansion, so if anyone would like to use it to base their own mapping on feel free. Hopefully new skinning system will make this unneeded

Nice, thanks for posting this. I'll compare my list with pyfa's.
SJ Astralana
Syncore
#20 - 2015-01-24 23:27:18 UTC
It's been some time since I wrote this, but between item meta group, and the existence of a blueprint where isPublished, I rather quickly filtered out all the faction crap.

Hyperdrive your production business: Eve Production Manager

12Next page