These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

External killmail link hashing algorithm

First post First post
Author
Joshua Foiritain
Coreli Corporation
Pandemic Legion
#1 - 2014-08-08 11:25:14 UTC
TL;DR: I was wondering if anyone knew how the hash in the external killmail link is generated?

I've been trying to figure out a way to source kills my corp didnt get the final blow on that arent on zkb while retraining the accuracy the eve API/zkb API provide (item hierarchy, specialized hangars, proper item flags, etc). Obviously if members have the external killmail link they can simply post that to add the kill but this can be a lot of work depending on how much you kill during a fleet op. There also the problem that you cant pull these links from external killboards which makes them hard to track down afterwards. Copying killmails from other killboards aren't really an option as the mails lack a lot of info.

Pulling from killboards run by our allies is a problem because the quality of the killfeed varies, obviously any kill posted via mail will lack the detailed info i want, even if its API verified afterwards it doesnt update the kill with info from the API and then theres killboards whose killfeed data is malformed (rigs in cargo, weapons in wardrobe slots, etc).

So i've been looking at the external killmail link, if i pull the kill's api id + killmail from an external board i should technically have all the info needed to create a killmail link for the kill which i can then use to pull the kill directly from CCP, the only question is how the hash is generated; which algorithm is used, is it just the killmail or a killmail + salt?

I ran a killmail through a few of the standard algorithms but none of them matched the external killmail hash for that kill so either they're the wrong ones or extra info beyond the killmail is added prior to hashing. Does anyone know?

The Coreli Corporation is recruiting.

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#2 - 2014-08-08 13:00:51 UTC
nope. It's not open, and I doubt it will become so. Because if it did, then you could scrape kills you have no 'right' to.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Joshua Foiritain
Coreli Corporation
Pandemic Legion
#3 - 2014-08-08 13:39:15 UTC  |  Edited by: Joshua Foiritain
Steve Ronuken wrote:
nope. It's not open, and I doubt it will become so. Because if it did, then you could scrape kills you have no 'right' to.

Well you'd still need the kill's API ID so its not like being able to convert the mail into hash alone allows you to access any kill you want through the feed.

The only thing being able to hash the mails would allow you to do is verify the correctness of kills you already have the API ID and the killmail for. If you already have the killmail you already have access to the kill so its not like you're trying to scrape for hidden mails. The problem is that killmails lack a lot of details.

Obviously upgrading mails to contain this data would be an option but since the CREST option exists and is better that seems like a better option.

Alternatively if the killlog api returned all kills an entity was involved on rather then just the ones the entity has the final blow on all of this wouldn't matter because the corp api would provide me with a 100% accurate corp killboard, but Im assuming there's technical limitations preventing that. Sad

The Coreli Corporation is recruiting.

Dragonaire
Here there be Dragons
#4 - 2014-08-08 22:39:48 UTC
I just will have to say this seems to get asked about every couple of weeks and the few times CCP has replied they say no they won't do it. You can do a search on the forums here to find the details on why.

I did want to say something on still allowing people to add kills by the kill mails they receive also. It came up the last time there was a bigger discussion about making all kill mails public. Approximately 99% of the kills get posted already so if you get tied in with a few well maintain kill boards you don't need to allow them. Understand in most cases there are at least 4 people that have access to the kills through the API. There's the victim, their corp, the person with final blow, and their corp. As long as one of those people is running an API key driven kill board it should show up through the intra-board feeds and there's no need to use cut and pasted mails any more which as you said don't have all the info you might want. By not letting people use the cut & paste mails you also solve your other problems. Just some things to think about.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Joshua Foiritain
Coreli Corporation
Pandemic Legion
#5 - 2014-08-08 23:23:52 UTC  |  Edited by: Joshua Foiritain
Dragonaire wrote:
I just will have to say this seems to get asked about every couple of weeks and the few times CCP has replied they say no they won't do it. You can do a search on the forums here to find the details on why.

I did want to say something on still allowing people to add kills by the kill mails they receive also. It came up the last time there was a bigger discussion about making all kill mails public. Approximately 99% of the kills get posted already so if you get tied in with a few well maintain kill boards you don't need to allow them. Understand in most cases there are at least 4 people that have access to the kills through the API. There's the victim, their corp, the person with final blow, and their corp. As long as one of those people is running an API key driven kill board it should show up through the intra-board feeds and there's no need to use cut and pasted mails any more which as you said don't have all the info you might want. By not letting people use the cut & paste mails you also solve your other problems. Just some things to think about.

I don't think you actually read my post before responding? It seems like you're responding to me wanting to be able to pull any kill i want, including ones that havent been made public by the people involved, which is obviously not what im looking for...

Also iirc the number is 95% and eve-kill/zkb no longer syndicates from other boards because of abuse. A problem my suggestion would also solve as they would now have a way to verify kills themselves without having to rely on the third party feeder board being trustworthy.

The Coreli Corporation is recruiting.

Dragonaire
Here there be Dragons
#6 - 2014-08-09 03:29:31 UTC
Actually did read your post the point was CCP said NO. And one of the reason I've been told that some of the main boards stop accepting feeds is because some many of the boards were accepting copy and paste kill mails that were bogus and they got/get tired of dealing with them. Last I know though they still had outgoing feeds you can receive from so you might check if they still do that.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Joshua Foiritain
Coreli Corporation
Pandemic Legion
#7 - 2014-08-09 08:51:12 UTC
Dragonaire wrote:
Actually did read your post the point was CCP said NO. And one of the reason I've been told that some of the main boards stop accepting feeds is because some many of the boards were accepting copy and paste kill mails that were bogus and they got/get tired of dealing with them. Last I know though they still had outgoing feeds you can receive from so you might check if they still do that.

I covered pulling from boards in my original post, hell my entire post is about the problems that come with that which can be resolved by being able to verifify the correctness of a kill via CCP's server, you bringing it up in your previous post is why i thought you didn't read mine P

As for devs saying no, do you have any links to these threads? Eve's wonderful search function can only find 3 threads related to external kill mail links, one of which is mine, one is the announcement of their creation and the third is Squizz Caphinator suggesting killmail hashes in 2012 P

The Coreli Corporation is recruiting.

Dragonaire
Here there be Dragons
#8 - 2014-08-09 17:16:02 UTC  |  Edited by: Dragonaire
http://lmgtfy.com/?q=eve+online+public+kill+mails
Second link under first result was one of them I was think of that happened just 3 months ago. There was some other threads before that I think in about the same time frame with some more in depth talk as well but that one is probably a fair summary of how it went.

Edit:
Other link you might find useful I know you said they had shutdown the feeds but I have a feeling it more likely something just changed and you'll need to update your config
http://eve-kill.net/?a=idfeed

I tried a query manually in browser for a corp and it worked just fine so you really need to check your config as the feed is still active. If you use a different killboard you'll just need to do some searching around for how they do their feeds but I'll bet they are still available. They may have turned off receiving non API mails from outside but I'm sure they still make theirs available. I know there was talk about coming up with a better feed system a year or so ago but wasn't directly involved in them so don't know if they did anything or not.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

CCP FoxFour
C C P
C C P Alliance
#9 - 2014-08-09 20:27:50 UTC
Basically what has been said. No we will not tell you how to generate the hash.

zKillboard still syndicates their feeds from what I understand. I believe they have a publicly accessible stomp feed along with just their regular API.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Joshua Foiritain
Coreli Corporation
Pandemic Legion
#10 - 2014-08-10 21:49:33 UTC  |  Edited by: Joshua Foiritain
Dragonaire wrote:
http://lmgtfy.com/?q=eve+online+public+kill+mails
Second link under first result was one of them I was think of that happened just 3 months ago. There was some other threads before that I think in about the same time frame with some more in depth talk as well but that one is probably a fair summary of how it went.

I'm not asking for hidden killmails to be made public... I'm asking for a way to verify kills that provided to me from another killboards feed are real and accurate.

Dragonaire wrote:
Edit:
Other link you might find useful I know you said they had shutdown the feeds but I have a feeling it more likely something just changed and you'll need to /update your config
http://eve-kill.net/?a=idfeed

I tried a query manually in browser for a corp and it worked just fine so you really need to check your config as the feed is still active. If you use a different killboard you'll just need to do some searching around for how they do their feeds but I'll bet they are still available. They may have turned off receiving non API mails from outside but I'm sure they still make theirs available. I know there was talk about coming up with a better feed system a year or so ago but wasn't directly involved in them so don't know if they did anything or not.

Eve-Kill/Zkillboard no longer pulls from third party boards, of course they still have a feed you can pull from, i mentioned this in my first post... Straight

CCP FoxFour wrote:
Basically what has been said. No we will not tell you how to generate the hash.

:sadface: Any chance you could elaborate on why not though? I'm rather curious as to what the concerns behind releasing it are.

As Steve's scrapping concern isn't actually a risk, its more efficient to scrape every possible API/Hash combination then to try and generate every possible killmail combination, hash those, combine it with every possible API ID and scrape with that.

Dragonaire's concern that it could be used to find kills that arent public also isnt a risk because you need both the API ID and the killmail to access CREST, and having that information means the kill was made public by one of the people involved via their api.

CCP FoxFour wrote:
zKillboard still syndicates their feeds from what I understand. I believe they have a publicly accessible stomp feed along with just their regular API.

They still have a feed that can be pulled from but they no longer pull from other boards. Which means they end up missing kills which are available on the internet but not to them directly. The reason they stopped pulling kills from other boards is because of abuse, if they could use the CREST endpoint to verify that kills provided to them by other boards are real (since these boards provide the API ID and killmail, so you have all he data needed to access the CREST feed for confirmation except for the hashing method).

I recall you suggesting somewhere that killboards could provide the CREST link they receive from users on their kills like they do with killmails so that users could manually copy the CREST url into their board to transfer the kill over to their board. However killboards would only be able to provide the link if the kill was added via the link, if the kill was pulled directly from the API they could not provide the link because they dont have it. Wouldn't being able to generate the link if you already have the information required be logical next step?

Alternatively, any chance the text mails can be upgraded to include item flags so that we can see fleet hangars, specialized bays and such on them instead of everything being lumped under 'cargo'? Big smile

The Coreli Corporation is recruiting.

Kali Izia
GoomWaffe
#11 - 2014-08-10 23:43:31 UTC
It's definitely possible to generate the hash.
I've just thrown up http://km.goomfleet.com/ as a proof of concept, but it's not terribly useful for anything as it is.

If CCP doesn't want it shared I'm not going to go against their will and post details about the hash, although I do agree that there isn't much harm in it. You do need both the kill ID and various information from the killmail to do anything so scraping kills etc shouldn't really be a concern.
CCP FoxFour
C C P
C C P Alliance
#12 - 2014-08-11 10:40:23 UTC
Kali Izia wrote:
It's definitely possible to generate the hash.
I've just thrown up http://km.goomfleet.com/ as a proof of concept, but it's not terribly useful for anything as it is.

If CCP doesn't want it shared I'm not going to go against their will and post details about the hash, although I do agree that there isn't much harm in it. You do need both the kill ID and various information from the killmail to do anything so scraping kills etc shouldn't really be a concern.


Hahaha guess it was only time until someone figured it out. If you feel like sharing feel free to. :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Kali Izia
GoomWaffe
#13 - 2014-08-11 11:57:27 UTC
Alright, so the hash is:

sha1(victimCharacterID + attackerCharacterID + shipTypeID + killTime)

Where:

  • victimCharacterID = the character ID of the victim. If the characterID == 0, such as a kill for a POS mod or similar, this is the string "None" instead
  • attackerCharacterID = the character ID of the attacker that got the final blow. If the characterID == 0, such as an NPC, this is the string "None" instead
  • shipTypeID = the typeID of the victim's ship
  • killTime = a 64-bit timestamp. CCP luckily doesn't store precision greater than 1 second otherwise we couldn't do this, so you can use the following math to convert from a regular unix timestamp: (unixtime * 10000000) + 116444736000000000


Just grabbing a random kill from the front page of zKill that doesn't have a CREST link, we'll test with kill 40583728:
Our values are None + 93808108 + 24646 + 130521094800000000

sha1(None9380810824646130521094800000000) = efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d
= http://public-crest.eveonline.com/killmails/40583728/efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d/
CCP FoxFour
C C P
C C P Alliance
#14 - 2014-08-11 13:51:14 UTC
Kali Izia wrote:
Alright, so the hash is:

sha1(victimCharacterID + attackerCharacterID + shipTypeID + killTime)

Where:

  • victimCharacterID = the character ID of the victim. If the characterID == 0, such as a kill for a POS mod or similar, this is the string "None" instead
  • attackerCharacterID = the character ID of the attacker that got the final blow. If the characterID == 0, such as an NPC, this is the string "None" instead
  • shipTypeID = the typeID of the victim's ship
  • killTime = a 64-bit timestamp. CCP luckily doesn't store precision greater than 1 second otherwise we couldn't do this, so you can use the following math to convert from a regular unix timestamp: (unixtime * 10000000) + 116444736000000000


Just grabbing a random kill from the front page of zKill that doesn't have a CREST link, we'll test with kill 40583728:
Our values are None + 93808108 + 24646 + 130521094800000000

sha1(None9380810824646130521094800000000) = efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d
= http://public-crest.eveonline.com/killmails/40583728/efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d/


Think you are about to get a lot of love.

Mind posting it over here as well: https://wiki.eveonline.com/en/wiki/Equations

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Joshua Foiritain
Coreli Corporation
Pandemic Legion
#15 - 2014-08-11 14:12:13 UTC
Kali Izia wrote:
Alright, so the hash is:

sha1(victimCharacterID + attackerCharacterID + shipTypeID + killTime)

Where:

  • victimCharacterID = the character ID of the victim. If the characterID == 0, such as a kill for a POS mod or similar, this is the string "None" instead
  • attackerCharacterID = the character ID of the attacker that got the final blow. If the characterID == 0, such as an NPC, this is the string "None" instead
  • shipTypeID = the typeID of the victim's ship
  • killTime = a 64-bit timestamp. CCP luckily doesn't store precision greater than 1 second otherwise we couldn't do this, so you can use the following math to convert from a regular unix timestamp: (unixtime * 10000000) + 116444736000000000


Just grabbing a random kill from the front page of zKill that doesn't have a CREST link, we'll test with kill 40583728:
Our values are None + 93808108 + 24646 + 130521094800000000

sha1(None9380810824646130521094800000000) = efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d
= http://public-crest.eveonline.com/killmails/40583728/efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d/

Awww yissss, Pulling kills from EDK boards without having to worry about the quality of the data in the feed. Bear

You are a hero.

The Coreli Corporation is recruiting.

zKillboard
zKillboard.com
zKillboard Alliance
#16 - 2014-08-11 14:27:53 UTC
Kali Izia wrote:
awesome killmail stuff/


I <3 you
Masao Kurata
Perkone
Caldari State
#17 - 2014-08-11 14:47:38 UTC  |  Edited by: Masao Kurata
Now please add a random fraction of a second to kill any possibility of kill scraping. Or just outright salt it.

EDIT: I should be clearer about why this is, as it stands, a problem. 86400 seconds to a day is a fairly trivial number of requests if you're looking at one specific character, the main issue is that you have to combine two characters UNLESS it's an npc kill. Scraping NPC losses (CONCORD, sentry guns, pve accidents) is quite trivial and I'm sure someone could easily demonstrate that given the information in this thread. In the case of CONCORD and sentry guns that would give you a system and a narrow time range for a pvp killmail, which leaves the second character and ship type unknown. A bit more difficult to scrape but probably still possible especially combined with public data on active characters, of which there is some.

Combine these and you can scrape either pve losses or gank killmails, maybe some lowsec gate engagements too (although death by sentry there should be uncommon compared to highsec).

Salting the hash and including the salt in the kill information would do for verification.
Kali Izia
GoomWaffe
#18 - 2014-08-11 15:38:49 UTC
Masao Kurata wrote:
EDIT: I should be clearer about why this is, as it stands, a problem. 86400 seconds to a day is a fairly trivial number of requests

You also need the ID of the ship which adds to the number of requests.
Since you mentioned NPC losses you could always reduce this to common ratting/missioning/whatever ships, but even then it's still a large number of requests.

Assuming you respect the rate limits, it would take about 2.5 hours per ship per character to scrape a single day for NPC losses.
While it's possible to do in a day if you're targeting a specific character on a specific day and know what types of ships they might be flying, it's not something that can easily scale and the resulting intel is extremely minimal.

I can't see anyone putting in the time and effort just to figure out what fit you have on your ratting ship.
Masao Kurata
Perkone
Caldari State
#19 - 2014-08-11 18:31:08 UTC  |  Edited by: Masao Kurata
The fact that it's possible to do at all should be concerning. Scraping kills is not intended to be possible. And I picked the example of a highsec ganker for a reason, there are only 5 or 6 truly good ganking ships.
Squizz Caphinator
The Wormhole Police
#20 - 2014-08-11 20:35:35 UTC
Masao Kurata wrote:
The fact that it's possible to do at all should be concerning. Scraping kills is not intended to be possible. And I picked the example of a highsec ganker for a reason, there are only 5 or 6 truly good ganking ships.


Unless you already have the information as outlined above, you can't scrape killmails. If you do have that information, you already know everything but the killID - which is only of marginal value to those gathering intel.

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com

12Next page