These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

zKillboard.com

First post
Author
Previous Topic Next Topic
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#221 - 2013-04-17 19:21:53 UTC
STOMP, for zkill, is providing a stream of kill mails. As they come in, they get flung out onto the stomp stream.

Pretty much like what happens with EMDR.

Not quite so good for filling in gaps, but handy for keeping up to date. Or if you want to see indicative data (such as: what modules are popular)

It's not just PHP. There are stomp clients in many languages. http://stomp.github.io/implementations.html

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Poetic Stanziel
The Scope
Gallente Federation
#222 - 2013-04-17 20:35:49 UTC
Steve Ronuken wrote:
STOMP, for zkill, is providing a stream of kill mails. As they come in, they get flung out onto the stomp stream.

Pretty much like what happens with EMDR.

Not quite so good for filling in gaps, but handy for keeping up to date. Or if you want to see indicative data (such as: what modules are popular)

It's not just PHP. There are stomp clients in many languages. http://stomp.github.io/implementations.html
Thanks. That tells me a lot more.
Karbowiak
Sacred Templars
Fraternity.
#223 - 2013-04-17 20:48:51 UTC
Shellac Brookdale wrote:
Squizz Caphinator wrote:

We did some discussion on your request and decided that between the API and Stomp we already have the tools in place for everyone to get ahold of as many killmails as they like.


Wrote a simple python client one or two weeks ago to test the stomp push. It seems to work fine, but got killmails aged years ago which I'm not interested in. How can I only consume kms as they happen?


It might have been when we were moving over kills from EVE-KILL..

It only spits out new positive numbered killIDs (our manual mails are denoted by a negative killID)
Some kills from EVE-KILL wasn't on zKillboard and vice versa.

But there is a whole heap of documentation crap to write about the stomp stuff, there is for example various paths to listen to (like specific characterIDs or allianceIDs etc.)

That said, we MIGHT abandon the stomp stuff, we MIGHT keep it, we aren't entirely sure yet. Still a lot of stuff to figure out with it. Like, how do we keep it locked down and secure, but at the same time allow users to contribute servers to the overall network, and how do we allow user run clients (such as EVEMon) to push data to it, without being able to push fake data.

Lots of stuff to contemplate for it :)
Poetic Stanziel
The Scope
Gallente Federation
#224 - 2013-04-18 21:58:07 UTC
Corporate API keys for zKill are better than individual API keys.

So that we can encourage more people to supply corporate API keys to zKill, why not supply a page that is searchable, so that we can see if corps have supplied a corp-level API key to zKill/EVEKill? Before somebody posts a personal key, they can check to see if there's already corporate keys in place.

***

Since the corporate API only pulls the last 100 kills ... at what point does it become difficult to ensure that all killmails are grabbed? For a corporation on a particularly busy night, it could be possible to miss killmails, because over 100 kills occurred between API grab A and API grab B?
Karbowiak
Sacred Templars
Fraternity.
#225 - 2013-04-19 07:51:27 UTC
Poetic Stanziel wrote:
Corporate API keys for zKill are better than individual API keys.

So that we can encourage more people to supply corporate API keys to zKill, why not supply a page that is searchable, so that we can see if corps have supplied a corp-level API key to zKill/EVEKill? Before somebody posts a personal key, they can check to see if there's already corporate keys in place.

***

Since the corporate API only pulls the last 100 kills ... at what point does it become difficult to ensure that all killmails are grabbed? For a corporation on a particularly busy night, it could be possible to miss killmails, because over 100 kills occurred between API grab A and API grab B?


Currently you can only lookup corporate API keys for corps in an alliance.
But good idea. please drop it into: https://github.com/EVE-KILL/zKillboard/issues with a good description.

The solution to the issue of only getting the last 100 kills, you just supplement by either adding more corporate keys (The more keys there are, the more often zKB will fetch for said corp. Say there is 5 keys, it'll fetch every 12 minutes)
Poetic Stanziel
The Scope
Gallente Federation
#226 - 2013-04-19 08:40:57 UTC
Karbowiak wrote:
Since the corporate API only pulls the last 100 kills ... at what point does it become difficult to ensure that all killmails are grabbed? For a corporation on a particularly busy night, it could be possible to miss killmails, because over 100 kills occurred between API grab A and API grab B?


Currently you can only lookup corporate API keys for corps in an alliance.
But good idea. please drop it into: https://github.com/EVE-KILL/zKillboard/issues with a good description.

The solution to the issue of only getting the last 100 kills, you just supplement by either adding more corporate keys (The more keys there are, the more often zKB will fetch for said corp. Say there is 5 keys, it'll fetch every 12 minutes)[/quote]Are you reaching any hardware limitations, though? If you have 100,000 API keys, are you reaching a point where it is becoming difficult to pull all those keys in a single hour?

How many API keys do you guys track?

Do you have a method of prioritizing the keys? Perhaps based on past activity? Keys that generate more activity are given a high priority?
Poetic Stanziel
The Scope
Gallente Federation
#227 - 2013-04-19 09:52:46 UTC
Strange problem here:

Check this link -> http://poeticstanziel.blogspot.ca/2013/04/top-100-most-active-corporations-march.html

Second row, check the Red Fed and Blue Rep values for kills and losses.

For some reason my numbers are only half of what's being reported on EVE-Kill and zKillboard. Yet, my numbers for every other corp I've checked are (more or less) on the ball.

This continues for February too. My numbers for Red Fed and Blue Rep are approx. half of what's being reported on the EVE-Kill/ZKill websites.

I'm wondering, if perhaps, the problem is with EVE-Kill/zKill? Perhaps their numbers are being doubled up for some reason? Perhaps there's something else happening, that I'm not aware of.

I can't imagine I'm missing KMs just for the RvB corps ... especially considering I pull my KMs by region.

I'm kinda baffled.

Thanks.
Karbowiak
Sacred Templars
Fraternity.
#228 - 2013-04-19 16:21:03 UTC
Poetic Stanziel wrote:
Karbowiak wrote:
Since the corporate API only pulls the last 100 kills ... at what point does it become difficult to ensure that all killmails are grabbed? For a corporation on a particularly busy night, it could be possible to miss killmails, because over 100 kills occurred between API grab A and API grab B?

Currently you can only lookup corporate API keys for corps in an alliance.
But good idea. please drop it into: https://github.com/EVE-KILL/zKillboard/issues with a good description.

The solution to the issue of only getting the last 100 kills, you just supplement by either adding more corporate keys (The more keys there are, the more often zKB will fetch for said corp. Say there is 5 keys, it'll fetch every 12 minutes)


Are you reaching any hardware limitations, though? If you have 100,000 API keys, are you reaching a point where it is becoming difficult to pull all those keys in a single hour?

How many API keys do you guys track?

Do you have a method of prioritizing the keys? Perhaps based on past activity? Keys that generate more activity are given a high priority?


We don't have any hardware limits, we currently have 29724 valid APIs, where we do about 45k API requests pr. hour (~12 pr. second).

Which is nowhere near what we can actually pull through, i've seen it do ~80k once.

As for prioritizing, no, we just fetch based on which are available this minute. Everything is cached for an hour, and refetched after an hour.


Poetic Stanziel wrote:
Strange problem here:

Check this link -> http://poeticstanziel.blogspot.ca/2013/04/top-100-most-active-corporations-march.html

Second row, check the Red Fed and Blue Rep values for kills and losses.

For some reason my numbers are only half of what's being reported on EVE-Kill and zKillboard. Yet, my numbers for every other corp I've checked are (more or less) on the ball.

This continues for February too. My numbers for Red Fed and Blue Rep are approx. half of what's being reported on the EVE-Kill/ZKill websites.

I'm wondering, if perhaps, the problem is with EVE-Kill/zKill? Perhaps their numbers are being doubled up for some reason? Perhaps there's something else happening, that I'm not aware of.

I can't imagine I'm missing KMs just for the RvB corps ... especially considering I pull my KMs by region.

I'm kinda baffled.

Thanks.


Probably some sort of issue with the fetching on your end, or maybe it's cause a lot of their mails are manually posted, dunno :)
Poetic Stanziel
The Scope
Gallente Federation
#229 - 2013-04-19 21:00:04 UTC  |  Edited by: Poetic Stanziel
Karbowiak wrote:
Probably some sort of issue with the fetching on your end, or maybe it's cause a lot of their mails are manually posted, dunno :)
Are a lot of RvB mails manually posted (not API verified)? Half of them? (Because I am pulling API-only KMs.)

I wouldn't begin to know how to solve the issue, if it is a fetching problem. But why would my fetching problem only be affecting RvB?

Could you run a query for me? Tell me how many of their March 2013 lossmails and killmails are manually posted? And perhaps tell me the KillID (the KillID that CCP uses) range for March 2013 (starting KillID and ending KillID for that month). Maybe I can pinpoint the problem.
Poetic Stanziel
The Scope
Gallente Federation
#230 - 2013-04-20 03:09:14 UTC
Could Squizz or Karb read the following and set me straight on any errors I might have made?

http://poeticstanziel.blogspot.ca/2013/04/your-api-keys-killmails-and-you.html

Thanks.
Karbowiak
Sacred Templars
Fraternity.
#231 - 2013-04-20 06:44:27 UTC
Poetic Stanziel wrote:
Karbowiak wrote:
Probably some sort of issue with the fetching on your end, or maybe it's cause a lot of their mails are manually posted, dunno :)
Are a lot of RvB mails manually posted (not API verified)? Half of them? (Because I am pulling API-only KMs.)

I wouldn't begin to know how to solve the issue, if it is a fetching problem. But why would my fetching problem only be affecting RvB?

Could you run a query for me? Tell me how many of their March 2013 lossmails and killmails are manually posted? And perhaps tell me the KillID (the KillID that CCP uses) range for March 2013 (starting KillID and ending KillID for that month). Maybe I can pinpoint the problem.


You can't really solve this issue, unless we can somehow retroactively get the kills from both R and B api verified from a trusted source.

I think you got the answer for the query part from Squizz last night :)

Poetic Stanziel wrote:
Could Squizz or Karb read the following and set me straight on any errors I might have made?

http://poeticstanziel.blogspot.ca/2013/04/your-api-keys-killmails-and-you.html

Thanks.


Yep, it seems about right, however the 119 issue SHOULD HOPEFULLY go away before Odyssey is out. Atleast PrismX hinted as much.

And as for a public kill feed, zKB will try and fill that gap as best possible both by the use of it's API, and with the use of the STOMP stuff Blink
Poetic Stanziel
The Scope
Gallente Federation
#232 - 2013-04-20 07:03:23 UTC  |  Edited by: Poetic Stanziel
Karbowiak wrote:
And as for a public kill feed, zKB will try and fill that gap as best possible both by the use of it's API, and with the use of the STOMP stuff Blink
You keep talking STOMP, and also keep saying you might not use it in the future. So I'll ignore writing any code specific to it, until I know you guys are going to stick with it. Could you describe the benefit of using STOMP tho, versus the traditional API calls to zKill? I think I kind of understand what STOMP is doing, but not quite sure what the benefit of it is over just doing what I'm doing now.

In terms of your API ... let's say I post a manual lossmail to zKill/EVEKill, which I often do. How long, maximum, before that manual lossmail (with the negative ID) is transformed into an API-verified killmail (with a positive ID)? I'm guessing no more than two hours, basically the length of time it will take you to request my kills/losses using the API key I have registered with you folks.

Check the following API call (I don't use this in my code, I was using it to check out some data associated with manual killmails):

https://zkillboard.com/api/xml/no-items/corporationID/1699307293,1741770561/beforeKillID/29598374/page/40/

In the call, all I do is ask for any killmail before ID 29598374 and 40 pages back. Now, there are a lot of manual killmails here, all with the expected negative IDs. Your query obviously isn't just using a "WHERE ID < 29598374" clause, otherwise all those manual IDs wouldn't show up at all. Are you creating a dummy ID (in the expected range) for the manual kills, or are you transforming the "WHERE ID < someID" into a "WHERE date < someDate" filter? Curious.

Also ... would it be possible to add in a /manual-only/ parameter to the API calls, much as you have an /api-only/ parameter? In case I do want to add manual kills in the future, I can filter them specifically, rather than end up grabbing a bunch of api-only data that I already have.
Karbowiak
Sacred Templars
Fraternity.
#233 - 2013-04-20 07:20:34 UTC
Poetic Stanziel wrote:
Karbowiak wrote:
And as for a public kill feed, zKB will try and fill that gap as best possible both by the use of it's API, and with the use of the STOMP stuff Blink
You keep talking STOMP, and also keep saying you might not use it in the future. So I'll ignore writing any code specific to it, until I know you guys are going to stick with it. Could you describe the benefit of using STOMP tho, versus the traditional API calls to zKill? I think I kind of understand what STOMP is doing, but not quite sure what the benefit of it is over just doing what I'm doing now.

In terms of your API ... let's say I post a manual lossmail to zKill/EVEKill, which I often do. How long, maximum, before that manual lossmail (with the negative ID) is transformed into an API-verified killmail (with a positive ID)? I'm guessing no more than two hours, basically the length of time it will take you to request my kills/losses using the API key I have registered with you folks.

Check the following API call (I don't use this in my code, I was using it to check out some data associated with manual killmails):

https://zkillboard.com/api/xml/no-items/corporationID/1699307293,1741770561/beforeKillID/29598374/page/40/

In the call, all I do is ask for any killmail before ID 29598374 and 40 pages back. Now, there are a lot of manual killmails here, all with the expected negative IDs. Your query obviously isn't just using a "WHERE ID < 29598374" clause, otherwise all those manual IDs wouldn't show up at all. Are you creating a dummy ID (in the expected range) for the manual kills, or are you transforming the "WHERE ID < someID" into a "WHERE date < someDate" filter? Curious.

Also ... would it be possible to add in a /manual-only/ parameter to the API calls, much as you have an /api-only/ parameter? In case I do want to add manual kills in the future, I can filter them specifically, rather than end up grabbing a bunch of api-only data that I already have.


The benefit of STOMP is that we can push out mails as we get them, without people having to poke our api for it.
Other than that, there is no real benefit :)

It takes a maximum of two hours (if we have the api key where the kill is located, and it's not 119'ing) before we have the kill.

You set it to beforeKillID, meaning anything before 26598374 (meaning both negative and positive upto 26598374) - also you didn't add /api-only/ - thats why it shows manual kills aswell :)

/manual-only/ is doable yes, throw it at our issues list and someone will do it soonishly :)

https://github.com/EVE-KILL/zKillboard/issues
Poetic Stanziel
The Scope
Gallente Federation
#234 - 2013-04-20 07:35:08 UTC
Karbowiak wrote:
You set it to beforeKillID, meaning anything before 26598374 (meaning both negative and positive upto 26598374) - also you didn't add /api-only/ - thats why it shows manual kills aswell :)
I know that.

I'm wondering why the manual kills are appearing in date sequence order that ... never mind ... you're probably just doing an ORDER BY killDate on all the queries by default.

killIDs (API verified only) are guaranteed to be in sequential order? (I should have asked this weeks ago, but assumed it was the case.) A killID on Feb 01 2013 will always be less than a killID on Feb 02 2013, as an example?
Karbowiak
Sacred Templars
Fraternity.
#235 - 2013-04-20 07:38:51 UTC
Poetic Stanziel wrote:
Karbowiak wrote:
You set it to beforeKillID, meaning anything before 26598374 (meaning both negative and positive upto 26598374) - also you didn't add /api-only/ - thats why it shows manual kills aswell :)
I know that.

I'm wondering why the manual kills are appearing in date sequence order that ... never mind ... you're probably just doing an ORDER BY killDate on all the queries by default.

killIDs (API verified only) are guaranteed to be in sequential order? (I should have asked this weeks ago, but assumed it was the case.) A killID on Feb 01 2013 will always be less than a killID on Feb 02 2013, as an example?


It's all ordered by dateTime, but assuming CCP increments the killID, then yes, they're pretty surely in sequential order, with a few gaps here and there since we can't get all mails there has ever been.
Squizz Caphinator
The Wormhole Police
#236 - 2013-04-20 20:30:42 UTC
Poetic, yes, when you specify before a killID it is converted into a date. All CCP killiDs are sequential in order and date, so killID 123 happened before killID 321. This does NOT apply to manual killID's, which are purely random because we cannot allocate an ID for a kill we don't know about (obviously).

If you haven't already, create an issue for a manual-only filter on the API and I'll get it done Monday for you. HOWEVER! Any oddities that come from the API involving only manual kills will just be ignored. You have to deal with those yourself :)

( I never wanted manual kills, I still don't, and the less I have to do with them the happier I am )

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Poetic Stanziel
The Scope
Gallente Federation
#237 - 2013-04-20 20:36:09 UTC
Squizz Caphinator wrote:
Poetic, yes, when you specify before a killID it is converted into a date. All CCP killiDs are sequential in order and date, so killID 123 happened before killID 321. This does NOT apply to manual killID's, which are purely random because we cannot allocate an ID for a kill we don't know about (obviously).

If you haven't already, create an issue for a manual-only filter on the API and I'll get it done Monday for you. HOWEVER! Any oddities that come from the API involving only manual kills will just be ignored. You have to deal with those yourself :)

( I never wanted manual kills, I still don't, and the less I have to do with them the happier I am )
I'll wait until after Fanfest. There may be news on the API via the devtracks. Seems PrismX wants to redo the Kill Log API.
Karbowiak
Sacred Templars
Fraternity.
#238 - 2013-04-21 18:14:15 UTC
So apparently the TEST devs have switched to zKB..

https://zkb.pleaseignore.com/

Incase you guys look here, please do send code fixes back incase you fix something Blink
Karbowiak
Sacred Templars
Fraternity.
#239 - 2013-04-23 16:16:23 UTC
Tiny updates.

1. We changed the license from the derivative Kingboard derivative GPL license, to AGPLv3 - should probably help alleviate any licensing concerns some of you may have had.

2. We are currently developing a live map showing kills as they happen, yep, you read correct. Big smile

3. We have just revamped the interface for the Subdomain stuff, so now it actually works. Multiple entities is still missing, and so is a way to use a custom subdomain. But atleast one of them will get fixed soonishly Smile

Other than that, work progresses!
Poetic Stanziel
The Scope
Gallente Federation
#240 - 2013-04-24 03:39:51 UTC  |  Edited by: Poetic Stanziel
I think one of the problems with a lot of the RvB killmails not being API verified is that many times they are recording greater than 100 kills+losses per hour. Which means killmails that cannot be retrieved (unless you're lucky enough to have other keys which can capture those "lost" killmails.)

I've been talking with Mangala Solaris and have convinced him to register more RvB keys for each CEO/Director they have.

Now, if those keys are used within a couple minutes of each other, the problem is not really being solved.

I'm wondering if your code spaces multiple keys from a single corporation with an appropriate timeframe? For instance, let's say RedFed has three keys, are those three keys being pulled twenty minutes apart from each other?