These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

zKillboard.com

First post
Author
Previous Topic Next Topic
Karbowiak
Sacred Templars
Fraternity.
#301 - 2013-11-05 20:16:24 UTC
Caliginous Ferus wrote:
Karbowiak wrote:

Hi!

The API documentation is a bit misleading atm, and lack some of the proper header attributes and such, but it's on the todolist for being rewritten.

As for the headers you have
x-bin-attempts-allowed = Tells you how many requests you can make before having to wait x-bin-seconds-between-requests
x-bin-requests = Tells you how many requests you have actually made in this bin.
x-bin-seconds-between-requests = Tells you the time to wait between filling each bin.

It was the most elegant way to tell do it.

However, we are considering switching to api keys that tie amount of requests to a users account on zkillboard. But before we go that route, we'd have to revamp parts of the API, to be easier to extend.


Hello again,
Thanks for the answer, do you mind if I follow up with a few other questions?
a) If/When you decide to change the way the API works, is it going to break all previous applications? IE, should I wait for the change to happen or is it not in your immediate plans?

b) If I need to grab a large amount of data (all kills for a number of regions in the past 2 weeks for example), what would be the best way to approach it?
Try to get as much data as I can in each request that happens every "x-bin-seconds-between-requests" seconds or get a couple of instances of an application that requests smaller pieces of data every "x-bin-seconds-between-requests" seconds?

Is there any other way to obtain this amount of data? Do you release any dumps?

Thanks again.


I don't mind, you ask all the questions you want! :)

a. We'll try not to make sweeping changes to the API as much as possible, we DO have a complete overhaul coming sometime in the near future, but we'll update the website, twitter and eve-kill a week or two before we actually do the change.

b. Fastest way is just to request page1,2,3,4,5 etc. of one region at a time. Just remember to space it out, if you want to space it out more than the x-bin-seconds-between-access dictates, go for it :)

Sadly we don't release any dumps, it's all available via the API :)

However, the 2nd version of the API, will ditch the requests pr. second idea, and instead impose a flat request amount, so you default get 5k requests pr. day, which is 1 request every 17.2 seconds, and you'll be in control of how fast you want to request stuff.

We can obv. add more to it on a case by case basis.

iskflakes wrote:
Hi guys,

Have you made any progress on API limits or API keys? The current limits are quite tight and hard to work with for realtime apps. Could you provide daily dumps of kills, or some kind of bulk access? This would solve most of my own access problems (I could cache the kills the day before and not have to hammer your server). Many thanks for the great tool.


The 2nd version of the API will use an API key, where we can up the amount of requests you can get access to.
But what realtime application are you trying to develop?

If you jump onto IRC and have a chat with us, we can more than likely come to some sort of agreement :)
Caliginous Ferus
Royal Amarr Institute
Amarr Empire
#302 - 2013-11-05 20:33:32 UTC
5k requests as in max 1m killmails per day (200*5000) or 5k killmails per day?
iskflakes
#303 - 2013-11-06 17:02:06 UTC
Karbowiak wrote:
However, the 2nd version of the API, will ditch the requests pr. second idea, and instead impose a flat request amount, so you default get 5k requests pr. day, which is 1 request every 17.2 seconds, and you'll be in control of how fast you want to request stuff.

We can obv. add more to it on a case by case basis.


This would work well for my app. The app I'm working on is a quick kill history checker (get a name, get all kill history, do some searching). If a user gives me a name which has 30 pages of kill history they currently have to wait something like 30 seconds for the app to fetch it all. Under the new API the app can grab those pages in a few seconds then sit inactive for the rest of the day (i.e. the average requests per day will be tiny, but the app will request in bursts of about 30 requests).

If you need any beta testers for the new API let me know :)

-
Karbowiak
Sacred Templars
Fraternity.
#304 - 2013-11-06 18:54:33 UTC
Caliginous Ferus wrote:
5k requests as in max 1m killmails per day (200*5000) or 5k killmails per day?


5k requests as 5k page requests.

loading https://zkillboard.com/api/blablablabla/ 5000 times.

iskflakes wrote:
Karbowiak wrote:
However, the 2nd version of the API, will ditch the requests pr. second idea, and instead impose a flat request amount, so you default get 5k requests pr. day, which is 1 request every 17.2 seconds, and you'll be in control of how fast you want to request stuff.

We can obv. add more to it on a case by case basis.


This would work well for my app. The app I'm working on is a quick kill history checker (get a name, get all kill history, do some searching). If a user gives me a name which has 30 pages of kill history they currently have to wait something like 30 seconds for the app to fetch it all. Under the new API the app can grab those pages in a few seconds then sit inactive for the rest of the day (i.e. the average requests per day will be tiny, but the app will request in bursts of about 30 requests).

If you need any beta testers for the new API let me know :)


you can play around with it on https://zkillboard.com/apiv2/ the requests are the same as before.

Just register an account on zKillboard, go to your account panel, click zKillboard API, and grab the API key that was generated :)
iskflakes
#305 - 2013-11-10 04:42:41 UTC
I tried to use apiv2 earlier and got an "invalid hash" error. The hash I used came straight from my account page. I guess it's still work in progress though :)

-
Squizz Caphinator
The Wormhole Police
#306 - 2013-11-11 14:36:07 UTC
iskflakes wrote:
I tried to use apiv2 earlier and got an "invalid hash" error. The hash I used came straight from my account page. I guess it's still work in progress though :)


I was looking over Karbo's apiv2 code the other day and there are some things that need to be addressed. Karbo said he'd get around to it....

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Squizz Caphinator
The Wormhole Police
#307 - 2013-11-12 02:14:22 UTC
Some changes have been made that will make folks using the API happy. A new field has been added to the JSON that is returned, zkb, and this field contains points and totalValue as calculated by zKillboard.

At this time, only kills processed after I pushed the changes are affected. I haven't decided yet how to handle the other 19.7 million killmails. But don't worry I'll come up with something!

Cheers!

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Squizz Caphinator
The Wormhole Police
#308 - 2013-11-13 13:53:59 UTC
Some more fixes to the API were done last night. This includes removing apiv2 (sorry karbo, but it had issues), and a severe fix to the rate limiting we put on folks using the API. While working on the API I noticed someone hit the API 98 times in 1 second... they had figured out a workaround, and I went ahead and fixed that.

Every API request will include the header X-Bin-Seconds-Between-Request, which tells you how long to wait between each request. If you fail to wait this long, you'll be given an error message telling you that you're requesting too fast, and then you'll have to wait X-Bin-Seconds-Between-Request seconds again to make your next request.

Right now X-Bin-Seconds-Between-Request is set to 10 seconds, which is very reasonable and allows you to make 8,640 calls a day.

If anyone has any questions feel free to post them, I'll get back to you. Or, you can create a ticket on zkb if you'd like to be a bit more private.

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Squizz Caphinator
The Wormhole Police
#309 - 2013-11-14 15:30:16 UTC
Eve-Kill's manual mail posting has been disabled. If you want to post manual mails you *must* use zKillboard.

Mails posted manually to zKillboard will not be shared with Eve-Kill. This is an executive decision for which we aren't sorry for (we hate manual mails).

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
iskflakes
#310 - 2013-11-14 17:46:24 UTC
Squizz Caphinator wrote:
Some more fixes to the API were done last night. This includes removing apiv2 (sorry karbo, but it had issues), and a severe fix to the rate limiting we put on folks using the API. While working on the API I noticed someone hit the API 98 times in 1 second... they had figured out a workaround, and I went ahead and fixed that.


This is unfortunate. When will APIv2 be returning? This is a killer-feature as far as I'm concerned.. opening up more killmail data to the wider community can only lead to good things.

-
Squizz Caphinator
The Wormhole Police
#311 - 2013-11-14 19:28:42 UTC
iskflakes wrote:
Squizz Caphinator wrote:
Some more fixes to the API were done last night. This includes removing apiv2 (sorry karbo, but it had issues), and a severe fix to the rate limiting we put on folks using the API. While working on the API I noticed someone hit the API 98 times in 1 second... they had figured out a workaround, and I went ahead and fixed that.


This is unfortunate. When will APIv2 be returning? This is a killer-feature as far as I'm concerned.. opening up more killmail data to the wider community can only lead to good things.


All the data is open, what makes you think it isn't?

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Karbowiak
Sacred Templars
Fraternity.
#312 - 2013-11-14 19:31:18 UTC
iskflakes wrote:
Squizz Caphinator wrote:
Some more fixes to the API were done last night. This includes removing apiv2 (sorry karbo, but it had issues), and a severe fix to the rate limiting we put on folks using the API. While working on the API I noticed someone hit the API 98 times in 1 second... they had figured out a workaround, and I went ahead and fixed that.


This is unfortunate. When will APIv2 be returning? This is a killer-feature as far as I'm concerned.. opening up more killmail data to the wider community can only lead to good things.


APIv2 was just going to be the 2nd version of the API for zKB, since it would have been a little easier to expand upon.

zKB however, still has an API that is fully open :)
https://zkillboard.com/information/api/
iskflakes
#313 - 2013-11-15 18:30:08 UTC
Squizz Caphinator wrote:
All the data is open, what makes you think it isn't?


I agree it's open, and that's great. I like the idea of apiv2 because of the more flexible request limits. The 10 second timer in apiv1 is quite hard to work with. I know you're still working on these things though :)

-
Squizz Caphinator
The Wormhole Police
#314 - 2013-11-17 07:41:55 UTC
iskflakes wrote:
Squizz Caphinator wrote:
All the data is open, what makes you think it isn't?


I agree it's open, and that's great. I like the idea of apiv2 because of the more flexible request limits. The 10 second timer in apiv1 is quite hard to work with. I know you're still working on these things though :)


Ah... well... apiv2 was going to limit you to about 5k requests per 24 hour period, api v1 gives you 8,640 requests per 24 hour period. So you're actually better off

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Kristoffon vonDrake
Forceful Resource Acquisition Inc
#315 - 2013-11-18 13:22:27 UTC  |  Edited by: Kristoffon vonDrake
So I'm trying to scratch up an app to do some stuff and I realized I can't just pull all kills because it has me specify regions or w-space. That is stupid. I'm sure you have good reasons for that but what I'll have to do instead is have to make more api calls to get all kills, one stupid region at a time.

Also if I use the AfterKillID to only get incremental updates are kill ids incremented by insertion order on the zkb database or am I risking missing some kills if they're provided by CCP and they're inserted out of order (ie are they sequential or not?)
iskflakes
#316 - 2013-11-18 19:51:46 UTC
Squizz Caphinator wrote:
Ah... well... apiv2 was going to limit you to about 5k requests per 24 hour period, api v1 gives you 8,640 requests per 24 hour period. So you're actually better off


For interactive purposes it's much worse if more than one page needs to be requested (the user has to wait 10 seconds). For bulk data gathering apiv1 is obviously better as you said.

Do you have a policy on people trying to download all killmails ever? Do you try and stop them or do you permit it as long as they stick to request limits?

-
Squizz Caphinator
The Wormhole Police
#317 - 2013-11-20 14:33:08 UTC  |  Edited by: Squizz Caphinator
iskflakes wrote:
Squizz Caphinator wrote:
Ah... well... apiv2 was going to limit you to about 5k requests per 24 hour period, api v1 gives you 8,640 requests per 24 hour period. So you're actually better off


For interactive purposes it's much worse if more than one page needs to be requested (the user has to wait 10 seconds). For bulk data gathering apiv1 is obviously better as you said.

Do you have a policy on people trying to download all killmails ever? Do you try and stop them or do you permit it as long as they stick to request limits?


I've modified the code to allow 8,640 requests per day at any rate you want. Same limits as before, just not as limiting, heh...

The previous headers are no more, there are now the following headers:
X-Bin-Request-Count - the number of requests you have in the last 24 hours (including the current request)
X-Bin-Max-Requests - the number of requests allowed in a 24 hour period

If someone wants to download all our killmails via the API they are more than welcome to do so, we have no policy for or against it. Its all there for anyone to use, we believe in being open about it. However, we aren't going to provide a dump of the killmails, they'll have to use the API. At 1.6 million mails per day it'll take them a couple of weeks, they'll just have to be patient :)

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Kristoffon vonDrake
Forceful Resource Acquisition Inc
#318 - 2013-11-20 17:36:25 UTC
Squizz, if I want to make incremental pulls from the database using the AfterKillID to get only kills since my last query is it guaranteed I'll get all killmails (IOW is that a sequential value zkb creates as it parses killmails) or do I risk missing some if they're inserted out of order (IOW is that a CCP-created value that could be out-of-order ie api for killmail #100 is pulled before api for killmail #101)?

Thank you!
Squizz Caphinator
The Wormhole Police
#319 - 2013-11-20 20:03:50 UTC
Kristoffon vonDrake wrote:
Squizz, if I want to make incremental pulls from the database using the AfterKillID to get only kills since my last query is it guaranteed I'll get all killmails (IOW is that a sequential value zkb creates as it parses killmails) or do I risk missing some if they're inserted out of order (IOW is that a CCP-created value that could be out-of-order ie api for killmail #100 is pulled before api for killmail #101)?

Thank you!


At this time you risk missing some if they are inserted out of order. I don't like it, you don't like it, but it is what it is at the moment. The data that handles *when* we got the kill is stored separately from the data that stores when the kill happened, and the size of the tables prevents joins to execute in a timely manner.

As we polish zkb into a more admin installer friendly killboard, this will be addressed. In the meantime please exercise patience.

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
iskflakes
#320 - 2013-11-21 21:08:11 UTC
Squizz Caphinator wrote:
I've modified the code to allow 8,640 requests per day at any rate you want. Same limits as before, just not as limiting, heh...


Thank you! This is much more usable, at least in my case. I'm getting much better responsiveness in my app now.

Squizz Caphinator wrote:
If someone wants to download all our killmails via the API they are more than welcome to do so, we have no policy for or against it. Its all there for anyone to use, we believe in being open about it. However, we aren't going to provide a dump of the killmails, they'll have to use the API. At 1.6 million mails per day it'll take them a couple of weeks, they'll just have to be patient :)


Thanks for the answer. It was mostly a hypothetical question because of the timeframe involved.

Keep up the great work!

-