These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

[EVSCO/EVE-KILL] EPIC API (Yes, it is actually EPIC) (DEAD)

Author
Khorkrak
KarmaFleet
Goonswarm Federation
#81 - 2012-06-17 14:49:59 UTC  |  Edited by: Khorkrak
Not quite seeing that. Here's a different example - note the dates and times specified:

-- Get everything for January 2012 but limit it to 500 since I seem to be getting stuff at the end of the end anyway so lets try it
> wget -O results.dat http://eve-kill.net/epic/startDate:2012-01-01_00.00.00/endDate:2012-01-31_23.59.59/mailLimit:500/mask:1607166

> python
Python 2.7.3 (default, Apr 20 2012, 22:44:07)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import simplejson as json
>>> kills = json.load(file("results.dat"))
>>> dates = [k["timestamp"] for k in kills]
>>> dates.sort()
>>> dates[0]
'2012-01-31 22:42:00'
>>> dates[-1]
'2012-01-31 23:59:00'
>>>

So the earliest datetime is 2012-01-31 22:24:00 which makes sense based on the descending order you mentioned above. That's fine. Now lets set the end date to the earliest date from above and requery.

> wget -O results.dat http://eve-kill.net/epic/startDate:2012-01-01_00.00.00/endDate:2012-01-31_22.42.00/mailLimit:500/mask:1607166

> python
Python 2.7.3 (default, Apr 20 2012, 22:44:07)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import simplejson as json
>>> kills = json.load(file("results.dat"))
>>> dates = [k["timestamp"] for k in kills]
>>> dates.sort()
>>> dates[0]
'2012-01-31 22:42:00'
>>> dates[-1]
'2012-01-31 23:59:00'
>>>

Now I'm perplexed, I got the same data even with a different end datetime Why are kills returned that are outside of the datetime range specified? What am I doing wrong? Seems like the time part is being ignored - maybe I'm not specifying it correctly - I'll double check the instructions.

Developer of http://www.decloaked.com and http://sourceforge.net/projects/pykb/

Etil DeLaFuente
Aliastra
Gallente Federation
#82 - 2012-06-17 22:57:18 UTC
Might have something to do with time. I never specify it. Maybe you should filter a bit more too, do you really need all the killmails ?
Nomad I
University of Caille
Gallente Federation
#83 - 2012-06-25 19:41:53 UTC
What is with the values like Pilot Efficiency, Damage Done, Damage received, Kills and losses related to a month or a year?
Karbowiak
Sacred Templars
Fraternity.
#84 - 2012-07-12 18:39:42 UTC
I decided to add rawmail support, seeing as i'll be working on a feed mechanism for EDK, using EPIC rather than IDFeed.

Seeing as IDFeed is horrible..

To use it, use the mask id 2097152
Everything is serialized, gzipped and base64_encoded, to decode it and get the rawmail, in php it's done the following way.

$mail = unserialize(gzuncompress(base64_decode($rawmail)));

Anyway, have fun with it, and don't abuse it, please :)
Shellac Brookdale
Cutting Edge Incorporated
#85 - 2012-07-18 19:16:10 UTC
Any chance to provide the killmail in the same xml format idfeed uses? That would make migration from idfeed to epic much easier.
Karbowiak
Sacred Templars
Fraternity.
#86 - 2012-07-19 18:07:48 UTC
Shellac Brookdale wrote:
Any chance to provide the killmail in the same xml format idfeed uses? That would make migration from idfeed to epic much easier.


that'd require i translate it all into an xml sheet, which isn't gonna happen, ever..
It's json and then this pseudo gzipped thingy raw mail :)
Karbowiak
Sacred Templars
Fraternity.
#87 - 2012-07-19 18:23:01 UTC  |  Edited by: Karbowiak
Added a new method called orderBy

It can be used with DESC and ASC..

But now the startDate and endDate methods work as expected (unless you select to use DESC, then it fails again)
And minKLLid and maxKLLid also works, which was due to a bug in the else if structure (apparently it kicked kllid before minkllid, who knew!)

Now im off to code a mail fetcher for EDK using EPIC! woo Pirate
Karbowiak
Sacred Templars
Fraternity.
#88 - 2012-07-19 21:13:17 UTC
another bug fixed, apparently i forgot to urldecode stuff, so it would fail on some names, that is now fixed! Lol
Karbowiak
Sacred Templars
Fraternity.
#89 - 2012-07-19 21:53:18 UTC
I have altered the space to a + instead of a _

No worries, if you havn't done it, it does it for you, aslong as CCP doesn't allow _ in names, we'll be fine Lol
Karbowiak
Sacred Templars
Fraternity.
#90 - 2012-07-20 15:25:41 UTC
im currently redoing the raw part, i'll update this post with information (apparently doing it with a raw mail was insanely horrible, who knew! Lol)
Karbowiak
Sacred Templars
Fraternity.
#91 - 2012-07-21 20:46:11 UTC
raw mail fetching has been fixed, and i have also altered the layout of items.
Sorry to those of you who used it for something, but it's better this way Big smile

Also tossed together an EDK4 killmail fetcher, that uses EPIC, which you can find here
http://eve-id.net/forum/viewtopic.php?f=505&t=18815

Enjoy!
Veshai
NED-Clan
Goonswarm Federation
#92 - 2012-07-21 22:33:07 UTC  |  Edited by: Veshai
Fetching raw mails doesn't seem to be working anymore. Using mask '2097152' I don't get any raw mails anymore. It has worked before... did something break?

[edit]
(I also get data returned that I didn't ask for btw, so something is up with the mask parsing)

[edit2]
Oh god, from your php script for the killboard I see you're reassembling the killmail by parsing the different things. Does this mean you won't fix the "raw" mail anymore? If CCP changes the format and the killboard is updated for that, you still need to update your script. You just added an extra place to fix it in whenever that happens.
Karbowiak
Sacred Templars
Fraternity.
#93 - 2012-07-22 01:13:54 UTC
Veshai wrote:
Fetching raw mails doesn't seem to be working anymore. Using mask '2097152' I don't get any raw mails anymore. It has worked before... did something break?

[edit]
(I also get data returned that I didn't ask for btw, so something is up with the mask parsing)

[edit2]
Oh god, from your php script for the killboard I see you're reassembling the killmail by parsing the different things. Does this mean you won't fix the "raw" mail anymore? If CCP changes the format and the killboard is updated for that, you still need to update your script. You just added an extra place to fix it in whenever that happens.


I'll sort a rawApi tomorrow, that spits things out just like the API does (just this is in JSON)
This said however, i'll be spending most of my time coding on kingboard.

However, EPIC will be ported to that aswell (Where i can also fix the some of the more, uh, glaring mistakes i made this time around)
Jack Tronic
borkedLabs
#94 - 2012-07-22 01:43:02 UTC  |  Edited by: Jack Tronic
Glad to see we are server banned for still querying ID feed from a server running multiple killboards.

Quote:
# What are you doing with idfeed?/FB
Karbowiak
Sacred Templars
Fraternity.
#95 - 2012-07-22 08:32:23 UTC
Jack Tronic wrote:
Glad to see we are server banned for still querying ID feed from a server running multiple killboards.

Quote:
# What are you doing with idfeed?/FB


i dont remember the specific URL you used to query idfeed, i just know that we banned several users of idfeed, for hammering it either too much over a period of itme, or by using way, WAY too many concurrent IDs (allianceID=id1,id2,id3,id4,id5 etc.)

You probably ended up among them.

Best way to make sure that doesn't happen is by not fetching more than once every 15 minutes, and by doing multiple requests (one / two id's pr. request)
Even if you decided to add 200 id's, you'd still be limited to 50 results via idfeed.

EPIC is better tho, since it can deliver 500 results in less time than it takes idfeed to deliver 50, so there's that.. P
Veshai
NED-Clan
Goonswarm Federation
#96 - 2012-07-22 11:16:21 UTC  |  Edited by: Veshai
Karbowiak wrote:
I'll sort a rawApi tomorrow

Glad to hear, thanks in advance.

I've also noticed a serious problem when you use minKLLid. I think I can best explain it with an example. The mask is simplified to make it more visible.

Retrieving up to 50 mails when my last mailid was known to be 14087198:
http://eve-kill.net/epic/mask:4/mailLimit:50/minKLLid:14087198/combinedCorp:NED-Clan

I receive only one mail. However, new killmails are present, which becomes visible when increasing the limit:
http://eve-kill.net/epic/mask:4/mailLimit:100/minKLLid:14087198/combinedCorp:NED-Clan

Now I get two mails. Increasing the limit even further:
http://eve-kill.net/epic/mask:4/mailLimit:150/minKLLid:14087198/combinedCorp:NED-Clan

I now get three mails.

My conclusion is that mailLimit seems to mimic maxKLLid by doing a minKLLid:14087198 and maxKLLid:14087198+50

Your script has an implied maiLimit of "100" when unspecified:
Quote:
mailLimit // Limits amount of mails show, minimum 1, maximum 500, default is 100
This means that if there are more than 100 kills posted in the time that it takes your corp to get on another killmail, you will never get a new killmail anymore as it never finds a newer mail than the one you are quering from. For example, if I would use the first link with a limit of "50" that I posted above, my script would never find out that there are new kills.

Your cron_epic.php also uses the minKLLid mechanism btw, so it is also vulnerable to this problem.
Jita Bloodtear
Bloodtear Labs
#97 - 2012-07-22 19:58:13 UTC
I haven't sifted through this entire thread, and I don't normally come to these forums. But over the past few weeks we've noticed that evekill has become increasingly unstable and is prone to being down for lengthier periods of time. I'm assuming that it's related to this new API stuff you're up to. The message on eve-kill says that it's currently cutoff from CCP's API system entire. Do you have any more information on why this is or what's going on?
Karbowiak
Sacred Templars
Fraternity.
#98 - 2012-07-23 01:14:48 UTC
Veshai wrote:
Karbowiak wrote:
I'll sort a rawApi tomorrow

Glad to hear, thanks in advance.

I've also noticed a serious problem when you use minKLLid. I think I can best explain it with an example. The mask is simplified to make it more visible.

Retrieving up to 50 mails when my last mailid was known to be 14087198:
http://eve-kill.net/epic/mask:4/mailLimit:50/minKLLid:14087198/combinedCorp:NED-Clan

I receive only one mail. However, new killmails are present, which becomes visible when increasing the limit:
http://eve-kill.net/epic/mask:4/mailLimit:100/minKLLid:14087198/combinedCorp:NED-Clan

Now I get two mails. Increasing the limit even further:
http://eve-kill.net/epic/mask:4/mailLimit:150/minKLLid:14087198/combinedCorp:NED-Clan

I now get three mails.

My conclusion is that mailLimit seems to mimic maxKLLid by doing a minKLLid:14087198 and maxKLLid:14087198+50

Your script has an implied maiLimit of "100" when unspecified:
Quote:
mailLimit // Limits amount of mails show, minimum 1, maximum 500, default is 100
This means that if there are more than 100 kills posted in the time that it takes your corp to get on another killmail, you will never get a new killmail anymore as it never finds a newer mail than the one you are quering from. For example, if I would use the first link with a limit of "50" that I posted above, my script would never find out that there are new kills.

Your cron_epic.php also uses the minKLLid mechanism btw, so it is also vulnerable to this problem.


minKLLid needs maxKLLid to work.
If you don't set maxKLLid it auto defines it as minKLLid+mailLimit
And if you don't define mailLimit, it defaults to 100.

also it's picky (due to me being a tard when coding), so it needs maxKLLid to be BEFORE minKLLid in the url.

Also we've discovered that sometimes maxkllid have to be way way WAY bigger for it to still get mails (due to there being no mails between two IDs for a pilot / corp / alliance)

Some have reported it having to be minKLLid+50000 others +500000
So it's a bit weird.

It's all due to how EDK is coded, so there isn't much i can do about that.
It'll get better with Kingboard tho! :)


Jita Bloodtear wrote:
I haven't sifted through this entire thread, and I don't normally come to these forums. But over the past few weeks we've noticed that evekill has become increasingly unstable and is prone to being down for lengthier periods of time. I'm assuming that it's related to this new API stuff you're up to. The message on eve-kill says that it's currently cutoff from CCP's API system entire. Do you have any more information on why this is or what's going on?


We have about 20k API keys that we fetch every hour, apparently CCP doesn't like that, and decided to ban us.
But we have been unbanned on a few of our IPs now, and have started fetching mails via the API again, including opened up API key submissions.

No guarantees we wont get banned again tho :\
( and yes, we have lowered the speed of which we fetch stuff aswell )
Veshai
NED-Clan
Goonswarm Federation
#99 - 2012-07-23 11:36:43 UTC  |  Edited by: Veshai
Karbowiak wrote:
Also we've discovered that sometimes maxkllid have to be way way WAY bigger for it to still get mails (due to there being no mails between two IDs for a pilot / corp / alliance)

Some have reported it having to be minKLLid+50000 others +500000
So it's a bit weird.

I expected it would be possible to say "here's the last known killid that I know, give me up to 100 new mails starting from that id", but as I understand from what you say, this is not possible?

What is your recommended way of retrieving new mails?

Currently I'm just using http://eve-kill.net/epic/mask:2097164/mailLimit:500/combinedPilot:NED-Clan so that I always get the latest 500 mails for NED-Clan.

If however our website would go down for a few days, and our corp would get >500 kills in that time, any kills older than those last 500 wouldn't get synced.

Would you say using minKLLid of last known killid and a maxKLLid of minKLLid+10000000 would really be the best way to go?
Desmont McCallock
#100 - 2012-07-23 17:20:22 UTC
Is there any interest in establishing a data relay network for Kill reports much like EMDR (EVE Market Data Relay)?

If so I'm more than willing to provide format standards (much like the UUDIF) and add uploading capabilities to EVEMon.
As I have already spoken with Ilyk (the pioneer of the EMDR), his software can be tweaked to serve the purpose but he's not willing to take on maintaining the network (totally understandable as he has already too much to do).