These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

[EVSCO/EVE-KILL] EPIC API (Yes, it is actually EPIC) (DEAD)
Author
Previous Topic Next Topic
Karbowiak
Sacred Templars
Fraternity.
#101 - 2012-07-24 05:47:30 UTC  |  Edited by: Karbowiak
Veshai wrote:
Karbowiak wrote:
Also we've discovered that sometimes maxkllid have to be way way WAY bigger for it to still get mails (due to there being no mails between two IDs for a pilot / corp / alliance)

Some have reported it having to be minKLLid+50000 others +500000
So it's a bit weird.

I expected it would be possible to say "here's the last known killid that I know, give me up to 100 new mails starting from that id", but as I understand from what you say, this is not possible?

What is your recommended way of retrieving new mails?

Currently I'm just using http://eve-kill.net/epic/mask:2097164/mailLimit:500/combinedPilot:NED-Clan so that I always get the latest 500 mails for NED-Clan.

If however our website would go down for a few days, and our corp would get >500 kills in that time, any kills older than those last 500 wouldn't get synced.

Would you say using minKLLid of last known killid and a maxKLLid of minKLLid+10000000 would really be the best way to go?


minKLLid with a maxKLLid of, say, +2000 or +20000 should do fine.

But due to the way the db is split, sometimes the next id is in a different table (say last mail is from 2012_07, next is in 2012_08) and it messes it up, which is annoying, and worse is that there is no clear way to get around it.
Someone smarter than me can probably figure it out, lol..

Desmont McCallock wrote:
Is there any interest in establishing a data relay network for Kill reports much like EMDR (EVE Market Data Relay)?

If so I'm more than willing to provide format standards (much like the UUDIF) and add uploading capabilities to EVEMon.
As I have already spoken with Ilyk (the pioneer of the EMDR), his software can be tweaked to serve the purpose but he's not willing to take on maintaining the network (totally understandable as he has already too much to do).


Something like this has already been proposed, but the idea then spun into the killboard being api only in that case.
The thinking behind that is, corporations and pilots might be more inclined to giving away their api killlog, if they know they are the only ones who can see their api key.
But the entire system has to be protected against killlog manipulation somehow.

This said, i'm more than a little interested in seeing such a system put into Kingboard. The thought was to use it for realtime comments and killlist updating aswell, but im not sure if that is feasible (or even practical).

oh and yes, we could maintain the network, god knows we have so much bandwidth available atm, that we aren't even using 1/10th of the amount we're paying for. lol
Desmont McCallock
#102 - 2012-07-24 06:57:59 UTC
Kill report manipulation is indeed a possibility but I think that the time it will need to code such a system or alter an existing uploader to send malicious data, is just not worth it.
Protective mechanism can be applied in those case like cross-checking the kill report between victim-attackers.

Indeed the greater issue here is the privacy of the info, which probably is the blocker in bringing this project to life.
Etil DeLaFuente
Aliastra
Gallente Federation
#103 - 2012-07-24 07:22:36 UTC
Did you change the json format of destroyed and dropped items ? Would be nice to give us a notice about incoming format changes.
Karbowiak
Sacred Templars
Fraternity.
#104 - 2012-07-24 09:37:15 UTC  |  Edited by: Karbowiak
Desmont McCallock wrote:
Kill report manipulation is indeed a possibility but I think that the time it will need to code such a system or alter an existing uploader to send malicious data, is just not worth it.
Protective mechanism can be applied in those case like cross-checking the kill report between victim-attackers.

Indeed the greater issue here is the privacy of the info, which probably is the blocker in bringing this project to life.


Actually, i think they are more inclined to using a system like this, if they know they are the only ones who can see their apikeys.

As for protection (And this is just an idea!), what if a primary site (say eve-kill, or battleclinic if they are up for it) are used, and then each killlog upload is tied to a specific username+password hash. And if they in any way tamper with a killlog, they will have their entire account revoked. (Or maybe just ignore all their killogs, and do it silently so it'll take them time to figure out they're not getting anything posted, lol Lol)

Should deter would be fakers i guess.

Etil DeLaFuente wrote:
Did you change the json format of destroyed and dropped items ? Would be nice to give us a notice about incoming format changes.


i did, but i'm currently working on two ways to get raw mails, involved parties and items.
One which involves returning names and whatnot else is needed to create a mail similar to ingame.
And one which mimics the API killlog data (just in json, you can turn that into xml yourself if you want :P)

But yeah, sorry about not giving advance notice :\
Etil DeLaFuente
Aliastra
Gallente Federation
#105 - 2012-07-24 10:02:07 UTC
No problem, while you're at it, do you have any way to add the "Blueprint copy flag" aswell ?
Desmont McCallock
#106 - 2012-07-24 10:25:57 UTC  |  Edited by: Desmont McCallock
The EVEMon uploader is designed to be user unaware (and I like to keep it that way).
Because I'm unaware how the killboard system functions I can't comment on that but each kill report has involved parties (victim-attackers) which can be used to tie the report to them.

@Etil
AFAIK the BPC flag is defected in the API call and is not represented correctly in the Kill report.
Etil DeLaFuente
Aliastra
Gallente Federation
#107 - 2012-07-28 11:30:45 UTC  |  Edited by: Etil DeLaFuente
Recent changes completely broke your API.

Example :

Url : http://eve-kill.net/epic/combinedAllianceExtID:1900696668/mask:1606910/mailLimit:200/

That kill http://eve-kill.net/?a=kill_detail&kll_id=14146417

Is transfered as :

{
"timestamp": "2012-07-28 00:48:00",
"internalID": 14146417,
"externalID": 24965491,
"victimName": "Dionysus Caldoron",
"victimExternalID": 91272806,
"victimCorpName": "Open University of Celestial Hardship",
"victimAllianceName": "Art of War Alliance",
"victimShipID": 587,
"solarSystemName": "8G-MQV",
"involved": [{
"characterID": 1417371863,
"characterName": "Pandoralica",
"corporationID": 680022174,
"corporationName": "DEFCON.",
"allianceID": 1900696668,
"allianceName": "The Initiative.",
"factionID": 0,
"factionName": "",
"securityStatus": 2.2,
"damageDone": 1569,
"finalBlow": 1,
"weaponTypeID": 3178,
"shipTypeID": 11172
}],
"items": {
"destroyed": [{
"typeName": "125mm Gatling AutoCannon I",
"typeID": 484,
"itemSlot": "1",
"qtyDropped": 0,
"qtyDestroyed": 3
},
{
"typeName": "1MN Afterburner I",
"typeID": 439,
"itemSlot": "2",
"qtyDropped": 0,
"qtyDestroyed": 1
},
{
"typeName": "Inferno Rocket",
"typeID": 2514,
"itemSlot": "4",
"qtyDropped": 0,
"qtyDestroyed": 1
},
{
"typeName": "Overdrive Injector System I",
"typeID": 1244,
"itemSlot": "3",
"qtyDropped": 0,
"qtyDestroyed": 2
},
{
"typeName": "Phased Plasma S",
"typeID": 184,
"itemSlot": "1",
"qtyDropped": 0,
"qtyDestroyed": 2
},
{
"typeName": "Stasis Webifier I",
"typeID": 526,
"itemSlot": "4",
"qtyDropped": 0,
"qtyDestroyed": 1
}],
"dropped": [{
"typeName": "125mm Gatling AutoCannon I",
"typeID": 484,
"itemSlot": "1",
"qtyDropped": 1,
"qtyDestroyed": 0
},
{
"typeName": "EMP S",
"typeID": 185,
"itemSlot": "4",
"qtyDropped": 1,
"qtyDestroyed": 0
},
{
"typeName": "Mjolnir Rocket",
"typeID": 2512,
"itemSlot": "1",
"qtyDropped": 2,
"qtyDestroyed": 0
},
{
"typeName": "Nanofiber Internal Structure I",
"typeID": 2603,
"itemSlot": "3",
"qtyDropped": 1,
"qtyDestroyed": 0
},
{
"typeName": "Overdrive Injector System I",
"typeID": 1244,
"itemSlot": "3",
"qtyDropped": 1,
"qtyDestroyed": 0
},
{
"typeName": "Phased Plasma S",
"typeID": 184,
"itemSlot": "1",
"qtyDropped": 2,
"qtyDestroyed": 0
},
{
"typeName": "Rocket Launcher I",
"typeID": 10629,
"itemSlot": "1",
"qtyDropped": 1,
"qtyDestroyed": 0
},
{
"typeName": "Stasis Webifier I",
"typeID": 526,
"itemSlot": "2",
"qtyDropped": 1,
"qtyDestroyed": 0
},
{
"typeName": "Warp Scrambler I",
"typeID": 447,
"itemSlot": "2",
"qtyDropped": 2,
"qtyDestroyed": 0
}]
}
}

Quantities are wrong and your new attribute itemSlot is a string instead of an integer.

And why do you put qtyDropped and qtyDestroyed in both dropped and destroyed items ? you could save some place and only use one quantity attribute.

Also, it would be awesome to explain somewhere what these itemSlots numbers are refering to.
Etil DeLaFuente
Aliastra
Gallente Federation
#108 - 2012-07-28 11:35:44 UTC
You REALLY should consider doing some unit testing with that API
Veshai
NED-Clan
Goonswarm Federation
#109 - 2012-08-03 09:58:35 UTC  |  Edited by: Veshai
Is anything wrong with the API?

http://eve-kill.net/epic/mask:2097164/mailLimit:500/combinedPilot:NED-Clan

I'm getting no results at all, while there should be LOTS (and at least 17 this month alone).

[edit]nvm, needed coffee (combinedPilot vs combinedCorporation)[/edit]
Strucker
School of Applied Knowledge
Caldari State
#110 - 2012-08-03 11:31:32 UTC
Karbowiak wrote:

But due to the way the db is split, sometimes the next id is in a different table (say last mail is from 2012_07, next is in 2012_08) and it messes it up, which is annoying, and worse is that there is no clear way to get around it.
Someone smarter than me can probably figure it out, lol..

So let me get this straight, you think idfeed sucks you create another api which is faster but sucks as well?
It baffles me that you can make an api and not figure out how to get the kills from another table...
Did you even create the api?

Don't get me wrong, I like improvement but this is no improvement.
The only way to get this sorted out is to make 2 queries every time so you don't miss out on kills..
Fine by me but don't come crying that people are "hammering" your api then.
Messoroz
AQUILA INC
Verge of Collapse
#111 - 2012-08-07 02:54:46 UTC
If someone learned how to use memcached properly, there would have been no need for separate tables for kills.
Gnaw LF
Viziam
Amarr Empire
#112 - 2012-08-07 15:39:40 UTC
Karb, will you re-enable IDFeed for eve-kill or are you working on fixing this issue with EPIC API?
I DontLikeWhatYoureDoing
The Executioners
#113 - 2012-08-08 08:00:26 UTC
I'm trying to understand the KLLid command. You indicate that it can take either evekill-internal or ccp-external IDs. But, aren't these integer ranges overlapping? It also doesn't seem to be handling the external ID at the moment. I get exception 'Unable to retrieve the YEAR and MONTH from the database' when I pass in a ccp id.
Durzel
Questionable Ethics.
Ministry of Inappropriate Footwork
#114 - 2012-08-10 02:28:33 UTC
Is there any way of retrieving kills/losses for a corporation for a given system, when you don't know when (or if) they lost anything there?

Right now if you don't specify a startDate and endDate then it seems the current month is assumed, so the only way I can see of retrieving this date is to iteratively scrape the data changing the startDate and endDate by -1 month over and over until a non-zero result is returned.

Thanks in advance!
Veshai
NED-Clan
Goonswarm Federation
#115 - 2012-08-10 16:53:35 UTC
Durzel wrote:
Is there any way of retrieving kills/losses for a corporation for a given system, when you don't know when (or if) they lost anything there?

Right now if you don't specify a startDate and endDate then it seems the current month is assumed, so the only way I can see of retrieving this date is to iteratively scrape the data changing the startDate and endDate by -1 month over and over until a non-zero result is returned.

Thanks in advance!

As queries do not span months as indicated by the developer, the only option is to query month by month until you find a kill. I recommend you cache your results ;)
Durzel
Questionable Ethics.
Ministry of Inappropriate Footwork
#116 - 2012-08-15 08:35:08 UTC
Veshai wrote:
Durzel wrote:
Is there any way of retrieving kills/losses for a corporation for a given system, when you don't know when (or if) they lost anything there?

Right now if you don't specify a startDate and endDate then it seems the current month is assumed, so the only way I can see of retrieving this date is to iteratively scrape the data changing the startDate and endDate by -1 month over and over until a non-zero result is returned.

Thanks in advance!

As queries do not span months as indicated by the developer, the only option is to query month by month until you find a kill. I recommend you cache your results ;)

That's what I've been doing, was hoping for a out-of-the-box solution. Thanks Smile
Durzel
Questionable Ethics.
Ministry of Inappropriate Footwork
#117 - 2012-08-16 11:14:32 UTC
Just noticed an anomaly, want to make sure it's not me...

http://eve-kill.net/?a=system_detail&sys_name=Tintoh vs http://eve-kill.net/epic/system:Tintoh/mailLimit:1/mask:655359

The former shows the last kill being a Megathron at 21:47, but the last kill shown by the API is the one below it that occured at 21:23. Neither are API Verified so I don't think it's that?
Gnaw LF
Viziam
Amarr Empire
#118 - 2012-08-16 15:41:07 UTC
Durzel wrote:
Just noticed an anomaly, want to make sure it's not me...

http://eve-kill.net/?a=system_detail&sys_name=Tintoh vs http://eve-kill.net/epic/system:Tintoh/mailLimit:1/mask:655359

The former shows the last kill being a Megathron at 21:47, but the last kill shown by the API is the one below it that occured at 21:23. Neither are API Verified so I don't think it's that?



This EPIC API method is far from epic, in fact its far from functional. Because all the kills are spread around various tables its anyone guess what needs to be passed to eve-kill to actually fetch them. I tried fetching all kills from w-space from the beginning of August, only worked for 8 regions. Tried fetching kills for some notable w-space alliances, only certain ones are fetched. No telling what or where. Like walking a minefield.
Heinrich Hertz
Jerkasaurus Wrecks Inc.
Sedition.
#119 - 2012-08-17 07:12:38 UTC
Thanks for EPIC man, works great. Thought I'd share this simple tool (T-I-T-S) that uses EPIC to show eft fits from killmails by ship type. You can also see the c# source code for example using .Net 4 and serializer to read feed.

https://forums.eveonline.com/default.aspx?g=posts&t=140260&find=unread
Durzel
Questionable Ethics.
Ministry of Inappropriate Footwork
#120 - 2012-08-17 14:52:33 UTC
Found another bug, "orderBy" doesn't seem to work properly at all.

For example, I'd expect this query to get the most recent kill in August:
http://eve-kill.net/epic/system:J122818/mask:655359/mailLimit:1/startDate:2012-08-01/endDate:2012-08-31/orderBy:desc

..but in reality the latest kill it reports is 21:27:00, which isn't the latest.

Increasing the mailLimit fixes it, but it doesn't return it in descending order!

e.g.
http://eve-kill.net/epic/system:J122818/mask:655359/mailLimit:100/startDate:2012-08-01/endDate:2012-08-31/orderBy:desc

..returns mails starting at 21:27:00 as per the mailLimit 1 query, but also mails at 21:40, 21:41 after the 21:27 one.

/orderBy:asc doesn't seem to work either - the same query with /orderBy:asc returns 2012-08-04 19:40:00 as the first kill, but there are 19:38 kills after it, etc.

As a consequence even if I just want to get the latest kill in the system I have to query like 100 of them and re-order the output myself because mailLimit:1/orderBy:desc just doesn't work.