These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

Kill Grabber (Downloader for Kill Mails)
Author
Previous Topic Next Topic
Nexuscrawler
Evil Moustache
#1 - 2012-01-17 16:27:23 UTC  |  Edited by: Nexuscrawler
Hello Community,

i programmed a tool that can save raw Kill-Mails from any EDK Killboard page to the local PC and still offers sufficient filtering capabilities.
Whether this program is helpful or not, you have to decide on your own....but please report any errors you discover.

To the download: http://killboards.de/download.php?f=kill_grabber.zip

Original Posts at eveger.de: https://www.eveger.de/forum/showthread.php?t=43244 (German)


If you encounter any problems, please do not just skip the error message, but help me to avoid these mistakes in future and give feedback to the program!

The clan's are marching `gainst the law, Bagpipers play the tunes of war, Death or glory i will find, Rebellion on my mind.
Shellac Brookdale
Cutting Edge Incorporated
#2 - 2012-01-17 20:45:50 UTC
Clever way to ddos other killboards so people need to signup with killboards.de
Nexuscrawler
Evil Moustache
#3 - 2012-01-18 10:34:51 UTC  |  Edited by: Nexuscrawler
Shellac Brookdale wrote:
Clever way to ddos other killboards so people need to signup with killboards.de


You did not consider, that "killboards.de" could also be DDOSed with that, did you?

Listen....I am not responsible for your (ab)usement of this program.
It was just thought as a simple way to get your kill-mails from your board, i dont want to make anyone uncompetitive.

Like i wrote, I am hosting a GERMAN Killboard-Service, so it would not make any sense, to release that program in the ENGLISH Forum, if my only goal was to massacre other hosters to get their customers, would it?

But however, the program checks for blocked sides, before it allows access to the kill-mails of a board.
So if any hoster will get massively DDOSed, i'll just add him to that block list and further request to his board trough this program will fail.

The clan's are marching `gainst the law, Bagpipers play the tunes of war, Death or glory i will find, Rebellion on my mind.
Karbowiak
Sacred Templars
Fraternity.
#4 - 2012-01-18 15:46:26 UTC
Dude..

http://files.karbowiak.dk/462z0D3C061T2H280t29

what the ****.. Please for the ******* love of god take this **** down before you ruin something.
Also, learn to use feed / idfeed or atleast code an eve-kill specific part that allows for the usage of epic..

https://forums.eveonline.com/default.aspx?g=posts&t=18842 here if you need information on how to use epic..
Briess Maulerant
Federal Navy Academy
Gallente Federation
#5 - 2012-01-18 15:50:34 UTC
Scraping a webpage to get killmails is not the right way to do this >.>
Peter Powers
Terrorists of Dimensions
#6 - 2012-01-18 15:54:22 UTC  |  Edited by: Peter Powers
as someone who runs a board let me point out a few things:

a) edk boards offer an idfeed which allows people to fetch mails through its api (and the old feed aswell)
b) scrapping websites is usually, no matter where, something that gets people angry
c) edk wastes resources, lots of, meaning such scrapping puts *alot* on the servers.
d) i've seen people scrap it, or excessively requesting the feeds before (like every few seconds) - and it allways resulted in the same action

most larger boards will ban you from the board when running such tool. Either automatic or manually when the load on the server explodes and the admins are notified - don't do it.

and such ban will not hit you as the author directly, and most admins (including me) wont bother talking to you about your ban list on your side, the user of your tool will get banned.

3rdPartyEve.net - your catalogue for 3rd party applications
Nexuscrawler
Evil Moustache
#7 - 2012-01-18 16:01:25 UTC
Well....maybe i should invert this and use a "White List" instead of a "Block List", which lets the tool only work for sides, where it is wanted.

The clan's are marching `gainst the law, Bagpipers play the tunes of war, Death or glory i will find, Rebellion on my mind.
Karbowiak
Sacred Templars
Fraternity.
#8 - 2012-01-18 16:03:20 UTC  |  Edited by: Karbowiak
Nexuscrawler wrote:
Well....maybe i should invert this and use a "White List" instead of a "Block List", which lets the tool only work for sides, where it is wanted.


You should rather learn to use feed or idfeed m8..

Cause this is just stupid Ugh

edit://
also noone will want to be scraped by this, noone...
Mangala Solaris
Blue Republic
RvB - BLUE Republic
#9 - 2012-01-18 16:20:35 UTC
Karbowiak wrote:
Nexuscrawler wrote:
Well....maybe i should invert this and use a "White List" instead of a "Block List", which lets the tool only work for sides, where it is wanted.


You should rather learn to use feed or idfeed m8..

Cause this is just stupid Ugh

edit://
also noone will want to be scraped by this, noone...


+1ing Karbo here.

Use IDFEED for this, its so much simpler.
Squizz Caphinator
The Wormhole Police
#10 - 2012-01-18 16:44:44 UTC
eve-kill has now implemented an anti-scraping tool because of this stupid little program. Use idfeed

Quote:
You are exceeding the maximum threshold for requests per hour. Fix your **** or die in a fire, thank you and have a blessed day.

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Nexuscrawler
Evil Moustache
#11 - 2012-01-18 16:54:13 UTC
That implementation was added before this tool was released, because someone hammered the idfeed with 20 requests per second. (not me) -.-

The clan's are marching `gainst the law, Bagpipers play the tunes of war, Death or glory i will find, Rebellion on my mind.
Peter Powers
Terrorists of Dimensions
#12 - 2012-01-18 16:58:12 UTC
Nexuscrawler wrote:
That implementation was added before this tool was released, because someone hammered the idfeed with 20 requests per second. (not me) -.-

prevention of hammering idfeed is not the same as an anti-scrapping tool.
and trying to tell the guy who put the anti scraping tool in - after he announced it - when he did it.. wtf?

3rdPartyEve.net - your catalogue for 3rd party applications
Karbowiak
Sacred Templars
Fraternity.
#13 - 2012-01-18 16:59:31 UTC
Nexuscrawler wrote:
That implementation was added before this tool was released, because someone hammered the idfeed with 20 requests per second. (not me) -.-


So hitting upwards of +100 pages just to get a bunch of killids for a pilot/corp/alliance, so you can then get x mails for that person, is way better than using idfeed ?..

idfeed is way less resource intensive than the way you are doing it right now.
The way you are doing it, is by itself very ineffecient, but not exactly a ddos.

But imagine if 50 people do it at the same time to a board that can barely run smooth with 10 people..

Now imagine +100 doing it to EVE-KILL, constantly requesting new stuff that isn't cached anywhere..

scary right ?...
Nexuscrawler
Evil Moustache
#14 - 2012-01-18 17:23:23 UTC  |  Edited by: Nexuscrawler
The program was updated, downloads from eve-kill or any other side are not possible any more yet....so because of that idfeed.....
I dont have problems with reading them or at working with xml files, but i just dont get a useful response from most boards, therefor this solution was done.

For example, if i try to get the kills of: http://[sub].killboards.de/?a=idfeed&pilotname=*****&lastintID=***** this will work fine and the feed will begin at the ID i entered.

But if i try the same at http://eve-kill.net/?a=idfeed&pilotname=*****&lastintID=***** it will ignore the field "lastintID" and always responds with the first found internal ID of the pilot.

Is there any reason for this?

The clan's are marching `gainst the law, Bagpipers play the tunes of war, Death or glory i will find, Rebellion on my mind.
Karbowiak
Sacred Templars
Fraternity.
#15 - 2012-01-18 19:15:49 UTC
Nexuscrawler wrote:
The program was updated, downloads from eve-kill or any other side are not possible any more yet....so because of that idfeed.....
I dont have problems with reading them or at working with xml files, but i just dont get a useful response from most boards, therefor this solution was done.

For example, if i try to get the kills of: http://[sub].killboards.de/?a=idfeed&pilotname=*****&lastintID=***** this will work fine and the feed will begin at the ID i entered.

But if i try the same at http://eve-kill.net/?a=idfeed&pilotname=*****&lastintID=***** it will ignore the field "lastintID" and always responds with the first found internal ID of the pilot.

Is there any reason for this?


IDFeed on EVE-KILL is a bit weird, always has been, but it shouldn't ignore &lastintID, i'll investigate.
But instead of idfeed on EVE-KILL, i'd say use epic..
Karbowiak
Sacred Templars
Fraternity.
#16 - 2012-01-18 22:23:20 UTC
http://pastebin.com/XeCX5z0h

http://pastebin.com/x0rvGNU7

So much code, to do something so simple as getting the mail from ?a=kill_mail..
Nexuscrawler
Evil Moustache
#17 - 2012-01-19 07:35:58 UTC  |  Edited by: Nexuscrawler
This Code does not just contain a reader for the web response of the Kill-Mail.

There are also different approaches for different structures, such as eve-kill with the &month and &year, the usement of normal or simple links, differences between private Board-Structures and preparedness of any possible error that "could" happen....and the walk trough of the responded source codes with all possible structures.
I admit, that the code is totaly overblown, but it does, what it should do so far, even if that was resolved a little bit stupid.
But the idfeeds didnt work first, so i prefered the slow and working way, not the fast an bugged.

The reason, why i didnt use epic, is, that on the one hand i didnt know it and on the other hand i dont wanted to depend my own software on other third party software, to get it running.
I hope my reasons are understandable so far, but if you might would help me to get a working possibility with the idfeed-links, I would of course change the code.
I will also look at "epic", if that is the only way, to fetch kills from eve-kill....but it was never intended, to get the kill mails mainly from there, only to work as a simple backup solution for everyone, not just the admin, that can use the export function.
Programming a separately function for eve-kill will blow up the code again, so i would prefer an working idfeed.

The clan's are marching `gainst the law, Bagpipers play the tunes of war, Death or glory i will find, Rebellion on my mind.
Nexuscrawler
Evil Moustache
#18 - 2012-01-25 19:05:11 UTC  |  Edited by: Nexuscrawler
Sooooo....

After reprograming the Kill Grabber, it will now use ID-Feed instead of parsing the HTML-Code of each Kill Mail.

The Kill Grabber will now create a Cache File for each Server and for each Pilot/Corporation/Alliance that was requested before from that server.
As a result, the program will not request a feed twice....it will only download new Kill Mails and create the other ones from Cache.

For converting the IDs of the feeds content to their names, I first tried to use local XML Files....but they were far to slow, because checking the ID of every ship, item and system in every Mail by parsing the XML files took some time.
Then i tried to use a local, portable SQL-Database, which will start itself with the Kill Grabber....that worked, but it has overblown the download size from about 200 KB to 28 MB.
So i skipped both....the Kill Grabber will now use the MySQL Database of my own Killboard-Server to convert the IDs.


Please feel free to download and test the Kill Grabber again.
The download link in the start post will now lead to the new version.
I also shortened the start post a little bit.



For the coders of you, here's a simple overview about what's behind the source:

Quote:
# Step 1
# Updating the cache file by requesting new ID-Feeds

while (true)
{
Checking last internal ID in Cache File (if existing)

if (last internal ID from Cache File not null)
{
Download new feed starting with this ID (and optional additional filters)

if (last internal ID in downloaded feed = last internal ID in Cache File OR the downloaded feed doesnt contain Kill Mails)
{
Stop the download process of new feeds (we are at the end)
break the "while"
}
}
else
{
Download new feed (with optional additional filters)
}

Store the Kills from the IDFeed in the Cache File (if Kills doesnt already exist there)
}

# Step2
# When the "while" is broken, the Cache File should contain all new Kill Mails
# We will now create the kill mails from the Cache File

foreach (Kill Mail in Cache File)
{
if (the Kill Mail passed the filters)
{
ID to Name Conversion with the MySQL Database
Create and save Kill Mails
}
}

# finished

The clan's are marching `gainst the law, Bagpipers play the tunes of war, Death or glory i will find, Rebellion on my mind.
Nexuscrawler
Evil Moustache
#19 - 2012-01-28 15:32:47 UTC
The Kill Grabber will now use a local SQL database instead of my own MySQL server, so there is no more need for a permanent connection.
Due to the local storage, you can also create raw kill mails from your cache file now, when you are offline.

The Program is now 20 times faster and will use lesser space for the cache.
The automatic update function has been removed due to problems with the manifest, so please download the new version by clicking the link.


More features are planned...like statistics.

How much ISK did I, my corporation or my alliance destroy in my/their EVE life(s)?
How much ISK did I lost?
What ship types exactly and how much of them did I kill?
Who is my favourite victim?
What is my favourite ship?
What is my favourite weapon?
Who is dominating me? ;-)

Much to do....I am working on it.


Please feel free to test the Kill Grabber again, the speed should be at the top now.

The clan's are marching `gainst the law, Bagpipers play the tunes of war, Death or glory i will find, Rebellion on my mind.
Tiger's Spirit
Templars of the Shadows
#20 - 2012-02-04 08:29:21 UTC
Nexuscrawler wrote:
Hello Community,

i programmed a tool that can save raw Kill-Mails from any EDK Killboard page to the local PC and still offers sufficient filtering capabilities.
Whether this program is helpful or not, you have to decide on your own....but please report any errors you discover.

To the download: http://killboards.de/downloads/kill_grabber.zip

Original Posts at eveger.de: https://www.eveger.de/forum/showthread.php?t=43244 (German)


If you encounter any problems, please do not just skip the error message, but help me to avoid these mistakes in future and give feedback to the program!



Dont work. Program freezing when KillGrabber want to write to disk the raw mails. (Win7 64bit)

Tried with compatibility mode too, but still freezing.

Problem Event Name: CLR20r3
Problem Signature 01: killgrabber.exe
Problem Signature 02: 1.0.0.0
Problem Signature 03: 4f282f95
Problem Signature 04: mscorlib
Problem Signature 05: 4.0.0.0
Problem Signature 06: 4e181ae3
Problem Signature 07: e45
Problem Signature 08: 96
Problem Signature 09: System.FormatException
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 1038
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
12Next page