These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVE and Google Drive API - Python scripting your way to cloud-based data analysis
Author
Previous Topic Next Topic
Vipre Morte
Team JK
#1 - 2013-06-21 20:04:48 UTC

  1. Introduction

    2. With all the wonderful data provided through Eve about skill points and markets, it surprised me to realize that there were no reports you could generate about even simple things, such as how much your wallet has grown over time. Luckily, we have Python. In this post I will show you how to grab the data you want with it through the Eve API (which has been covered well before) and send that data to your Google Drive cloud storage (which hasn't been covered so well) using Google's Drive API. Finally, I will show you how to configure your Windows or Linux box to run the script as often as you want.

  2. Install Python

    3. I used Python 2.7.3 for this, though any reasonable version will do. Download and install from www.python.org or in any Debian-based Linux distro:
    Quote:
    sudo apt-get install python2.7
    Test that it works with
    Quote:
    python --version
    Note to website maintainers: the bbCode "[code]" brackets would be a nice addition to this forum.

  3. Get Your Python Modules for EVE and Google

    • EVEAPI
      • Get the latest files from https://github.com/ntt/eveapi. From a Linux distro, I made a folder like this: “mkdir /opt/eve” then changed to its directory (“cd /opt/eve/”) and cloned the API files with “sudo git clone https://github.com/ntt/eveapi.git” - You can then run the setup.py script after changing into its the cloned directory.
    • Google Drive API
      • Get the latest Google Drive Python module, unpack it, and run its install script, all in this same directory the EveAPI files are (for convenience). In Linux, follow these commands:
      Quote:
      sudo wget https://gdata-python-client.googlecode.com/files/gdata-2.0.17.tar.gz
      sudo tar -xvf gdata-2.0.17.tar.gz
      sudo chmod a+rw gdata-2.0.17/
      sudo -i
      cd /opt/eve/eveapi/gdata-2.0.17/
      ./setup.py install

      Windows users must follow the documentation included with the download to get this up and running.


  4. Generate EVE API Key
    5. For security, EVE utilizes an API key structure where you generate the key for your application to authenticate with. Log in to https://community.eveonline.com and from the Support drop-down menu, select “API Key Management.” Hit the “Create a new key” button and call it something significant, such as “Python API.” Keep the defaults, and select if you want one or all characters from your account. Select the pieces of private info you’ll want to access. I selected “All” on everything because I’m a crazy son of a beach. When you’re all done, you’ll want to take note of the Key ID and Verification Code. You’ll need them later.

  5. Generate Your Application-Specific Google Password
    6. This only applies if you have two-factor authentication set up on your Google account. If you don’t, you may skip this step. Application-specific Google passwords are hashes that bypass the need to authenticate with a second-factor token. I couldn’t explain it any better than Google’s official support site, so here you go: https://support.google.com/accounts/answer/185833?hl=en

  6. Write The Code
    7. Finally! Coding! I stuck my script in /opt/eve/eveapi/ and called it wallet.py, since for this example I will be simply tracking my wallet balance over time. You’ll soon see that there are more useful ways you can use this API.

    • Input: EVEAPI Data
      • In the code below, simply trade out your EVE API Key ID and Verification Code for the placeholder texts in KEYID and VCODE at the top. After all the imports are done, you'll see the variable "api" creates the connection to EVE's servers, "auth" performs the authentication by passing on KEYID and VCODE, you can create a list of your characters by assigning auth.account.Characters(), and I end up cycling through that list to grab their balances:
      Quote:
      #!/usr/bin/python

      KEYID = #######
      VCODE = "BlahBlahBlah01234567890BlahBlahBlah"

      import time, tempfile, cPickle, zlib, os, locale, datetime
      import os.path, gdata.data, gdata.acl.data, gdata.docs.client, gdata.docs.data, gdata.sample_util
      from gdata.spreadsheet.service import SpreadsheetsService
      from os.path import join, exists
      from httplib import HTTPException
      locale.setlocale(locale.LC_ALL, '')

      import eveapi
      api = eveapi.EVEAPIConnection()
      auth = api.auth(keyID=KEYID, vCode=VCODE)

      CharactersResult = auth.account.Characters()
      TotalISK=0
      for character in CharactersResult.characters:
      char_wallet = auth.char.AccountBalance(characterID=character.characterID)
      isk = char_wallet.accounts[0].balance
      TotalISK+=isk

      wallet = {'timestamp': str(datetime.datetime.now())[:19], 'amount': str(TotalISK)}


    • Output: To Google Drive
      • Here you'll need to fill in your correct email, password, and call the source anything. Note that the q['title'] portion picks your spreadsheet out of your documents list, so fill that in accordingly too. If you generated an application-specific password in step 5, use that in place of your normal Google password.
      Quote:
      # Insert row into spreadsheet on Google Drive:
      spr_client = SpreadsheetsService()
      spr_client.email = 'YourEmail@gmail.com'
      spr_client.password = 'YourPassword'
      spr_client.source = 'Test GData App'
      spr_client.ProgrammaticLogin()

      q = gdata.spreadsheet.service.DocumentQuery()
      q['title'] = 'EVE_Wallet'
      q['title-exact'] = 'true'
      feed = spr_client.GetSpreadsheetsFeed(query=q)
      spreadsheet_id = feed.entry[0].id.text.rsplit('/',1)[1]
      feed = spr_client.GetWorksheetsFeed(spreadsheet_id)
      worksheet_id = feed.entry[0].id.text.rsplit('/',1)[1]

      spr_client.InsertRow(wallet,spreadsheet_id,worksheet_id)


    Test it out! Make sure your Google document fills in a row with data before your very eyes!


[Continues in next post]
Vipre Morte
Team JK
#2 - 2013-06-21 20:05:03 UTC  |  Edited by: Vipre Morte
[Continued from previous post]

7. Schedule Job To Run
If you’ve got a computer that’s always on, or at least is on during the times you’d like your data logged from EVE into Google Drive, you can schedule it to run your script. Note that this computer doesn’t store anything besides the API data and your script, so it actually doesn’t matter what machine you run it on. For example, I did all of this on my Raspberry Pi which acts as my home Web/FTP/SSH/IRC server, so it’s always on. Maybe you keep your Windows gaming PC on all the time? You could use it as well.


  • Task Scheduler in Windows
    • Finding the Task Scheduler in Windows beyond XP is easy: Type “Task Scheduler” into the Start menu search. On XP it’s under Start → All Programs → Accessories → System Tools. Simply follow the Create Task wizard, or Add a Scheduled Task to set the time and frequency for your script to run, and include the target of your script.

  • cron in *nix
    • Look up a tutorial on cron if you’re confused or run into trouble (there’s PLENTY out there!), but the gist of it is running the “crontab -e” command to edit your cron table, and at the very bottom at the line “[m] [h] * * * [target]” where m=minutes, h=hours, and target is your full path to the script, such as “/opt/eve/eveapi/wallet.py.” I wanted my script to run every noon and midnight, so my “m” is 0, and my “h” is “0,12” - you can also get fancy with division in this field, but that’s beyond the scope of this tutorial.



That’s it! Now you’ll probably want to get more info than just your wallet balance, so I implore you to check out the “eveapi.py” and “apitest.py” files that came with your EveAPI download. It includes examples of pulling data from Skill Training, calculating taxes paid in the past x-number of days, etc. According to http://wiki.eveonline.com/en/wiki/Eveapi there is no official API documentation on the Python implementation of this, they simply say to check out those files, so feel free to do so!

And if you haven't used Google Drive much, creating charts are a simple one-click operation:
http://s12.photobucket.com/user/Suebeebaby300/media/eveisk_zps0070aff4.png.html

You can see that my wallet is in need of some friendly in-game donations, so if this tutorial helps you at all, I welcome them!
Elim Solo
Science and Trade Institute
Caldari State
#3 - 2013-07-02 14:46:18 UTC
Cool. Nice tutorial. Even if I used it I don't have much spare ISK to donate P