These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
123Next pageLast page
 

Reverence - 100% compatible EVE cache library for Python

First post
Author
Entity
X-Factor Industries
Synthetic Existence
#1 - 2011-09-06 14:16:39 UTC  |  Edited by: Entity
From old forum thread here


RΞVΞRENCE

This is a strictly x86/x64 Python 2.7 cache decoder library, targeted mainly at developers using Windows, although it has been tested successfully on Linux.

FEATURES

  • High-performance iterative cache/bulkdata decoder.
  • 100٪ compatibility with all bulkdata, cache and settings files.
  • Programmatic access to data tables.
  • Transparent loading of bulkdata on accessing tables.
  • Simultaneous handling of data from multiple EVE installations/versions.
  • Container classes for all data items found in cache and bulkdata.
  • Offline RemoteSvc calls, provided the relevant cachefiles exist. Note that this software DOES NOT interact with the EVE Online client or server.
  • EmbedFS (.stuff) file support.
  • Various EVE related utility functions and constants.

The library provides programmers with an interface very similar to what EVE uses itself under the hood (yes, CCP reviewed the code and actually didn't mind it), with some variations where I deemed necessary for usability, efficiency, etc, as EVE was never designed to operate on the dataset of multiple EVE installations at once.

It's licensed under the terms of the BSD license so you can do almost anything you want with it (if you're mad enough to port it to other languages, let me know!)


DOWNLOAD

The project is available on github here:
http://github.com/ntt/reverence

Source, 32- and 64bit Binary distributions for Python 2.7 here:
Downloads


DEPENDENCIES

Reverence requires the PyYAML package.
If you're looking for a precompiled 64bit version of this, you can get it here


USAGE

Some examples have been provided (see link above), and some instructions are in the README.txt in the distribution.
You'll have to excuse me for the lack of depth of the documentation, but this has been a fairly rapidly changing project over its lifetime (it 'settled' only recently). I'll probably write more thorough documentation Soon™.

I hope you'll enjoy this thing as much as I did creating it.


THANKS
Special thanks go to CCP for granting permission to release this product, even though it is heavily inspired by EVE Online's design.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Pantload
The Underpants Gnomes
#2 - 2011-09-11 17:54:38 UTC
Ok so...stepping through asset list. Each asset has a flag that shows its exact location type. That flag relates to an invFlags or config.Flags table. Is reverance blue.config not capable of using that table? I don't see it in table list anywhere and it bitches if I try to use it. Any help would be greatly appreciated. Thanks.

Cheers,
-PL
Entity
X-Factor Industries
Synthetic Existence
#3 - 2011-09-11 20:59:19 UTC
Pantload wrote:
Ok so...stepping through asset list. Each asset has a flag that shows its exact location type. That flag relates to an invFlags or config.Flags table. Is reverance blue.config not capable of using that table? I don't see it in table list anywhere and it bitches if I try to use it. Any help would be greatly appreciated. Thanks.

Cheers,
-PL


MMMmmh.. it'd help if you gave more specific info :P
There's location flags in const.py, line 44 to 106, if you mean those.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Pantload
The Underpants Gnomes
#4 - 2011-09-18 19:29:38 UTC  |  Edited by: Pantload
Hey thanks for quick response. Sorry for long response here.

I was not aware of those in the const.py. Those could be useful to me later.

You're right. I should be much more specific. Here is a snippet of code to illustrate what I'm after:


import os
import time
import cPickle as pickle
import eveapi
from reverence import blue

USER_ID = 0
API_KEY = ""
CHAR_NAME = ""
EVE_PATH = ""

class APICacher(object):
def __init__(self):
self.file_ext = ".api_result"
self.file_path = "./API_Cache/"
if not os.path.exists(self.file_path):
os.makedirs(self.file_path)

def retrieve(self, host, path, params):
hash_string = str(hash(str(host) + str(path) + str(params)))
file_name = self.file_path + hash_string + self.file_ext
cache_file= None
try:
cache_file= open(file_name, 'rb')
except IOError:
return None
else:
cache_object = pickle.load(cache_file)
cache_file.close()
if time.time() < cache_object.cachedUntil:
return cache_object
else:
return None

def store(self, host, path, params, doc, obj):
hash_string = str(hash(str(host) + str(path) + str(params)))
file_name = self.file_path + hash_string + self.file_ext
cache_file = open(file_name, 'wb')
pickle.dump(obj, cache_file)
cache_file.close()


eve = blue.EVE(EVE_PATH)
invtypes = eve.cfg.invtypes
api = eveapi.EVEAPIConnection(cacheHandler=APICacher())
auth = api.auth(userID=USER_ID, apiKey=API_KEY)
chars = auth.account.Characters().characters
char_id = None
for char in chars:
if char.name == CHAR_NAME:
char_id = char.characterID
corp_auth = auth.corporation(char_id)

for i in corp_auth.AssetList().assets:
print invtypes.get(i.typeID).typeName, "| flag= ", i.flag
if hasattr(i,"contents"):
for j in i.contents:
print " "*4, invtypes.get(j.typeID).typeName, "| flag= ", j.flag
if hasattr(j,"contents"):
for k in j.contents:
print " "*8, invtypes.get(k.typeID).typeName



The flag field gives me a number here that relates to this table: link here

How do I look up the flag number in this table and get either flagName or flagText?


I'm going to apologize in advance if this is stupid or obvious but humor me anyway, if you would, please.

Thanks for any help!


Cheers,
-PL


*edit*
unfortunately, the forum destroys the indentations in the code. Oddly, if I go to Edit it...all the indentations are shown.
Pantload
The Underpants Gnomes
#5 - 2011-09-18 19:56:41 UTC  |  Edited by: Pantload
*edit*

mistaken reply. disregard
Entity
X-Factor Industries
Synthetic Existence
#6 - 2011-09-18 23:27:32 UTC
Ah, not entirely sure that invFlags table exists in the client. A quick glance over the code suggests it's not there.

I would suggest hardcoding that table as a dict.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Netheranthem
Viziam
Amarr Empire
#7 - 2011-09-28 12:52:30 UTC
So, I never used this before, but I found a script that should've worked to get CSV data about all items cached

http://webcache.googleusercontent.com/search?q=cache:http://pastie.org/821014

But it doesn't work, any pointers? (When I run it, it does nothing, actually. Not even a word).
Elmore Jones
New Eden Mining Organisation
The Craftsmen
#8 - 2011-09-29 11:23:30 UTC  |  Edited by: Elmore Jones
The script you have there is one to read the price history, not the current prices. I assume thats what you want? There is also :

key[1]=="GetNewPriceHistory"

GetNewPriceHistory(regionID, typeID)

Returns: CRowset (historyDate, lowPrice, highPrice, avgPrice, volume, orders)

This returns only one list entry, not multiples as you would expect for market histories.

historyDate apparently is the number of 100-nanosecond intervals since January 1st, 1600, which a common Microsoft way of storing time values. To represent it in the more familar "seconds since 1-1-1970" 32-bit timestamp:

secondsSinceUnixEpoch = (historyDate - 116444736000000000) / 10000000

(info from http://wiki.eve-id.net/Cache_Resources#GetNewPriceHistory.28regionID.2C_typeID.29)

for current prices you need the GetOrders key.

Something in my current script to read current prices from cache is broken since the changes to the bulkdata so thats all I can suggest till I find out what that issue is :(

Addendum : and I have it - there are / and " characters in the apparel names that are killing my script. Nerfed by space barbie o\

+++ Reality Error 404 - Reboot Cosmos +++

Elmore Jones
New Eden Mining Organisation
The Craftsmen
#9 - 2011-09-30 07:02:30 UTC
Netheranthem wrote:
So, I never used this before, but I found a script that should've worked to get CSV data about all items cached

http://webcache.googleusercontent.com/search?q=cache:http://pastie.org/821014

But it doesn't work, any pointers? (When I run it, it does nothing, actually. Not even a word).


had a little play with this this morning as I've never requested history from cache before...

EVEROOT = r"C:\Games\CCP\EVE"
OUTPATH = r"c:\evetemp"

Make sure your first 2 lines look like this (with your paths of course). Any output directory has to be manully made as the script stands. With just this change the script worked for me but oddly only gave results for 1 item (dairy prodcts in this case). My cache was cleaned yesterday but I know theres more than that in there for deliberatley checked prices.

The following was added inside main for loop to catch any apparel names with illegal characters but made no change

# incarna note : need to scan filename for '/' and '"' character as used in apparel
item.name = item.name.replace('/', ' ')
item.name = item.name.replace('"', '`')

+++ Reality Error 404 - Reboot Cosmos +++

Entity
X-Factor Industries
Synthetic Existence
#10 - 2011-12-05 21:46:11 UTC
Updated version 1.4.0 for Crucible now available (see OP).

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

roigon
TURN LEFT
#11 - 2011-12-21 09:58:11 UTC  |  Edited by: roigon
I'm building an application that uses reverence to passively process marketdata. I'm running into an issue though, after a while revenance opens too many files and it throws an exception. (see below)

Is there some cleanup method somewhere I can call for reverence to cleanup it's open files?

Quote:
Exception in thread Thread-16:
Traceback (most recent call last):
File "C:\Python27_32b\lib\threading.py", line 552, in __bootstrap_inner
File "C:\Users\erik\git\marketTool\marketTool\src\eve\marketCacheImporter.py", line 42, in run
File "C:\Users\erik\git\marketTool\marketTool\src\eve\eveLib.py", line 18, in __init__
File "C:\Python27_32b\lib\site-packages\reverence\blue.py", line 126, in __init__
File "C:\Python27_32b\lib\site-packages\reverence\blue.py", line 79, in __init__
File "C:\Python27_32b\lib\site-packages\reverence\embedfs.py", line 86, in __init__
File "C:\Python27_32b\lib\site-packages\reverence\embedfs.py", line 32, in __init__
IOError: [Errno 24] Too many open files: 'C:\\Program Files (x86)\\CCP\\EVE2\\resCharacterTattoos.stuff'



-edit-

Seem to have fixed the problem or at least made it less likely to happen by explicitly calling the garbage collector after each scan.
Entity
X-Factor Industries
Synthetic Existence
#12 - 2011-12-21 17:43:40 UTC  |  Edited by: Entity
roigon wrote:
IOError: [Errno 24] Too many open files: 'C:\\Program Files (x86)\\CCP\\EVE2\\resCharacterTattoos.stuff'

-edit-

Seem to have fixed the problem or at least made it less likely to happen by explicitly calling the garbage collector after each scan.


Looping over a large amount of files without calling close() after each tends to cause this issue.
The garbage collector simply kicks in too late for it to stop you running out of filehandles that way. Just explicitly close a file before moving on to the next one and you will have no issue.

Also, avoid using LoadCacheFolder() for this sort of thing (if you're using it), I kinda regret putting it in the published version.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

roigon
TURN LEFT
#13 - 2011-12-21 19:32:36 UTC
Entity wrote:
roigon wrote:
IOError: [Errno 24] Too many open files: 'C:\\Program Files (x86)\\CCP\\EVE2\\resCharacterTattoos.stuff'

-edit-

Seem to have fixed the problem or at least made it less likely to happen by explicitly calling the garbage collector after each scan.


Looping over a large amount of files without calling close() after each tends to cause this issue.
The garbage collector simply kicks in too late for it to stop you running out of filehandles that way. Just explicitly close a file before moving on to the next one and you will have no issue.

Also, avoid using LoadCacheFolder() for this sort of thing (if you're using it), I kinda regret putting it in the published version.


That's exactly what I used :)

I kinda liked that reverance did all the I/O for me so I could just treat it like a magic black box that spitted out good stuff, but I guess I'll have to get my hands dirty and actually tell it which cache files to look at and make sure they get processed in smaller batches so i can properly close the files.

Many thanks for creating it though, other then this issue it's been very easy to work with. I wish some of the code had some more comments* but that's also my lack of python knowledge at work.

* Yes I know the answer to this is "patches welcome" :P
Entity
X-Factor Industries
Synthetic Existence
#14 - 2011-12-21 21:40:52 UTC
roigon wrote:
I wish some of the code had some more comments* but that's also my lack of python knowledge at work.

* Yes I know the answer to this is "patches welcome" :P


Documentation is possibly not my strong point ;) Though, the deeper stuff (like cacheMgr) is there mostly for use in reverence itself anyway.
I'm fixing the filehandle issue though. Will post an update sometime Soon™

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Vaerah Vahrokha
Vahrokh Consulting
#15 - 2011-12-22 10:07:10 UTC
Works like a charm with latest patch.
malaire
#16 - 2011-12-22 17:43:12 UTC  |  Edited by: malaire
Today I started to get following error from blue.marshal.Load(open(filename,"rb").read()) call (full code).

Error is: "Attempted store of NULL at shared position #3 - type:0x2 ctype:0x14 len:23 share:64 pos:212 size:13151"

EDIT: I'm using reverence-1.4.1.win32-py2.6. Error was coming from one cache file and when I deleted it I was able to continue using my script again.

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else

Entity
X-Factor Industries
Synthetic Existence
#17 - 2011-12-22 18:59:58 UTC
malaire wrote:
Today I started to get following error from blue.marshal.Load(open(filename,"rb").read()) call (full code).

Error is: "Attempted store of NULL at shared position #3 - type:0x2 ctype:0x14 len:23 share:64 pos:212 size:13151"

EDIT: I'm using reverence-1.4.1.win32-py2.6. Error was coming from one cache file and when I deleted it I was able to continue using my script again.


I really do need the file to debug stuff like that :)

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

malaire
#18 - 2011-12-23 10:29:54 UTC
Entity wrote:
I really do need the file to debug stuff like that :)

Yep... I was so eager to get it working that I just deleted the file. If I get same error again I'll save the file.

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else

Magnomus
Perpetual Motion Machine
#19 - 2011-12-24 08:57:58 UTC  |  Edited by: Magnomus
I've managed to get Reverence to create workable tables to analyze market history data(yay!), and after hours of struggle finally got it to analyze real time market data in "cachedmethodcall" using this code.

However, I've not managed to successfully adapt the XML script that functions with other methodcalls such as those in master/examples/datadump.py to one that would simply fetch GetOrders. Can anyone advise me on creating a version of the top link that outputs in XML format?

I apologize that my python experience is quite limited (to what I've managed to get working thus far).
malaire
#20 - 2011-12-24 10:16:28 UTC
Magnomus wrote:
I've managed to get Reverence to create workable tables to analyze market history data(yay!), and after hours of struggle finally got it to analyze real time market data in "cachedmethodcall" using this code.

However, I've not managed to successfully adapt the XML script that functions with other methodcalls such as those in master/examples/datadump.py to one that would simply fetch GetOrders. Can anyone advise me on creating a version of the top link that outputs in XML format?

I apologize that my python experience is quite limited (to what I've managed to get working thus far).

Since simple XML is basicly same as CSV except is uses a lot of crap instead of simple commas, this might work: http://pastie.org/private/7ssdk0mde7gdsfqeqkwjw

(and no, I don't know how to split that big line into smaller parts, might need to rewrite section following it to do that.)

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else

123Next pageLast page