These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Reverence - 100% compatible EVE cache library for Python

First post
Author
Vaerah Vahrokha
Vahrokh Consulting
#141 - 2013-09-30 21:22:36 UTC
Ok I have finally fixed it.
The compilation still plops the dependencies errors but slamming the PYD files in its teeth in the right places it works at runtime.

What I have noticed is that unlike the other libraries, is that Reverence's PYDs get composite names in the final compiled folders.

Whereas the other packages tend to have _Name.pyd, Reverence PYDs get a "reverence" prefix:

_blue.pyd becomes reverence._blue.pyd
_pyFSD.pyd becomes reverence._pyFSD.pyd

This seems to utterly confuse the compilers.
Entity
X-Factor Industries
Synthetic Existence
#142 - 2013-09-30 21:34:19 UTC
Vaerah Vahrokha wrote:
This seems to utterly confuse the compilers.


Curious. gj on finding a solution though.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Eeki Tsaibit
Caldari Provisions
Caldari State
#143 - 2013-10-17 01:32:58 UTC
I hope this is not too trivial. After updating to the latest version of reverence, this code:

print "Setting up EVE resources..."

eve = blue.EVE(EVEROOT)
cfg = eve.getconfigmgr()

print "Loading map data..."

f = eve.ResFile()
f.Open("res:/UI/Shared/Maps/mapcache.dat")
mapcache = blue.marshal.Load(f.Read())

Generates this error:

Traceback (most recent call last):
File "...\boogie.py", line 34, in < module >
mapcache = blue.marshal.Load(f.Read())
File "C:\Python27\lib\site-packages\reverence\blue.py", line 208, in _find_global
raise RuntimeError("Unable to locate object: " + module + "." + name + " (not in module)")
RuntimeError: Unable to locate object: util.KeyVal (not in module)
Entity
X-Factor Industries
Synthetic Existence
#144 - 2013-10-17 01:59:53 UTC
Eeki Tsaibit wrote:
RuntimeError: Unable to locate object: util.KeyVal (not in module)


Hm, apparently there are still objects using the old class names.

I've pushed a fix for this on github.

If you can't compile the new version yourself but want to fix it, make these changes.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Eeki Tsaibit
Caldari Provisions
Caldari State
#145 - 2013-10-17 02:11:37 UTC
Thanks a lot for the incredibly quick and effective reply. Fixed.

Also, awesome work with the lib. =)
Nuke Cherenkov
The Scope
Gallente Federation
#146 - 2013-10-21 04:28:54 UTC
Google only found this exception once but no explanation... Need help sorting it out (self taught python skills), Win 7, Py 2.7.5/32, latest compiled Reverence:

UnmarshalError: find_global failed to resolve: eve.common.script.sys.eveCfg.StackSize

This is the code:

cache_files = glob.glob(cachemgr.machocachepath + os.sep +
'CachedMethodCalls' + os.sep +
'*.cache')

for cache_file in cache_files:
with open(cache_file, 'rb') as f:
data = f.read()

try:
obj = blue.marshal.Load(data)

except UnmarshalError:
#FIXME: UnmarshalError: find_global failed to resolve: eve.common.script.sys.eveCfg.StackSize
return

Appreciate any advice!
Entity
X-Factor Industries
Synthetic Existence
#147 - 2013-10-22 09:23:42 UTC
Nuke Cherenkov wrote:
UnmarshalError: find_global failed to resolve: eve.common.script.sys.eveCfg.StackSize


Yeah I figured I'd forget a few objects...

Fixed on git



╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Vaerah Vahrokha
Vahrokh Consulting
#148 - 2013-10-26 12:39:17 UTC  |  Edited by: Vaerah Vahrokha
Reading market history, since today I get this error:


Traceback (most recent call last):
File "EvEMarketHistoryDumpOHLC.py", line 111, in module
File "reverence\cache.pyc", line 259, in LoadCacheFolder
UnmarshalError: find_global failed to resolve: eve.common.script.sys.eveCfg.Stac
kSize
Exception TypeError: 'expected string or Unicode object, NoneType found' in
module 'threading' from 'E:\Python27\dist\library.zip\threading.pyc' ignored



I suppose it the same error posted above?


Edit: I tried downloading the updated version but no Windows installer is available. Could you please release it? ATM I can't access a computer with Visual Studio to compile from source code.
Entity
X-Factor Industries
Synthetic Existence
#149 - 2013-10-26 18:18:42 UTC
Vaerah Vahrokha wrote:
Edit: I tried downloading the updated version but no Windows installer is available. Could you please release it? ATM I can't access a computer with Visual Studio to compile from source code.


Done (see OP for location)

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Vaerah Vahrokha
Vahrokh Consulting
#150 - 2013-10-27 02:58:09 UTC
Entity wrote:
Vaerah Vahrokha wrote:
Edit: I tried downloading the updated version but no Windows installer is available. Could you please release it? ATM I can't access a computer with Visual Studio to compile from source code.


Done (see OP for location)


Thank you so much!
Elmore Jones
New Eden Mining Organisation
The Craftsmen
#151 - 2013-11-17 16:20:39 UTC  |  Edited by: Elmore Jones
I've been trying to update my market order scraper to the newer cache stuff, and I seem to be missing something crucial :(

EVEROOT = r"C:\Program Files (x86)\CCP\EVE"
OUTPATH = r"C:\Users\Me\Documents\EVE\logs\Marketlogs"

eve = blue.EVE(EVEROOT)
cfg = eve.getconfigmgr()
cachemgr = eve.getcachemgr()
# cmc = cachemgr.LoadCacheFolder("CachedMethodCalls")
path = os.path.join(cachemgr.machocachepath,"CachedMethodCalls")

print path

cmc = cachemgr.LoadCacheFolder(path)

print "Starting...\n"

print cmc

for key, obj in cmc.iteritems():

.....

Prior to changing the path for CachedMethodCalls the script just gave an empty cmc as has been covered in previous posts. However with the new path in place (as demonstrated in the error dump below and checked through explorer) reverence crashes out at the line cmc = cachemgr.LoadCacheFolder(path).

C:\Users\Me\AppData\Local\CCP\EVE\c_program_files_(x86)_ccp_eve_tranquility\cache\MachoNet\87.237.38.200\382\CachedMethodCalls
Traceback (most recent call last):
File "C:\Users\Me\Documents\Projects\mydump_b.py", line 29, in module
cmc = cachemgr.LoadCacheFolder(path)
File "C:\Python27\lib\site-packages\reverence\cache.py", line 259, in LoadCach
eFolder
what, obj = blue.marshal.Load(_readfile(filename))
UnmarshalError: find_global failed to resolve: eve.common.script.sys.rowset.Rows
et
Exception TypeError: 'expected string or Unicode object, NoneType found' in mod
ule 'threading' from 'C:\Python27\lib\threading.pyc' ignored

I'm not too good with python so please be patient with me on this :) What did I do wrong?

EDIT : I'm using Reverence 1.6.1 and latest python 2.7 32 bit on 64 bit OS

+++ Reality Error 404 - Reboot Cosmos +++

Entity
X-Factor Industries
Synthetic Existence
#152 - 2013-11-18 02:45:35 UTC
Elmore Jones wrote:
I'm not too good with python so please be patient with me on this :) What did I do wrong?


Not much. I suspect CCP did another little ninja change. I'll look into it after the expansion hits, it's not worth updating Reverence right before a major release.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Elmore Jones
New Eden Mining Organisation
The Craftsmen
#153 - 2013-11-18 08:23:50 UTC
Thanks :) I'll sit tight then.

+++ Reality Error 404 - Reboot Cosmos +++

Elmore Jones
New Eden Mining Organisation
The Craftsmen
#154 - 2013-11-19 22:47:46 UTC  |  Edited by: Elmore Jones
Quick update to last posts: I cleared my cache with rubicon today and the error went away :) Guess something got corrupted my end.

: EDIT : still not quite the full ticket though... Only some of the viewed market orders are going through to cache it seems. of 6 i viewed after clearing orders via evemon (and closing it again) only 3 showed up in list as my script ran the cache. Hopefully a bug ccps end and fixed soon.

+++ Reality Error 404 - Reboot Cosmos +++

Entity
X-Factor Industries
Synthetic Existence
#155 - 2013-11-21 16:26:00 UTC
1.6.2 now available.

Fixes a few minor issues and the rowset bug mentioned earlier.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Elmore Jones
New Eden Mining Organisation
The Craftsmen
#156 - 2013-11-22 00:28:39 UTC
/hug

+++ Reality Error 404 - Reboot Cosmos +++

Elmore Jones
New Eden Mining Organisation
The Craftsmen
#157 - 2013-11-23 14:30:21 UTC
Not sure if this is a bug or me doing it wrong... I'd like my script to optionally delete the source cache file after reading :

for key, obj in cmc.iteritems():

if "GetOrders" in key:
# grab the cache file name
cachefilename = cachemgr.GetCacheFileName(key)

..... do stuff ....

print os.path.join(cachepath, cachefilename)
os.remove(os.path.join(cachepath, cachefilename))

however the returned file names don't exist :s The print returns something like this :

C:\Users\Me\AppData\Local\CCP\EVE\c_program_files_(x86)_ccp_eve_tranquility\cache\MachoNet\87.237.38.200\384\CachedMethodCalls\6cae.cache

I've manually checked the path etc but no avail. Any clues would be much appreciated :) Reverence 1.6.2 in use.

+++ Reality Error 404 - Reboot Cosmos +++

Kindred Deninard
Federal Navy Academy
Gallente Federation
#158 - 2013-11-23 14:38:44 UTC  |  Edited by: Kindred Deninard
Quote:
Hmm does this library also have a version that works with Python 3?


Nevermind, I didn't read the entire thread to see that it already had been asked.
Entity
X-Factor Industries
Synthetic Existence
#159 - 2013-11-23 19:21:11 UTC
Elmore Jones wrote:
cachefilename = cachemgr.GetCacheFileName(key).


Known issue.. It probably has to do with the differences between int, long, str and unicode. They can hash to different things even though they have the same value. Haven't pinpointed the exact issue though.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?

Elmore Jones
New Eden Mining Organisation
The Craftsmen
#160 - 2013-11-23 19:32:59 UTC
No worries, thanks

+++ Reality Error 404 - Reboot Cosmos +++