These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Reverence - 100% compatible EVE cache library for Python

First post
Author
Previous Topic Next Topic
Makari Aeron
Imperial Shipment
Amarr Empire
#81 - 2012-06-28 21:19:33 UTC  |  Edited by: Makari Aeron
Entity wrote:
Makari Aeron wrote:
Hello, I continue to receive this error from the 1.4.2 install from github. Not quite sure what to do. (forums wouldn't let me post the traceback. GJ to the forums team for that)

http://pastie.org/4163047

Thanks,
Makari

EDIT: I have my input and output paths set correctly and I have not changed the output mode from XML>


You're probably using a 64bit python? The installers only work for 32bit python. (Though I'll upload some 64bit installers soon™)


Ahh ok. That would explain it. Yeah, I had to do a little RegEdit hack to just get it to install since the installer likes to read from the 64bit Software registry key area (I am guessing it's "default" and the 32bit installer read the "default" location) and Python installs default-ly (even 64bit) into the 32bit registry key area. Normally I'd just install the 32bit and be done with it, but I've got other software running off the 64-bit >.<

Well, I'll look forward to the soon™, I'll be waiting ;)


EDIT: I tried to compile the source myself, but it said I was missing a batch file (at work; don't remember the full name but it had VS in it; I'm guessing for M$ VS 2010 stuff)

CCP RedDawn: Ugly people are just playing life on HARD mode. Personally, I'm playing on an INFERNO difficulty.

CCP Goliath: I often believe that the best way to get something done is to shout at the person trying to help you. http://goo.gl/PKGDP
Entity
X-Factor Industries
Synthetic Existence
#82 - 2012-06-28 21:31:10 UTC  |  Edited by: Entity
Atsui Stenier-Tri wrote:
Hi, and first up:
Many Kudos to you sir, great work.
The sources never lie, you really know your stuff. Cool

just a dumb question:
do the CachedMethodCalls files somewhere contain a timestamp on when the function was called/values were returned?
If so, how to extract it?

if this already has been asked: forgiveth my humble self.


The first element in the 'version' key of those objects is a File Time value, so you'd do something like this:

from reverence import blue, util
data = open(FILENAMEHERE, "rb").read()
obj = blue.marshal.Load(data)
timestamp = obj[1]['version'][0] # this is what you want.
print util.FmtDate(timestamp) # and reverence can turn it into something human readable for you too :)


Edit: one could probably also get away with just pulling the modification date of the file itself.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Entity
X-Factor Industries
Synthetic Existence
#83 - 2012-06-28 21:37:30 UTC
Makari Aeron wrote:
Yeah, I had to do a little RegEdit hack to just get it to install since the installer likes to read from the 64bit Software registry key area (I am guessing it's "default" and the 32bit installer read the "default" location) and Python installs default-ly (even 64bit) into the 32bit registry key area.


That's odd. I've always had python properly detect eligible install locations, and I have 5 or 6 different python versions installed :)

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Makari Aeron
Imperial Shipment
Amarr Empire
#84 - 2012-06-28 22:55:18 UTC
Entity wrote:
Makari Aeron wrote:
Yeah, I had to do a little RegEdit hack to just get it to install since the installer likes to read from the 64bit Software registry key area (I am guessing it's "default" and the 32bit installer read the "default" location) and Python installs default-ly (even 64bit) into the 32bit registry key area.


That's odd. I've always had python properly detect eligible install locations, and I have 5 or 6 different python versions installed :)


I thought so too so I reinstalled; still had the same issue. *shrug* I'll just wait for the x64 version. I'm in no hurry.

CCP RedDawn: Ugly people are just playing life on HARD mode. Personally, I'm playing on an INFERNO difficulty.

CCP Goliath: I often believe that the best way to get something done is to shout at the person trying to help you. http://goo.gl/PKGDP
Atsui Stenier-Tri
Federal Navy Academy
Gallente Federation
#85 - 2012-06-30 13:57:46 UTC  |  Edited by: Atsui Stenier-Tri
Entity wrote:

timestamp = obj[1]['version'][0] # this is what you want.


Great. Thank you.
Feanos
Brutor Tribe
Minmatar Republic
#86 - 2012-07-03 10:28:27 UTC
Been working with this and a python based uploader for market data (Yeah, I know, another one!) and it looks like I'm running into some marshal issues:

MARSHALLERROR Bogus map data in marshal stream
Scanning
('marketProxy', 'GetOrders', 10000064L, 13924)
Adding message to the queue for upload.
('marketProxy', 'GetOrders', 10000064L, 3153)
Adding message to the queue for upload.
200 1
200 1
Scanning
MARSHALLERROR Bogus map data in marshal stream
MARSHALLERROR Bogus map data in marshal stream
Scanning
MARSHALLERROR Bogus map data in marshal stream
MARSHALLERROR Bogus map data in marshal stream
MARSHALLERROR Bogus map data in marshal stream
MARSHALLERROR Bogus map data in marshal stream

Around 80% of the market orders are showing bogus map data. From what I can see, this is due to issues with the size of the shared map from the blue.c file. Any suggestions on how to avoid this, as if I'm losing 80% of the data, it's a bit rough to utilize the maximum number of exports possible. I'm also getting these errors on login, or anything that touches cache, 80-90% of them tend to be invalid, so I'd like to know if I'm just missing something stupid, or if there's something else at play here.

Quote:

try:
key, obj = blue.marshal.Load(fobj.read())
except blue.marshal.UnmarshalError:
# The file probably wasn't finished writing when EMDU tried to read it.
return None


That's the the code returning the error, without the debug printing in place.
Entity
X-Factor Industries
Synthetic Existence
#87 - 2012-07-03 10:49:02 UTC
Feanos wrote:
Been working with this and a python based uploader for market data (Yeah, I know, another one!) and it looks like I'm running into some marshal issues:

MARSHALLERROR Bogus map data in marshal stream

That's the the code returning the error, without the debug printing in place.


Which reverence version, python version? 32 or 64 bit python?
Got a copy of one of the files giving the error?

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Feanos
Brutor Tribe
Minmatar Republic
#88 - 2012-07-03 13:12:11 UTC
Sigh. I used to do bug reports for a living, I should of known better :D

Reverence 1.4.2
Python 2.7
All 32 bit installs, tried to make a 64 work, but it barfed all over the place, not source compiled, installed from the provided executable on github.

I have a large number of the files available, I'll be glad to provide these to you as needed.
Entity
X-Factor Industries
Synthetic Existence
#89 - 2012-07-04 14:58:33 UTC  |  Edited by: Entity
Feanos wrote:
Sigh. I used to do bug reports for a living, I should of known better :D

Reverence 1.4.2
Python 2.7
All 32 bit installs, tried to make a 64 work, but it barfed all over the place, not source compiled, installed from the provided executable on github.

I have a large number of the files available, I'll be glad to provide these to you as needed.


Just give me one that throws up that error (the smaller the better :)

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Feanos
Brutor Tribe
Minmatar Republic
#90 - 2012-07-05 00:21:31 UTC
E-mailed off, hope it helps you track this down :)
Entity
X-Factor Industries
Synthetic Existence
#91 - 2012-07-05 11:18:48 UTC
Feanos wrote:
E-mailed off, hope it helps you track this down :)


Eh? You're not reading the files in binary mode. Roll

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Feanos
Brutor Tribe
Minmatar Republic
#92 - 2012-07-05 11:39:51 UTC
Annnnd that's what I get for being too used to writing code for linux, where it just doesn't care. My apologies, and as soon as I make it rb, it's perfectly happy.
Entity
X-Factor Industries
Synthetic Existence
#93 - 2012-07-10 14:55:09 UTC  |  Edited by: Entity
Reverence 1.5.0 binaries now available.

There are 64bit Windows installers now too, yay! Big smile

New things:
- Improved stuff file handling (due to new virtual file class).
- Localized static data support (use languageID keyword when instantiating blue.EVE)
- FileStaticData support (CCP's new data format. This is mostly transparent, but accessing the data is slightly different)

A couple of things to note:
- Reverence now requires PyYAML.
- Minimum required EVE version for full compatibility is now Inferno 1.1.
- Python 2.5 will no longer be supported.


Note there's no official 64bit PyYAML installer, but you can download an unofficial one here:
http://www.lfd.uci.edu/~gohlke/pythonlibs/

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Splodger
Brutor Tribe
Minmatar Republic
#94 - 2012-07-31 13:53:57 UTC  |  Edited by: Splodger
running both the example datadump.py and a marketorder script run without errors, but I do not get any files in my outpath folder.
outpath folder is already made aswell.

installed both PyYAML and rev 2.7 (64bit)

any help as iam fairly new with python and at a loss
edit : tried the implants example aswell, I do get an implants.html but it is blank.
Entity
X-Factor Industries
Synthetic Existence
#95 - 2012-07-31 15:27:22 UTC
Splodger wrote:
running both the example datadump.py and a marketorder script run without errors, but I do not get any files in my outpath folder.
outpath folder is already made aswell.

installed both PyYAML and rev 2.7 (64bit)

any help as iam fairly new with python and at a loss
edit : tried the implants example aswell, I do get an implants.html but it is blank.


Yeah, datadump.py is out of date, haven't updated that yet (lazy me)
implants.py works fine, make sure you point it at your eve install in the source.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Splodger
Brutor Tribe
Minmatar Republic
#96 - 2012-07-31 15:33:09 UTC  |  Edited by: Splodger
Entity wrote:
Splodger wrote:
running both the example datadump.py and a marketorder script run without errors, but I do not get any files in my outpath folder.
outpath folder is already made aswell.

installed both PyYAML and rev 2.7 (64bit)

any help as iam fairly new with python and at a loss
edit : tried the implants example aswell, I do get an implants.html but it is blank.


Yeah, datadump.py is out of date, haven't updated that yet (lazy me)
implants.py works fine, make sure you point it at your eve install in the source.


EVEPATH = "C:/Program Files (x86)/CCP/EVE"

if i remove the space or set it wrong, will get error saying path to cache is wrong so I know the path is correct. but alas no data.
Entity
X-Factor Industries
Synthetic Existence
#97 - 2012-07-31 15:39:49 UTC
Splodger wrote:
if i remove the space or set it wrong, will get error saying path to cache is wrong so I know the path is correct. but alas no data.


No idea. As I said, it works fine. Try running it just with

python implants.py

(no output redirect).

It should be spitting out html. if it doesn't, it should be giving an exception. if it doesn't even do that, then I dont know :P

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Splodger
Brutor Tribe
Minmatar Republic
#98 - 2012-07-31 15:46:36 UTC
Entity wrote:
Splodger wrote:
if i remove the space or set it wrong, will get error saying path to cache is wrong so I know the path is correct. but alas no data.


No idea. As I said, it works fine. Try running it just with

python implants.py

(no output redirect).

It should be spitting out html. if it doesn't, it should be giving an exception. if it doesn't even do that, then I dont know :P


well as per my normal coding practices, its now working lol, cheers again. cant wait to experiment with it.
Splodger
Brutor Tribe
Minmatar Republic
#99 - 2012-08-05 18:56:21 UTC  |  Edited by: Splodger
Iam not getting any output with the following, anyone able to send any light? Was reading up on python dictionaries but my for loop below doesnt output anything, so either Iam doign it wrong or something is up. (installed evementat to make sure my cache was working and it is - valid dat in there)

import time
import os
import sys
from reverence import blue

EVEROOT = 'C:\Program Files (x86)\CCP\EVE'

eve = blue.EVE(EVEROOT)
cfg = eve.getconfigmgr()
cachemgr = eve.getcachemgr()
cmc = cachemgr.LoadCacheFolder('CachedMethodCalls')

#cache = eve.getcachemgr()
#print "Protocol version :", cache.machoVersion
#print "Installation Root :", cache.root
#print "Installation Bulkdata :", cache.BULK_SYSTEM_PATH
#print "AppData Cache Root :", cache.cachepath
#print "AppData MachoNet Cache :", cache.machocachepath
#print "AppData Bulkdata Updates:", cache.BULK_CACHE_PATH



for key, obj in cmc.iteritems():
if key[1]=="GetOrders":
item = cfg.invtypes.Get(key[3])
region = cfg.evelocations.Get(key[2])
print "Processing " + item.name + " [" + region.locationName +"]... \n"

EDit: with help understanding python from Daneel Trevize, We have worked out my cmc dictionary has no entries. still any further help appreciated.
Cherab
Mad Tekkers
#100 - 2012-08-10 21:07:19 UTC
Splodger wrote:


EDit: with help understanding python from Daneel Trevize, We have worked out my cmc dictionary has no entries. still any further help appreciated.


Same, i think it's a problem with LoadCacheFolder