These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Reverence - 100% compatible EVE cache library for Python

First post
Author
Previous Topic Next Topic
D0main
Royal Amarr Institute
Amarr Empire
#101 - 2012-08-13 20:38:09 UTC
approve. i have the same problem with cmc = cachemgr.LoadCacheFolder("CachedMethodCalls") i have an empty variable.
D0main
Royal Amarr Institute
Amarr Empire
#102 - 2012-08-13 20:51:05 UTC
im modified code like this:

path = os.path.join(cachemgr.machocachepath,"CachedMethodCalls")
cmc = cachemgr.LoadCacheFolder(path)

and cache directory were loading, but i didnt have any records with key[1]=="GetOldPriceHistory".
is inferno 1.2 patch nerf this or i did something wrong?
Death2Amarr
Minmatar Death Squad
DECOY
#103 - 2012-08-13 23:25:02 UTC
D0main wrote:
im modified code like this:

path = os.path.join(cachemgr.machocachepath,"CachedMethodCalls")
cmc = cachemgr.LoadCacheFolder(path)

and cache directory were loading, but i didnt have any records with key[1]=="GetOldPriceHistory".
is inferno 1.2 patch nerf this or i did something wrong?


Thanks D0main, that fixed it for me. Big smile
D0main
Royal Amarr Institute
Amarr Empire
#104 - 2012-08-14 04:33:33 UTC
Death2Amarr, how do you parse price history?
D0main
Royal Amarr Institute
Amarr Empire
#105 - 2012-08-14 07:50:41 UTC
problem solved!

when you get market items you will need to select history tab. Lol
Verite Rendition
F.R.E.E. Explorer
#106 - 2012-08-28 00:14:24 UTC  |  Edited by: Verite Rendition
Hey Entity, thanks for adding Serenity to the server alias list. I just started working on some CEVE stuff, so that came in handy.Smile

WIth that said, I ran in to what amounts to a Reverence bug when used with the current version of CEVE. The minimum Machonet version listed for a few BulkData tables is too high. Specifically "fsdTypeOverrides" "fsdPlanetAttributes" and "graphics" are all present as of 324 (the current CEVE version) rather than being introduced in 327 as Reverence is currently programmed to assume.

Everything else is working well though, so nicely done.Smile

Finally, if there's any way to coax you into updating the datadump script, it would be appreciated. I bashed on it until it worked, but with your LoadCacheFolder changes I'm not all that sure whether my changes are even close to being reasonable code.Ugh
Entity
X-Factor Industries
Synthetic Existence
#107 - 2012-08-28 09:09:08 UTC
Verite Rendition wrote:
Hey Entity, thanks for adding Serenity to the server alias list. I just started working on some CEVE stuff, so that came in handy.Smile

WIth that said, I ran in to what amounts to a Reverence bug when used with the current version of CEVE. The minimum Machonet version listed for a few BulkData tables is too high. Specifically "fsdTypeOverrides" "fsdPlanetAttributes" and "graphics" are all present as of 324 (the current CEVE version) rather than being introduced in 327 as Reverence is currently programmed to assume.

Everything else is working well though, so nicely done.Smile

Finally, if there's any way to coax you into updating the datadump script, it would be appreciated. I bashed on it until it worked, but with your LoadCacheFolder changes I'm not all that sure whether my changes are even close to being reasonable code.Ugh


Thanks, lowered the required protocol version.
As for the dump script, yeah, I'll update at some point :) :lazy:

You might want to check out this one a friend made:
http://fisheye.dev.evefit.org/browse/phobos
It only does json but it should be easy to convert to sql.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Entity
X-Factor Industries
Synthetic Existence
#108 - 2013-05-30 16:35:23 UTC  |  Edited by: Entity
ISD Eshtir wrote:
Unlock test


Cheers ♥

Odyssey version of Reverence Soon™

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Entity
X-Factor Industries
Synthetic Existence
#109 - 2013-06-04 14:54:44 UTC  |  Edited by: Entity
Alright, Odyssey updates pushed.

Have to figure out some way to post the binaries though, since Github stopped providing a downloads section. So for now, you'll have to build it yourself.

Particularly noteworthy are the changes to the FSD loader. Odyssey features a new one that is incompatible with the Retribution one, and provides a slightly different interface to the data. If your program uses any of the FSD based data, your code may require updating.

Important note for those using stripped EVE installs just to satisfy Reverence's needs:
Reverence now requires EVE's lib folder to be present, as it needs one of the python libraries for decoding FSD (in particular; fsdSchemas-1.0.2.zip).

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Fight Dealer
University of Caille
Gallente Federation
#110 - 2013-06-09 00:26:25 UTC  |  Edited by: Fight Dealer
Entity wrote:
Alright, Odyssey updates pushed.

Have to figure out some way to post the binaries though, since Github stopped providing a downloads section. So for now, you'll have to build it yourself.

Particularly noteworthy are the changes to the FSD loader. Odyssey features a new one that is incompatible with the Retribution one, and provides a slightly different interface to the data. If your program uses any of the FSD based data, your code may require updating.

Important note for those using stripped EVE installs just to satisfy Reverence's needs:
Reverence now requires EVE's lib folder to be present, as it needs one of the python libraries for decoding FSD (in particular; fsdSchemas-1.0.2.zip).


Thanks for Odyssey version!

Could you please make a binary Windows installer (32-bit) and upload it to just any file hosting? I dont know how to build Reverence manually, not a Python developer :(

UPDATE: Finally, I was able to build it. Implants.py works.

But Autodumper suddenly stopped working with this version of Reverence.
Following is a stack trace:
https://dl.dropboxusercontent.com/u/3776896/Autodumper.png

I googled it, and it looks like a common issue with multithreading in Python on Windows. But it worked before...

Any ideas?
Fight Dealer
University of Caille
Gallente Federation
#111 - 2013-06-09 16:05:54 UTC  |  Edited by: Fight Dealer
Btw, datadump.py produces no output.
I 've set both EVE and output locations in script.

It doesnt look like we're getting any objects in both

cachedObjects = c.LoadCacheFolder("BulkData")
and
cachedObjects2 = c.LoadCacheFolder("CachedObjects")
Entity
X-Factor Industries
Synthetic Existence
#112 - 2013-06-09 21:32:55 UTC
Yes the examples are hopelessly outdated. Known issue :P

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Fight Dealer
University of Caille
Gallente Federation
#113 - 2013-06-10 09:46:23 UTC  |  Edited by: Fight Dealer
Actually, fixing the datadump.py example was easy. I changed the following basing on what was mentioned earlier in this thread:

From:

cachedObjects = c.LoadCacheFolder("BulkData")
cachedObjects2 = c.LoadCacheFolder("CachedObjects")

To:

path1 = os.path.join(c.machocachepath,"BulkData")
cachedObjects = c.LoadCacheFolder(path1)

path2 = os.path.join(c.machocachepath,"CachedObjects")
cachedObjects2 = c.LoadCacheFolder(path2)

And now this example spits some XMLs, so looks like Reverence itself works.

But, my main issue is about Autodumper launching Reverence. It refuces to work with regard to some locking/multithreading (details in my first post in this thread).

Do you have any ideas of what can be wrong about the threading?

If not, could you please provide me with some short code snippet of how to correctly get market orders (not current character's orders, but just dump of any market page visited in Market window) from cache using Reverence (so I can spit them to XML or whatever)?

Thanks in advance.
Taleden
North Wind Local no. 612
#114 - 2013-08-16 21:20:21 UTC  |  Edited by: Taleden
I wanted to give this a try, but can't get it compiled.

I have "Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32" on Windows 7 64bit, and I just grabbed the "revenance-master.zip" download from github and tried "python setup.py install".

At first I got "unable to find vcvarsall.bat" so I installed Visual C++ 2008 Express Edition with SP1; now the compile gets further, but I get an exception:

Quote:

building 'reverence._blue' extension
Traceback (most recent call last):
File "setup.py", line 67, in module
package_data = {"reverence": ['*.txt']},
File "C:\Python27\lib\distutils\core.py", line 152, in setup
dist.run_commands()
File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Python27\lib\site-packages\distribute-0.6.14-py2.7.egg\setuptools\command\install.py", line 73, in run
self.do_egg_install()
File "C:\Python27\lib\site-packages\distribute-0.6.14-py2.7.egg\setuptools\command\install.py", line 93, in do_egg_install
self.run_command('bdist_egg')
File "C:\Python27\lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Python27\lib\site-packages\distribute-0.6.14-py2.7.egg\setuptools\command\bdist_egg.py", line 179, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "C:\Python27\lib\site-packages\distribute-0.6.14-py2.7.egg\setuptools\command\bdist_egg.py", line 166, in call_command
self.run_command(cmdname)
File "C:\Python27\lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Python27\lib\site-packages\distribute-0.6.14-py2.7.egg\setuptools\command\install_lib.py", line 20, in run
self.build()
File "C:\Python27\lib\distutils\command\install_lib.py", line 111, in build
self.run_command('build_ext')
File "C:\Python27\lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Python27\lib\site-packages\distribute-0.6.14-py2.7.egg\setuptools\command\build_ext.py", line 46, in run
_build_ext.run(self)
File "C:\Python27\lib\distutils\command\build_ext.py", line 339, in run
self.build_extensions()
File "C:\Python27\lib\distutils\command\build_ext.py", line 448, in build_extensions
self.build_extension(ext)
File "C:\Python27\lib\site-packages\distribute-0.6.14-py2.7.egg\setuptools\command\build_ext.py", line 182, in build_extension
_build_ext.build_extension(self,ext)
File "C:\Python27\lib\distutils\command\build_ext.py", line 498, in build_extension
depends=ext.depends)
File "C:\Python27\lib\distutils\msvc9compiler.py", line 473, in compile
self.initialize()
File "C:\Python27\lib\distutils\msvc9compiler.py", line 383, in initialize
vc_env = query_vcvarsall(VERSION, plat_spec)
File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsall
raise ValueError(str(list(result.keys())))
ValueError: [u'path']


Any ideas?

EDIT: This appears to be a known issue with Visual Studio 2008 Express and python distutils; the workaround that solved it for me was to manually execute "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\Tools\vsvars32.bat" in my command prompt, followed by "python setup.py install".
Wafflehead
Garoun Investment Bank
Gallente Federation
#115 - 2013-08-17 18:54:07 UTC
Fight Dealer wrote:
Actually, fixing the datadump.py example was easy. I changed the following basing on what was mentioned earlier in this thread:

From:

cachedObjects = c.LoadCacheFolder("BulkData")
cachedObjects2 = c.LoadCacheFolder("CachedObjects")

To:

path1 = os.path.join(c.machocachepath,"BulkData")
cachedObjects = c.LoadCacheFolder(path1)

path2 = os.path.join(c.machocachepath,"CachedObjects")
cachedObjects2 = c.LoadCacheFolder(path2)

And now this example spits some XMLs, so looks like Reverence itself works.

But, my main issue is about Autodumper launching Reverence. It refuces to work with regard to some locking/multithreading (details in my first post in this thread).

Do you have any ideas of what can be wrong about the threading?

If not, could you please provide me with some short code snippet of how to correctly get market orders (not current character's orders, but just dump of any market page visited in Market window) from cache using Reverence (so I can spit them to XML or whatever)?

Thanks in advance.


Any news on this?.. I cannot get it to work..
Using this code: http://pastebin.com/DfuVgyL4

Doesn't produce any file..
Taleden
North Wind Local no. 612
#116 - 2013-08-18 05:34:07 UTC
Wafflehead wrote:

Any news on this?.. I cannot get it to work..
Using this code: http://pastebin.com/DfuVgyL4
Doesn't produce any file..


If you look closely at the post you quoted, you'll see the solution: LoadCacheFolder() no longer seems to work with just simply 'CachedMethodCalls'. Instead you need to give it the full path name, which means using os.path.join() and cache.machocachepath:

Quote:
cmc = cachemgr.LoadCacheFolder(os.path.join(cachemgr.machocachepath,'CachedMethodCalls'))



Now, I've got another question for Entity: is there any reason that you omitted the invMarketGroups table? We have invTypes, invGroups, invCategories, lots of other db tables, and we even have a nice pre-grouped typesByMarketGroups, but we don't seem to have the data available to translate marketGroupIDs into their labels or tree structure.
Entity
X-Factor Industries
Synthetic Existence
#117 - 2013-08-18 09:48:02 UTC
Taleden wrote:
Now, I've got another question for Entity: is there any reason that you omitted the invMarketGroups table? We have invTypes, invGroups, invCategories, lots of other db tables, and we even have a nice pre-grouped typesByMarketGroups, but we don't seem to have the data available to translate marketGroupIDs into their labels or tree structure.


Last I checked invMarketGroups is not in static data. It's fetched when accessing market.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Taleden
North Wind Local no. 612
#118 - 2013-08-18 14:25:32 UTC
Entity wrote:
Last I checked invMarketGroups is not in static data. It's fetched when accessing market.


This post provides conversions of the CCP-provided Static Data Export for Odyssey 1.0, and the sqlite version at least does include invmarketgroups.

Also, this page says he actually used Reverence to scrape the cache/bulkdata from the Odyssey client, and his "SQLite Database dump" also contains invmarketgroups, even though it comes with the warning that it does not contain everything that the official CCP dump has because it's a local client scrape.

So is there actually some way to get to this data using Reverence that I'm just not seeing?
Entity
X-Factor Industries
Synthetic Existence
#119 - 2013-08-18 18:01:06 UTC
Taleden wrote:
Entity wrote:
Last I checked invMarketGroups is not in static data. It's fetched when accessing market.


This post provides conversions of the CCP-provided Static Data Export for Odyssey 1.0, and the sqlite version at least does include invmarketgroups.

Also, this page says he actually used Reverence to scrape the cache/bulkdata from the Odyssey client, and his "SQLite Database dump" also contains invmarketgroups, even though it comes with the warning that it does not contain everything that the official CCP dump has because it's a local client scrape.

So is there actually some way to get to this data using Reverence that I'm just not seeing?


I didn't say the data wasn't there. But it is a cache entry.

access it with

bla = eve.RemoteSvc("marketProxy").GetMarketGroups()

you need to have accessed the market before on the EVE install you are using.
Ill let you figure ut what the datastructure is :P

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Taleden
North Wind Local no. 612
#120 - 2013-08-19 04:46:41 UTC
Entity wrote:
I didn't say the data wasn't there. But it is a cache entry.

access it with

bla = eve.RemoteSvc("marketProxy").GetMarketGroups()

you need to have accessed the market before on the EVE install you are using.
Ill let you figure ut what the datastructure is :P


Interesting.. it's a little roundabout but I think I see how to use it.

But when I try to recursively print the market group tree, some subheaders have no names. For example, Ship Equipment / Hull & Armor / Armor Plates is marketGroupID #133, and that should contain subgroups for "50mm Armor Plate" up to "1600mm Armor Plate". But in the returned structure:

Quote:

for row in getmarketgroups[133]:
print row

DBRow(parentGroupID:133,marketGroupID:1671,marketGroupName:,description:,graphicID:None,hasTypes:True,iconID:None,dataID:85722404,marketGroupNameID:285369,descriptionID:285375)
DBRow(parentGroupID:133,marketGroupID:1672,marketGroupName:,description:,graphicID:None,hasTypes:True,iconID:None,dataID:85722405,marketGroupNameID:285368,descriptionID:285373)
DBRow(parentGroupID:133,marketGroupID:1673,marketGroupName:,description:,graphicID:None,hasTypes:True,iconID:None,dataID:85722406,marketGroupNameID:285367,descriptionID:285372)
DBRow(parentGroupID:133,marketGroupID:1674,marketGroupName:,description:,graphicID:None,hasTypes:True,iconID:None,dataID:85722407,marketGroupNameID:285366,descriptionID:285371)
DBRow(parentGroupID:133,marketGroupID:1675,marketGroupName:,description:,graphicID:None,hasTypes:True,iconID:None,dataID:85722408,marketGroupNameID:285365,descriptionID:285374)
DBRow(parentGroupID:133,marketGroupID:1676,marketGroupName:,description:,graphicID:None,hasTypes:True,iconID:None,dataID:85722409,marketGroupNameID:285364,descriptionID:285370)


So the subgroup records are there, and i.e. config.typesByMarketGroups[1671] returns the right items, but the subgroup names are missing. They do have marketGroupNameID values however, so I wonder where the text is supposed to be pulled from?