These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Reverence - 100% compatible EVE cache library for Python

First post
Author
Previous Topic Next Topic
Entity
X-Factor Industries
Synthetic Existence
#161 - 2013-11-23 19:43:11 UTC  |  Edited by: Entity
Actually, I looked into it deeper, and it has to do with cPickle encoding the same data to something different.

Namely:

this is what one of the keys pickles to:
"(S'marketProxy'\np1\nS'GetOrders'\nL10000002L\nI15618\ntp2\n."
gives 6456.cache (wrong)

this is what it SHOULD be pickling to.
"(S'marketProxy'\np1\nS'GetOrders'\np2\nL10000002L\nI15618\ntp3\n."
gives ffb4.cache (correct)

Not entirely sure why it does that.

After some more digging it seems this is because CCP is being naughty and is relying on pickle to produce consistent output for the same object, which you should not do because pickle's result depends on the refcounts of the objects. Sigh Roll

The only fix for Reverence I can think of causes a memory leak though. CCP will have to change the hash method to not use pickle to fix this issue properly.

I've added a workaround for it which seems to work (basically recursively increasing the refcounts in the key).

Attention Reverence 1.6.3 is now available.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Elmore Jones
New Eden Mining Organisation
The Craftsmen
#162 - 2013-11-24 10:03:47 UTC
Once again thank you very much :) Outstanding support!

+++ Reality Error 404 - Reboot Cosmos +++
Reddit Originated
Doomheim
#163 - 2013-11-27 19:41:51 UTC
Granted, I'm very green at this, but I could really use some help.
In the default history.py example I found that converts Eve cache into readable .csv, I got a file formatting problem (I think).

It works fine until I get to items with period in them, for example R.A.M.- Armor/Hull Tech. Then I get an error:

Quote:

Traceback (most recent call last):
File "history.py", line 23, in [module]
csvfile = open(os.path.join(OUTPATH, item.name+"-"+region.locationName+".csv"), 'w')
IOError: [Errno 2] No such file or directory: u'H:/eve_dump\\R.A.M.- Armor/Hull Tech-The Forge.csv'


Line in question:
Quote:
csvfile = open(os.path.join(OUTPATH, item.name+"-"+region.locationName+".csv"), 'w')
Entity
X-Factor Industries
Synthetic Existence
#164 - 2013-11-27 21:16:49 UTC
Reddit Originated wrote:
Granted, I'm very green at this, but I could really use some help.
In the default history.py example I found that converts Eve cache into readable .csv, I got a file formatting problem (I think).

It works fine until I get to items with period in them, for example R.A.M.- Armor/Hull Tech. Then I get an error:

Quote:

Traceback (most recent call last):
File "history.py", line 23, in [module]
csvfile = open(os.path.join(OUTPATH, item.name+"-"+region.locationName+".csv"), 'w')
IOError: [Errno 2] No such file or directory: u'H:/eve_dump\\R.A.M.- Armor/Hull Tech-The Forge.csv'


Line in question:
Quote:
csvfile = open(os.path.join(OUTPATH, item.name+"-"+region.locationName+".csv"), 'w')


it's the slash causing the issue. replace the clash with something else.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
3Jana
3Jana Corporation
#165 - 2013-12-17 05:39:28 UTC
I'm seeing an issue where the "jumps" field in the DBRows for market entries in the cache is always 0.
Entity
X-Factor Industries
Synthetic Existence
#166 - 2013-12-17 05:41:20 UTC
3Jana wrote:
I'm seeing an issue where the "jumps" field in the DBRows for market entries in the cache is always 0.


That's normal. Server data doesn't need jumps. the client calculates it based on your autopilot settings.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
jita moneymaker
Space Butterfly
#167 - 2014-01-15 18:05:31 UTC
Giving you a heads up you should include this link
https://www.dropbox.com/sh/xd6id81qi6jo0o9/jdAjrZU2wP
on github.
this link
https://github.com/ntt/reverence/downloads
has older versions
Halika Androm
Aliastra
Gallente Federation
#168 - 2014-01-26 09:54:34 UTC
First off, thank you for this great tool!

I have a quick question - is it possible to quickly get the typeID for an item based on its typeName? I'd rather get this info from the bulkdata than from a separate csv list.

I was looking at the sources and trying to work out whether I would need to use intypes.Select() or something rather than invtypes.Get() (which only accepts typeID if I'm not mistaken).
Entity
X-Factor Industries
Synthetic Existence
#169 - 2014-01-26 14:18:33 UTC
Halika Androm wrote:
I have a quick question - is it possible to quickly get the typeID for an item based on its typeName?


Sure can.

typesByName = cfg.invtypes.IndexedBy("typeName")

then you can typesByName.Get(name)

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Halika Androm
Aliastra
Gallente Federation
#170 - 2014-01-26 14:35:44 UTC
Entity wrote:
Halika Androm wrote:
I have a quick question - is it possible to quickly get the typeID for an item based on its typeName?


Sure can.

typesByName = cfg.invtypes.IndexedBy("typeName")

then you can typesByName.Get(name)


Ah yes, makes sense! I don't know how I didn't see IndexedBy when I was looking at Select, SortBy etc heh. Cheers!
Entity
X-Factor Industries
Synthetic Existence
#171 - 2014-01-29 01:40:53 UTC
There is currently a crash bug when accessing specific tables in Rubicon 1.1.

A fix is being prepared and will be posted sometime later today or tomorrow.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Entity
X-Factor Industries
Synthetic Existence
#172 - 2014-01-29 18:14:44 UTC
1.6.4 is now available (see OP for download location)

Notable changes:
- updated for Rubicon 1.1
- fixed crash bug in FSD table loader.
- fixed more FSD related issues and refactored a few bits here and there.
- added example script demonstrating how to generate the new Traits tab from the data.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Entity
X-Factor Industries
Synthetic Existence
#173 - 2014-01-31 18:43:46 UTC
1.6.5 released

- Some signed/unsigned bug fix in FSD.
- Traits example fixed.

- Added averagePrice property to type records (as returned by cfg.invtypes.Get(typeID)), which should reflect the ingame price estimate (this requires the client having been run recently for the price data to be there and accurate).
- packaged container volumes moved to const.shipPackagedVolumesPerGroup.
- Removed a few deprecated methods from Rowset.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Halika Androm
Aliastra
Gallente Federation
#174 - 2014-02-18 15:38:30 UTC
It appears that the EVE update today has broken something. This line was working fine this morning before the patch:

Quote:
region_id = long(self._configmgr.regions.IndexedBy('regionName').Get(self.region).regionID)

but now I get this error:

Quote:

File "D:\Tools\Python27\lib\site-packages\reverence\config.py", line 390, in __get__
value = self.method(obj)
File "D:\Tools\Python27\lib\site-packages\reverence\config.py", line 403, in method
return self._loadbulkdata(tableName=attrName, storageClass=storageClass, rowClass=rowClass, primaryKey=primaryKey, bulkID=bulkID)
File "D:\Tools\Python27\lib\site-packages\reverence\config.py", line 1004, in _loadbulkdata
raise RuntimeError("Unable to load '%s' (bulkID:%d)" % (tableName, bulkID))
RuntimeError: Unable to load 'regions' (bulkID:1400009)
Dalden V
Yellow Lounge Industries
#175 - 2014-02-18 19:09:08 UTC
Thanks Entity for a great library. I've been using it in my own personal project for a while now.

After today's patch (Rubicon 1.2), I'm getting the following error in code that worked yesterday:

Code:

Quote:
EVEROOT = r"C:/Games/EVE"
eve = blue.EVE(EVEROOT)
cfg = eve.getconfigmgr()
cachemgr = eve.getcachemgr()
..
path = os.path.join(cachemgr.machocachepath,"CachedMethodCalls")
cmc = cachemgr.LoadCacheFolder(path)


Error:

Quote:
Traceback (most recent call last):
File "D:\eveweb\clientside\cache.py", line 126, in module
cmc = cachemgr.LoadCacheFolder(path)
File "C:\Python27\lib\site-packages\reverence\cache.py", line 277, in LoadCacheFolder
what, obj = blue.marshal.Load(_readfile(filename))
UnmarshalError: find_global failed to resolve: eve.common.script.universe.locationWrapper.SolarSystemWrapper
Exception TypeError: 'expected string or Unicode object, NoneType found' in module 'threading' from 'C:\Python27\lib\threading.pyc' ignored
Entity
X-Factor Industries
Synthetic Existence
#176 - 2014-02-18 22:06:07 UTC  |  Edited by: Entity
Looks like there are quite a few changes. This may take a while.

Edit: yep, lots of tables shuffled around/moved to FSD and some other stuff. Next update is going to break plenty of things I bet.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Kristoffon vonDrake
Forceful Resource Acquisition Inc
#177 - 2014-02-21 14:28:52 UTC  |  Edited by: Kristoffon vonDrake
Hi, I'm trying to get Phobos to work and I'm running into this problem, I assume it's related to the above?

screenshot because I tried pasting it but the stupid forums think there's some html and delete my whole post
Entity
X-Factor Industries
Synthetic Existence
#178 - 2014-02-21 23:58:04 UTC  |  Edited by: Entity
Reverence 1.7.0 for Rubicon 1.2 is now available in the usual place (see OP).

This update may break your stuff!

Notable changes:

- The translation pickles are now always loaded (defaults to "en-us").
This is due to changes forcing me to retrieve names from translation data. It also makes the code a bit cleaner since I no longer have to maintain 2 code paths in certain methods.

- regions, constellations, solarsystems in cfg are replaced by mapRegionCache, mapConstellationCache, mapSystemCache.
These tables do not contain the names of those locations! Use cfg.evelocations.Get(x).locationName for that.

- There's a new huge table using the MultiIndex feature of FSD: cfg.mapSolarSystemContentCache
It's a table indexed on solarSystemID and the values are rows with the content of a system (like gates and belts and stuff), it has alternate indices available on these attributes: celestials, asteroidBelts, stargates, planets, moons, npcStations.
Getting celestial names is a bit of a problem though, it's supposed to be done through evelocations as well, but I've not implemented the required dynamic lookup shenanigans for that yet.

- fsdPlanetAttributes is gone. this information is in table mentioned above. I don't think anyone even used this though.

- This version is NOT compatible with previous versions of EVE.

- There will probably still be some bugs.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Dalden V
Yellow Lounge Industries
#179 - 2014-02-22 07:20:49 UTC
Entity wrote:
Reverence 1.7.0 for Rubicon 1.2 is now available in the usual place (see OP).


Wow, that was quick! I really appreciate how much work you've put into this library. All my market code is working again with this version as far as I can tell (GetOrders, GetCharOrders, GetOldPriceHistory).

I'll let you know if I run into any problems.
Entity
X-Factor Industries
Synthetic Existence
#180 - 2014-02-22 18:09:27 UTC
[rant on]
On a side note, if you're wondering why Reverence's memory usage ramps up as soon as you access something that needs a translation...

it's because CCP failed to implement an optimization I told them about 2 years ago that reduces the memory used by the translation data to approx. 60MB (down from 100) Roll
[rant off]

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?