These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Reverence - 100% compatible EVE cache library for Python

First post
Author
Previous Topic Next Topic
Magnomus
Perpetual Motion Machine
#21 - 2011-12-24 19:27:51 UTC
malaire wrote:

Since simple XML is basicly same as CSV except is uses a lot of crap instead of simple commas, this might work: http://pastie.org/private/7ssdk0mde7gdsfqeqkwjw

(and no, I don't know how to split that big line into smaller parts, might need to rewrite section following it to do that.)


Thanks. I probably missed something but that code still creates a CSV file, just structured differently. I was looking to get XML files much like those produced from: examples/datadump.py

I'm not sure if it's easier to get datadump.py tweaked to pull data from CachedMethodCalls (which I can't seem to do) or if it's easier to write something into my cmc script that throws out an XML instead of a CSV.

Thanks in advance.
malaire
#22 - 2011-12-24 20:13:20 UTC
yep, you also need to change ".csv" in code to ".xml" to get right filename extension.

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else
Magnomus
Perpetual Motion Machine
#23 - 2011-12-24 20:44:16 UTC  |  Edited by: Magnomus
I tried that already, it creates an unreadable xml file.

Edit: Sorry, I should have included this, which is the error received when opening the created XML files in XML Notepad:

Error Loading: C:\orderdump\Tritanium-The Forge.xml
There are multiple root elements. Line 15, position 2.

This is the error received when trying to open it in a browser:

The XML page cannot be displayed
Cannot view XML input using XSL style sheet. Please correct the error and then click the Refresh button, or try again later.



Only one top level element is allowed in an XML document. Error processing resource 'file:///C:/orderdump/Tritanium-The For...
Magnomus
Perpetual Motion Machine
#24 - 2011-12-24 21:21:19 UTC
So I tried editing the resulting .xml file manually deleted all the orders except the top one with the result being a functional xml file that just has one order in it, so judging by the errors I was getting I'm thinking the code you gave me just needs to be tweaked so that each order isn't trying to be the root (sadly I'm not certain of how to do this).
malaire
#25 - 2011-12-25 11:39:54 UTC
Magnomus wrote:
So I tried editing the resulting .xml file manually deleted all the orders except the top one with the result being a functional xml file that just has one order in it, so judging by the errors I was getting I'm thinking the code you gave me just needs to be tweaked so that each order isn't trying to be the root (sadly I'm not certain of how to do this).

You probably just need to add new element as root element, like this:
http://pastie.org/private/cfphr8ytk20pu7pbqklwtw
At least this results in XML file which I can open in Firefox.

ps. I don't know much about XML - I prefer simple formats myself.

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else
Amon Tyr
#26 - 2011-12-29 21:08:40 UTC
I cannot seem to be able to install Reverence - I get an error saying the installer cannot find the Python dir in the registry, although I did install Python 2.6 AND 2.7 to their default directories under C:. Using Win 7 64-bit Home Premium. Is there a way to fix this? Thank you.
Entity
X-Factor Industries
Synthetic Existence
#27 - 2011-12-30 02:08:37 UTC
Amon Tyr wrote:
I cannot seem to be able to install Reverence - I get an error saying the installer cannot find the Python dir in the registry, although I did install Python 2.6 AND 2.7 to their default directories under C:. Using Win 7 64-bit Home Premium. Is there a way to fix this? Thank you.


did you install 32 or 64 bit python?

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Amon Tyr
#28 - 2011-12-30 07:24:54 UTC  |  Edited by: Amon Tyr
64-bit for both installations.

Just to be clear, I installed 2.6 after 2.7 did not seem to do the job - didn't install both versions at the same time.

Where in the registry does the Reverence installer check for the path? That may be the issue.
malaire
#29 - 2011-12-30 12:01:52 UTC
Amon Tyr wrote:
64-bit for both installations.

Just to be clear, I installed 2.6 after 2.7 did not seem to do the job - didn't install both versions at the same time.

Where in the registry does the Reverence installer check for the path? That may be the issue.

When I installed 32-bit 2.6 and 64-bit 2.6 at same time, reverence only detected 32-bit version. So maybe try installing 32-bit version also.

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else
Entity
X-Factor Industries
Synthetic Existence
#30 - 2011-12-31 00:12:05 UTC
Amon Tyr wrote:
64-bit for both installations.

Just to be clear, I installed 2.6 after 2.7 did not seem to do the job - didn't install both versions at the same time.

Where in the registry does the Reverence installer check for the path? That may be the issue.


It's the 64 bit that's the problem.
I'm currently unable to produce 64bit binary builds (mostly due to being lazy) so the installers are 32bit-only.
Building from source should work though :)

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Amon Tyr
#31 - 2011-12-31 10:28:37 UTC
Ah, that explains it. Thank you very much for your work! It is an awesome tool. Big smile
Entity
X-Factor Industries
Synthetic Existence
#32 - 2011-12-31 15:44:35 UTC
Hmm, it appears CCP has recently started to marshal instances of classes that only exist in Python 2.7+.

If you're still using 2.6, now would be a good time to upgrade as I'm not sure I will be supporting 2.5 and 2.6 builds much longer.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
6ie
The Kiwis
#33 - 2012-01-18 00:33:01 UTC
Slightly related - Anyone know why after some time (hours) of scanning the market, the eve client stops creating cache files in CachedMethodCalls? (Rather it creates 1 out of every 7 or so scans).
Entity
X-Factor Industries
Synthetic Existence
#34 - 2012-01-18 01:10:41 UTC
6ie wrote:
Slightly related - Anyone know why after some time (hours) of scanning the market, the eve client stops creating cache files in CachedMethodCalls? (Rather it creates 1 out of every 7 or so scans).


Because it's 16 bit hashes and thus tends to collide a lot (god knows why they didn't just use the hash() of the key for these things. I guess it's poor man's cache size limiter).
Check the existing cache files' modification date :)

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
malaire
#35 - 2012-01-18 08:24:03 UTC  |  Edited by: malaire
6ie wrote:
Slightly related - Anyone know why after some time (hours) of scanning the market, the eve client stops creating cache files in CachedMethodCalls? (Rather it creates 1 out of every 7 or so scans).

There seems to be at least 2 limitations for showMarketDetails javascript call.

1) max 1 call per second
2) max N calls in X seconds (or something like that)

If I use quick loop with 40 calls and 2.5 second wait between calls, it works. But if I leave that running repeatedly then after some time showMarketDetails just stops working without any warning about too many calls (i.e. it just doesn't open market details anymore).

Currently my script is using 10 second wait between calls, and 20 second wait between batches of 40 calls or so. This has never stopped working so far even after hours, but might be an overkill. But since this is enough for me I havn't tried finding optimal wait time.

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else
Kaladr
Viziam
Amarr Empire
#36 - 2012-01-19 18:39:39 UTC
malaire wrote:
6ie wrote:
Slightly related - Anyone know why after some time (hours) of scanning the market, the eve client stops creating cache files in CachedMethodCalls? (Rather it creates 1 out of every 7 or so scans).

There seems to be at least 2 limitations for showMarketDetails javascript call.

1) max 1 call per second
2) max N calls in X seconds (or something like that)

If I use quick loop with 40 calls and 2.5 second wait between calls, it works. But if I leave that running repeatedly then after some time showMarketDetails just stops working without any warning about too many calls (i.e. it just doesn't open market details anymore).

Currently my script is using 10 second wait between calls, and 20 second wait between batches of 40 calls or so. This has never stopped working so far even after hours, but might be an overkill. But since this is enough for me I havn't tried finding optimal wait time.


Another technique to refresh the IGB page (window.location = blah works).

Creator of EVE-Central.com, the longest running EVE Market Aggregator
roigon
TURN LEFT
#37 - 2012-01-22 13:51:49 UTC
I use the following javascript to loop over items.

http://paste2.org/p/1879109

Please note that [PLACEHOLDER] get's replaced by python with a list of items I want to loop over, and there's some js in there to create a progress bar, probably has a bug though.

But more or less I just queue up the calls with setTimeout with a 3s interval. Works for me.
roigon
TURN LEFT
#38 - 2012-01-24 11:57:44 UTC
I'm trying to move away from using a the database dump and getting more stuff via reverence. However I'm hitting a wall on this one.

What I want to do is build a tree of the market groups. This for configuration purposes of scanning market items. I can easily enough get this from the database but I'm getting a bit stumped when trying to get this info from reverence.

I've tried various of the bulkdata tables from reverence trying to find the info i need. invgroups, groupsByCategories and typesByMarketGroups. However because of my limited python knowledge I have some trouble interpreting the data I'm getting back and handling it.

I'm trying to parse the data in the following way.
http://paste2.org/p/1881080

This however just ends up running out of memory with a seemingly endless amount of records and groups.

If someone could point me into the right direction I would be quite thankful.
Entity
X-Factor Industries
Synthetic Existence
#39 - 2012-01-25 18:22:56 UTC  |  Edited by: Entity
New version 1.4.2 available! (see OP)

- Fixes keys and values switching places in certain mapping objects.

This issue has actually been there since the beginning of time. It never showed up until recently, because of a subtle change in the representation of certain data in CachedMethodCalls.

I recommend everyone update to this new version to make sure their cache data is decoded correctly.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Entity
X-Factor Industries
Synthetic Existence
#40 - 2012-01-25 18:40:39 UTC
roigon wrote:
This however just ends up running out of memory with a seemingly endless amount of records and groups.

If someone could point me into the right direction I would be quite thankful.


unfortunately FilterRowsets don't have a proper iterator yet (something i'll remedy in a future update).
you should use the iterkeys() method to work around that.

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?