EVE Technology Lab

Yeah, no cache file = nothing to grab.

Some possibly cheap fixes:

- When a type ID + region pair is handed out, update the record with a "last handed out" time stamp.

- Keep track of the last time that a particular typeID+region pair has gotten data. (Which you already do.)

1) With the presence of a "last handed out" time stamp, you could pull a list of stuff that needs checking, but exclude anything that was handed out in the previous N minutes (30 minutes might be good). That way if you have (2) clients checking a particular region, both will get slightly different lists to work from. This method does start to break down once you have (20+) clients all checking the same region at the same time, but that's a good problem to have.

2) In addition you might choose to add a few minutes/hours to the "last handed out" time stamp to delay checking them again. For example, if the "last update time" > 7 days, then push the "last handed out" time stamp 2.56 hours into the future. If the "last update time" > 14 days, then push the "last handed out time" 3.23 hours into the future.

"last handed out" = (now) + cube root of (# hours since last update / 10)

cube root (24/10) = 1.34
cube root (168/10) = 2.56
cube root (336/10) = 3.23
cube root (720/10) = 4.16
cube root(2160/10) = 6.00

(You may decide to divide by 30 or 50 instead of 10, to shorten up the last handed out bump.)

It's mostly to deal with the issue of things like the Deadeyes where if you haven't seen it in a month or three months, then it's probably not worth checking for near as often. Depending on what factors you use as the divisor, you'd still be handing it out a few times per day, just not every single time. As long as the different sites (such as EMD/EC) do their algorithms slightly differently, it shouldn't be a weakness.

And there's still the issue where the EMK scanner page "breaks" after 2-3 hours and stops opening up the market window, even though the page is still refreshing. My guess is that you're up against some sort of bug in the web browser, or you're generating requests slightly faster then the 3 second rule. I'm still using eve-marketdata's market scanner page whenever I do my overnight updates.

Yes, I noticed the same BPO trouble yesterday. And due always trying the things that have been uploaded last, the uploader just jams when there are 50 items that will never be uploaded at all. Today I'll take a look at the uploading page and ways in which it can be improved.

UPDATE: The uploader page has gotten a tiny overhaul, now loading the new items by way of AJAX, and having an upload interval of 3.5 seconds, to rule out any troubles with the browser. I hope this will make it more stable.

It tends to crash sooner if you leave the window open then if you minimize it (not fully verified, but a working hypothesis).

EVE Marketeer endpoint log:

Traceback (most recent call last):
File "Classes.pyo", line 423, in run
File "UploaderFrame.pyo", line 119, in log
File "wx\_controls.pyo", line 1850, in AppendText
PyAssertionError: C++ assertion "m_count == -1 || m_count == -2" failed at ..\..\src\msw\textctrl.cpp(140) in UpdatesCountFilter::UpdatesCountFilter(): wrong initial m_updatesCount value

EMD endpoint log:

Traceback (most recent call last):
File "Classes.pyo", line 423, in run
File "UploaderFrame.pyo", line 122, in log
File "wx\_controls.pyo", line 1818, in Remove
PyAssertionError: C++ assertion "m_count == -1 || m_count == -2" failed at ..\..\src\msw\textctrl.cpp(140) in UpdatesCountFilter::UpdatesCountFilter(): wrong initial m_updatesCount value

EC endpoint log:

Traceback (most recent call last):
File "Classes.pyo", line 423, in run
File "UploaderFrame.pyo", line 119, in log
File "wx\_controls.pyo", line 1850, in AppendText
PyAssertionError: C++ assertion "m_count == -1 || m_count == -2" failed at ..\..\src\msw\textctrl.cpp(140) in UpdatesCountFilter::UpdatesCountFilter(): wrong initial m_updatesCount value

file reader log - nothing.

I know, I've been having these messages for the better part of the day, and my uploader still hasn't crashed...

What does it do? Does it simply hang? Or does it seem to not do anything anymore (but still being responsive)?

Ah, yes, I've seen it, and it's been lodged firmly in the back of my mind since then :P

As to the not uploading anymore issue, that is probably (as I've found after a day of uploading) because there is nothing for the uploader to upload anymore. After cycling trought the entire market, the uploader starts anew, but if the same EVE client is running, and nothing has changed in the market information, the cache file isn't rewritten, thus there is nothing for the uploader to do.
To check this, your best bet would be to manually lookup Tritanium and see if the uploader responds to that, if it doesn't, try buying one unit of Trit and see if it works then (I'll refund it ). If it doesn't, then something is very certainly broken.

Of course, it could be that it suddenly starts uploading again when you restart the upload client, which is a practical guarantee of something going wrong.

It just keeps crashing on Win7 64bit - minimized, not minimized. I'm running 4 clients at it at the same time, so it's under a bit of stress. But it never uses much CPU and stays with a peak working set of about 39MB. It can run for an hour and not crash, or crash after 5 minutes, no set pattern - which means it's probably reacting to either a particular item, or a network condition when talking to the end point servers.

I can't tell whether the errors are related to the crash or just background noise, but it's the only thing showing up in the log files.

Not sure then - will do a fresh install folder and see what happens.

When I do my uploads overnight, I run multiple clients, all from the same cache directory. Sometimes 2 in the same region.

The new upload page does seem to be much better behaved and doesn't stop working like the old one did. (Nice "Christmas" theme *grin*)

Mine paused uploading today, new error message:

Traceback (most recent call last):
File "Classes.pyo", line 427, in run
AttributeError: 'NoneType' object has no attribute 'find'

Also, I finally got the EMK uploader to fight over a cache file with the EMD uploader and got this in the file reader log file:

Traceback (most recent call last):
File "Classes.pyo", line 310, in run
File "Classes.pyo", line 339, in addNewFiles
IOError: [Errno 13] Permission denied: 'C:\\Users\\ScrapyardBob\\AppData\\Local\\CCP\\EVE\\c_program_files_(x86)_ccp_eve31024_tranquility\\cache\\MachoNet\\87.237.38.200\\301\\CachedMethodCalls\\7d4d.cache'

Looks like it might need some tuning. (I often switch between using eve-marketdata's scanner page, eve-central's page, and EMK's page through the week.)

Thinks about the numbers... There are 6922 possible market items, 3.5 seconds per item. In an ideal world, that means you can scan the market with a single alt in the space of 6.73 hours (5.77 hours w/ the 3 sec version, but that breaks). If we have (2) bits of information (last upload, last handed out), then we need to penalize items that haven't updated recently and which have also been handed out recently. Logic dictates that you should create a 3rd field to handle this desired result and call it "next possible queue time", which makes the market scanner's page a lot simpler.

LUT = last upload time stamp (defaults to something like the start of the epoch, or something like January 2000)
LHO = last handed out time
NQT = next queue time (defaults to start of epoch)

NQT should be updated when either an item is handed out, or a successful upload arrives.

LHO event -> set NQT to current time + cuberoot(# hours since LUT / 25) + random(1800 seconds)
- So as items are handed out, they get pushed down deeper into the stack, which will bubble other items up to the top.
- Adds a bit of randomness to move things up/down by a maximum 15 minutes
- The "25" divisor gives good results for items which are only seen once a year (bumps them by 7.05 hours)

LUT event (successful upload) -> set NQT to current time + 8 hours + random(4 hours)
- When an item is successfully uploaded, it needs to move down towards the bottom of the stack
- Since the 6922*3.5 sec = 6.73 hours, the magic number is 8 hours plus up to 4 hours

Now for the reason that we use a NQT field instead of looking at the LHO/LUT fields when figuring out what the market scanner should hand out next. When the market scanner page needs to pull 20 items to hand out, it should query the item field, sorting on NQT in ascending order, top 20 items. This way, you always get 20 items, even if the NQT values are hours/days into the future.