EVE Technology Lab

It would pretty much destroy the inter-regional markets and we'd all end up as slaves to Jita prices. (The only variation would be "how far away are you from Jita".)

Some information about cross-border prices is good, especially if it's not always accurate or timely. Too much information ends up being a bad thing. One of the "interesting" things about the EVE economic market is that there are lots and lots of niches where someone who is willing to login an alt at both locations can profit from imperfect information flow.

(The reason that I support services like EVE-Central, EVE-MarketData and EVEMarketeer is because they are imperfect, always a few hours behind, but they provide a way that I can get reasonably good estimates of market prices on a daily basis for my spreadsheets.)

I'd love to see the other websites pick this up and run with it as their preferred uploader as well! It's going to take a combined effort between the sites to get the best possible data from all users. Also this will allow for the maintenance of the upload tool be shared between all of the sites.

50% O_o... I never noticed that when testing (well, no noticable slowdowns, so I didn't look ;) ). An EVE Marketeer upload key is required for uploading to EVE marketeer, otherwise I believe it just skips uploading to EMK but happily uploads to EVE Central at least.

Might be a good idea to add automatic registration so the entry barrier is not too high.

What version of windows (32bit or 64 bit) are you running this on? And what are the other specifications of your computer? Since I do not get the 50% cpu problem at all.

OK, today I'm seeing a different pattern from yesterday. Yesterday, the uploader was constantly at 42-48% CPU. Today I'm seeing it jump around from single to digits to 38%, with a figure in the 30's appearing roughly every 1-1.5 seconds.

I've created an Eve Marketeer account and entered my upload key but that seems to have no effect on CPU usage. At least jumping around on the CPU seems to be slowing down the computer less than yesterday. I'm going to try logging on to Eve and checking some market data to see what performance is like.

I'm eagerly waiting for the update on your site because I should rapidly make it into your top 10 uploaders.

Probably lacking a good sleep() in the main if there's no work to be done. it's likely cycling the main loop as fast as possible hence consuming an entire CPU Core hence the 50% CPU utilization (i'm assuming you're on a dual core or HT box here of course)

I would expect one core to go to 100% but it could be flopping back and forth between them but not running on both simultaneously to result in the 50%. I could also be talking out of my hind parts and completely wrong. It's a theory though!

So long as all three sights were interested in accepting data in the same format, it wouldn't be terribly difficult to have one uploader client that transmitted all data to a central hub, and then that hub did some very basic checking on whether the data was new/unique, and then forwarded the data on to all other market sites.

In terms of engineering this solution? Well, IRL I'm a Unix System Admin...so...

The primary upload receiver server would have Solaris 11 Express installed on it. Why? ZFS Data deduplication and ZFS Compression (dedup is done before compression, and de-duped blocks are not re-compressed). It is absolutely amazing with stuff like this.
-The server would have a daemon running on it with a (relatively) simple design and purpose:
1. Install a hook in the OS so that it notifies you of a new file being written (or every 1-2 seconds if file write counts are high and processing can keep up, and there would be logic to automatically switch between these two modes).
2. Use OS utilities (cksum should be sufficient) to check whether the file has already been received and uploaded in the last hour.
3. Attempt to transmit any new data to all downstream targets (eve central, eve markets, etc)
4. Remove all files that were successfully uploaded to all downstream targets.
5. Keep track of transmit queue lengths for each site (writing this memory to disk every few seconds, similar to a SQL replay log). If eve-central is only given 300 MB of pending upload queue, the oldest data in the queue is thrown out. This is to prevent an unintentional DOS in the event that one of the downstream targets is offline or unable to accept new data for some reason.
6. Continue to try and upload to downstream targets every few seconds until successful or queue ejection occurs.

As for why no one does it? Well, cost is one thing. I mean, the daemon could also log how much was sent to who and issue a bill accordingly, but who wants to pay for something they don't explicitly admin? That requires a *lot* of trust. Also, who wants to manage a Solaris 11 box? I mean, Solaris is a bit of a pain in the ass, especially when it comes to package management / patching. But the ZFS gains are *well* worth it in this case.

Additionally, right now there are only two ways to read cache files. libevecache, written in C++ and 100% undocumented (go figure the function calls out for yourself! hah!) or Reverence, written in Python, which means your app needs to either be written in Python, or you have to juggle python code in a C environment, which CCP can tell you is...messy.

A C cache reader, that did *not* require python, scons, or anything but GCC / MSVCC / ICC would go a long way to rectifying this as it would be build able for any 32 or 64 bit OS. People say C isn't portable...lololololol. Unless you have to use in-line assembly, or third party libraries (which for simple things like this, you should be using only 'libraries' that exist in .c and .h files), its the most portable language on the planet, you just have to recompile. :)

Anyway...chest-thumping about my favorite language aside, the reasons why it hasn't been done basically boil down to no one making an easy to use cache reader that *just works*. Making a client that everyone trusts to use. Making a client that elegantly accommodates multiple installs, and also automatically adapts to patch changes (IE: it finds the correct cachedmethodcalls folder automatically).

So yea, the reasons it hasn't been done, in my eyes, are largely technical. It is just plain annoying, and with no real world profit involved, and very unlikely in-game profits...why bother going through all the trouble of doing all that work, just so people can try to make money trading? They want to make money too, where's their cut? 50% technical, 50% economic.

Is Eve Marketeer processing data right now? I'm uploading but...

1. My upload count isn't rising
2. The stats page isn't showing uploads for any region in the past 24 hours
3. The upload backlog is showing almost 68000.

Does this mean it's accepting uploads into a queue but not processing that queue?