These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information
Author
Previous Topic Next Topic
Callean Drevus
Perkone
Caldari State
#301 - 2012-01-16 07:12:43 UTC  |  Edited by: Callean Drevus
It seems sensible enough, just wrongP. But I'll take a look at the unified uploader, and fix this. Thanks for the report :)

Developer/Creator of EVE Marketeer
6ie
The Kiwis
#302 - 2012-01-16 16:58:20 UTC
I run 3 eve clients at once, all scanning the market. The cache folder grows to 5,000 files, then EVE starts deleting them.
I think the uploader runs ok when I have 1 client running. Possibly the uploader cant handle 3 clients at once?

Zeropoint reader shows about 99% of the files are being deleted, but either they are not, or they are being missed by the uploader.
Callean Drevus
Perkone
Caldari State
#303 - 2012-01-16 17:19:16 UTC
Can you check the log files in the installation folder and see if you find anything unusual?

Developer/Creator of EVE Marketeer
6ie
The Kiwis
#304 - 2012-01-16 17:49:13 UTC
(Using: Unified 0.4b)
This is the folder that has the 5000 files:
MachoNet\87.237.38.200\303\CachedObjects

All 4 log files in the uploader dir just read:
Logger started
Scrapyard Bob
EVE University
Ivy League
#305 - 2012-01-16 19:31:46 UTC
6ie wrote:
Possibly the uploader cant handle 3 clients at once?


It shouldn't have an issue like that - I used to break 0.3 and 0.4 all the time with running (3) clients. But since 0.4b, haven't had a lick of trouble with the uploader (even with 3 clients, all feeding the same cache folder, or running off of diff cache folders, same region, diff region). So far, 5 days running, 235k files read, 234k files deleted by the Zeropoint reader.

But I'm on Win7 64bit.

I show about 20.4k files in my MachoNet cached objects folder.
Callean Drevus
Perkone
Caldari State
#306 - 2012-01-16 20:20:48 UTC  |  Edited by: Callean Drevus
Ah, yes, the uploader doesn't really care about CachedObjects at all, the only thing it's interested in is CachedMethodCalls, which should be empty.

I take it it is uploading okay otherwise? With the exception of your CachedObjects folder staying full (which the uploader is not having any effect on)?

Also EVE Central might have some trouble keeping up with the uploads you are doing if you're running 3 clients at the same time, it'll just permanently stay at a queue of 50 items, and discard any excess.

@Kaladr: Fixed the URL scheme, will be included in the next iteration of the UU.

UPDATE: For those of you reading this, EMK likes it when you request item information in batches of 25 instead of one at a time :P or even better, if you're going to request them anyway, all the items in the region at the same time (and discard some of the information).

Also, I wonder who the person is that is uploading 4 orders per second with (seemingly) the old uploader, may have to add the user agent string to the uploader too. Would be able to make more sense of the access log that way.

Developer/Creator of EVE Marketeer
6ie
The Kiwis
#307 - 2012-01-16 21:48:57 UTC
Ah ok, yeah i see the files come and go from CachedMethodCalls.
I also see EVE Central: unable to upload in the log
I don't know why CachedObjects gets so many objects - I guess I will just del them periodically.

Is there a way you can code the uploader can not del the cache files for X minutes? I like to import the cache files to Evementat (and eventually to a MYSQL DB with Reverence), but the cache files get deleted by the uploader. (or add an option to no del them at all).
Kaladr
Viziam
Amarr Empire
#308 - 2012-01-16 22:36:00 UTC
Callean Drevus wrote:

Also EVE Central might have some trouble keeping up with the uploads you are doing if you're running 3 clients at the same time, it'll just permanently stay at a queue of 50 items, and discard any excess.

@Kaladr: Fixed the URL scheme, will be included in the next iteration of the UU.


Thanks :)

And I demand of myself to fix the upload performance hole. Twisted

Creator of EVE-Central.com, the longest running EVE Market Aggregator
Callean Drevus
Perkone
Caldari State
#309 - 2012-01-17 06:54:00 UTC
Jup, the next thing I will add to the uploader is the ability to either move or not delete the cache files. It was just the easiest solution to delete them at first :)

Developer/Creator of EVE Marketeer
Endeavour Starfleet
#310 - 2012-01-17 08:45:22 UTC
Please forgive me if you have already answered this but...

Is there a way by chance to develop something in EM that helps you find the best use of LP? I realize that it will mean manually inputting prices for the faction items but I was curious if it is possible to do it with this?
6ie
The Kiwis
#311 - 2012-01-17 15:41:40 UTC
Callean Drevus wrote:
Jup, the next thing I will add to the uploader is the ability to either move or not delete the cache files. It was just the easiest solution to delete them at first :)


Nice, thanks :)
BTW, how are the credits per upload calculated?
Callean Drevus
Perkone
Caldari State
#312 - 2012-01-17 20:09:31 UTC
Damn, the forums ate my post again :S

As far as I know there is no way to retrieve either LP or faction prices (contracts) from the database. That means I could build something that allows you to input your own data, but it'd be kind of pointless. If the data is somewhere in the SSD though, it could be useful.

@6ie: Is mentioned somewhere on the previous page, basically means, the more influence your uploads have on the state of the market, the higher the reward.

Developer/Creator of EVE Marketeer
Scrapyard Bob
EVE University
Ivy League
#313 - 2012-01-18 04:02:10 UTC  |  Edited by: Scrapyard Bob
http://prices.c0rporation.com/

Looks like you have to hand-submit prices for items sold via contracts. Not sure whether viewing a contract causes it to be written out to a cache file.

(Can't think of any other faction price sites off-hand.)
Endeavour Starfleet
#314 - 2012-01-18 06:50:49 UTC
Callean Drevus wrote:
Damn, the forums ate my post again :S

As far as I know there is no way to retrieve either LP or faction prices (contracts) from the database. That means I could build something that allows you to input your own data, but it'd be kind of pointless. If the data is somewhere in the SSD though, it could be useful.

@6ie: Is mentioned somewhere on the previous page, basically means, the more influence your uploads have on the state of the market, the higher the reward.



I would figure an input window that says "Enter sell price" For a group of LP store items of your choosing. It then knows what said items cost in LP isk and then goes to the market DB to find region prices for tags and other items that can be retrieved.

Then it will tell you LP efficiency from those group of prices.

I know it wont be very accurate. Just faster than having a spreadsheet or having to write down the tag prices every time I want to do a rough calculation.
Trin Dainos
#315 - 2012-01-18 11:03:21 UTC
Don´t know if this question has been asked already, but I´d like to know how you get the movement data? Do you just use the historical date you got? But than also canceled orders would be in there, right?
Callean Drevus
Perkone
Caldari State
#316 - 2012-01-18 11:23:07 UTC  |  Edited by: Callean Drevus
The history data that is provided by CCP is based on transactions, so that is a fairly accurate indicator of the movement of the market. I do some more interesting things, like checking whether the average over time is closer to the highest or lowest price to determine whether there are more buy or sell orders being filled.

Today I found out that Jitanomic, the bot driven market information website had the information on loyalty points. I've tried to find it in other places, but wasn't able to find it, which must mean that it isn't normally available. Having no compunctions whatsoever about leeching off a known botter, I proceeded to scrape the website, which is now happily running :P

Hopefully means I will have some information on loyalty points, making only the faction item prices an issue of manual input.

Developer/Creator of EVE Marketeer
Endeavour Starfleet
#317 - 2012-01-18 11:33:19 UTC  |  Edited by: Endeavour Starfleet
Um instead of getting that info from a site you say is run from bots why not ask CCP if there is a way to gather that info from API or something?

And I don't mind putting in the amounts info a box during the calculation if that is needed.

Edit: Seriously how about we just send you the data on the LP items and you can update it. 90 percent of the battle is diving into the market and running the calculations to find isk per lp. Anyone actually using the data should check to make sure the requirements are the same before buying the LP item.

That would allow you to avoid having to use another site's data.
Callean Drevus
Perkone
Caldari State
#318 - 2012-01-18 11:46:39 UTC  |  Edited by: Callean Drevus
The reason for that is pretty simple. I've found no way to get the data from the API, nor from the static data dump. CCP doesn't make a habit of providing data to a single person (as in I've never heard of it happening), so the easiest solution available is getting the data from someone else.

I could ask CCP whether I have missed something, but I estimate they'll answer with 'not possible'.

This saves me both the time and trouble looking for it. That said, will have to verify it is actually accurate, there is a number of websites containing the information already, but they're either not up to date, or inaccurate (entered by players).

UPDATE: Historical Average data added to the API

Developer/Creator of EVE Marketeer
Endeavour Starfleet
#319 - 2012-01-18 12:13:54 UTC
How about you can start the feature with having us input the stuff ourselves while you make a topic in the forum asking if said data is retrievable?

Then maybe you can see if people will send screenshots of the LP page and use that to start filling in a database. If someone wants something added they need to provide a screenshot of the LP store showing the item in question. Tho I will be happy to move around to help fill holes if it is requested.

I personally think it is a hell of alot better than relying on any kind of bot to do this. And community involvement will help grow your site in my opinion.


If it falls flat on its face you could always go back to your original plan right?
Endeavour Starfleet
#320 - 2012-01-18 12:26:21 UTC
Wanted to show an example of what I was talking about

http://i.imgur.com/tz5ZV.jpg That was done in about 30 seconds copy pasta into paint with a crop. What I can do in gimp is put one on top of the other to get a big image that spans half or all the LP store at once. Then use extreme JPEG compression to keep it small.