These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information

Author
Callean Drevus
Perkone
Caldari State
#141 - 2011-12-04 18:46:54 UTC
Ah, awesome, this is the new datadump at work.

It is fixed, and the other problem is fixed as well. I do not show specific rat bounties anymore. This data was not in the SDD anyway, and scraping them from the EVE cache with every database update is going to be obnoxious, and even then it will be inaccurate.

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#142 - 2011-12-04 19:09:51 UTC
I think the rat bounties were correct, but the totals are not, that I am sure of.
Callean Drevus
Perkone
Caldari State
#143 - 2011-12-04 19:33:04 UTC  |  Edited by: Callean Drevus
Well, the rat bounties were off even when I implemented it, that's why I put the +/- behind the value. It might be that the totals of rat bounties are off too, but I think those are generally correct (unless the API gave me errors).

And I added another feature. You are now able to total your buy and sell orders in the transaction history screen, making for some interesting view of how much you earned on a specific kind of item. It seems PLEXes are one of my major sources of income.

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#144 - 2011-12-04 21:31:07 UTC  |  Edited by: Controller Vrelk
Callean Drevus wrote:
Well, the rat bounties were off even when I implemented it, that's why I put the +/- behind the value. It might be that the totals of rat bounties are off too, but I think those are generally correct (unless the API gave me errors).

And I added another feature. You are now able to total your buy and sell orders in the transaction history screen, making for some interesting view of how much you earned on a specific kind of item. It seems PLEXes are one of my major sources of income.

Well something is wrong. Since I rat in null a lot I have gotten to know about how much the rat bounties are. They seemed right on the site, but the image I linked on the previous page shows how much it is off, and the my calculated amount seems right, but your amount was about 20mill too high. If view the last 90 days the number seems a few hundred mill too high, so I know something is wrong. In my opinion, might as well leave the bounties in there as they seemed accurate. Just from looking at your total, it seems about double what it should be.

Also, when will the uploader be fixed? I have been uploading with 1 client for about 2 hours now and just noticed the uploader stopped processing them at 1081. (EMD enabled, EC disabled)
Scrapyard Bob
EVE University
Ivy League
#145 - 2011-12-05 04:55:53 UTC
http://www.evemarketeer.com/item/info/coolant

I see there is now a "top 5 regions buy/sell" box.

How is that calculate? Is it dropping regions that are more then 7-14 days out of date?
Scrapyard Bob
EVE University
Ivy League
#146 - 2011-12-06 18:39:55 UTC
Getting strange results from the EMK API feed.

http://www.evemarketeer.com/api/info/37_40_36_11399_38_35_34_39/xml/10000030/sell_lowest5

- This is only returning 7 rows, with the price for Tritanium (34) completely missing.
- Also happens in The Forge region.
Callean Drevus
Perkone
Caldari State
#147 - 2011-12-06 22:40:27 UTC  |  Edited by: Callean Drevus
Scrapyard Bob wrote:
Getting strange results from the EMK API feed.

http://www.evemarketeer.com/api/info/37_40_36_11399_38_35_34_39/xml/10000030/sell_lowest5

- This is only returning 7 rows, with the price for Tritanium (34) completely missing.
- Also happens in The Forge region.


Both seem fine to me. Are you still experiencing the issue? I really cannot imagine this happening. The row would have to be deleted from my summary table, and I never actually delete stuff from there unless it's replaced right away.

I'll see if there is any reason this could be happening.

As for the calculation. No, it simply takes the regions that have the most orders, though I imagine it would be a good idea to take the top 5 regions that with the most orders that are not out of date. I'll add that to the calculation.

In regards to bounties. If you put it like that, I'm a whole lot more inclined to think something is wrong with the way I retrieve data from my tables, which could cause a problem. I'll still have to export the bounty prices again though (which I will be doing eventually, but for now I'll first look into the correctness of the totals).

The uploader is a bit of problem right now, as my computer has crashed making me lose the source code of the uploader... which means I'll have to rebuild it. Which is good in the sense that it will probably be without errors, but bad in the sense that it's going to take some time. If you were experiencing no issues with a previous version of the uploader and have disabled EVE Central uploading anyway, you could consider downgrading to a previous version.

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#148 - 2011-12-06 22:56:20 UTC
Alright. It's not version control, but have you considered using dropbox? That would save you the hassle in the future for losing things, instant backup and I think it keeps the past 5 versions of a file or something. I only mention that because it is just something you could install and then forget about it.
Callean Drevus
Perkone
Caldari State
#149 - 2011-12-06 23:58:49 UTC  |  Edited by: Callean Drevus
I am actually using jungledisk and it is making me very happy. It was the reason I couldn't care less that my harddisks suddenly stopped (that, and all my projects being securely in an offsite version control system, which is a backup by default).

The problem was that it didn't backup my downloads folder (which for the sanity of the backup solution, is a good thing) and the uploader source was in that one folder for some reason, so it's vaporized. But don't worry, as long as EVE Marketdata still has their source I can at least start off a decent basis.

Oh, and I also use dropbox, just not for backups ;)

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#150 - 2011-12-07 00:42:00 UTC
Callean Drevus wrote:
I am actually using jungledisk and .............


Alright, thats good. As for the new uploader, maybe put the field for EMK at the top, it is your uploader after all.

Second suggestion, add some way to the uploader to decrease the time. I know its there for people with slower connections so they don't slow it down and miss 1/2 the items, but for me it loads in about 1.5-2 seconds. (Remember, max of 1 JS call per second with the IGB)
Callean Drevus
Perkone
Caldari State
#151 - 2011-12-07 09:06:49 UTC  |  Edited by: Callean Drevus
Actually, the minimum for the market call is 3 seconds, which is why it's at that value by default :P you could make it slower, but you can't make it any faster (unless CCP has changed something without my notice).

The new uploader should follow an alltogether different format of entering that data.

Developer/Creator of EVE Marketeer

Maeste Madeveda
The Spawning Pool
#152 - 2011-12-08 12:40:27 UTC
Hello Callean,

I was starting to use your website because it the shiniest one and I love shiny stuff. However as I wished to register, the site denied me the entry without sending me back any error message. Am I not good enough for your upload club ? :(

Win 7 x64
Chrome 13.0.767.1
Turhan Bey
EVE University
Ivy League
#153 - 2011-12-08 19:56:59 UTC  |  Edited by: Turhan Bey
I've just noticed that I'm receiving a lot of:

Debug: upload: uploading to http://www.evemarketeer.com/api/upload
Debug: upload: uploader.send = 200, OK
Debug: upload: response=0: The upload cannot come from the future
Error: Upload unknown problem: 0: The upload cannot come from the future
Debug: returned from upload function

I'm definitely not doing anything that allows me to see future market values, but I sure wish that I could. :)

EDIT: BTW, it doesn't happen for all items. There doesn't seem to be a pattern to exactly which items as far as I can tell. Some example item ID's that caused it in the Verge Vendor region, in case it helps:

20843
13239
30078
32108
28471
21238
22577
Zaepho
Goosewarms
Tactical Hunters and Gatherers
#154 - 2011-12-08 20:08:24 UTC
Turhan Bey wrote:
I've just noticed that I'm receiving a lot of:

Debug: upload: uploading to http://www.evemarketeer.com/api/upload
Debug: upload: uploader.send = 200, OK
Debug: upload: response=0: The upload cannot come from the future
Error: Upload unknown problem: 0: The upload cannot come from the future
Debug: returned from upload function


Possibly clock skew by a few seconds or something along those lines?
Callean Drevus
Perkone
Caldari State
#155 - 2011-12-08 22:21:29 UTC  |  Edited by: Callean Drevus
Maeste, I've just send you an EVE Mail, registration should be working again. It was indeed broken for a few days, but people were very fast to point it out :)

The only reason for the values to come from the future is indeed if your system clock is by some reason ahead of the EVE clock. This has been built in mostly to prevent people from sending illegal data that will always be seen as the most recent. I might relax the requirement a bit so as not to fall over a few seconds difference :)

Developer/Creator of EVE Marketeer

Zaepho
Goosewarms
Tactical Hunters and Gatherers
#156 - 2011-12-08 22:49:52 UTC
Callean Drevus wrote:

The only reason for the values to come from the future is indeed if your system clock is by some reason ahead of the EVE clock. This has been built in mostly to prevent people from sending illegal data that will always be seen as the most recent. I might relax the requirement a bit so as not to fall over a few seconds difference :)

I don't suppose you have a log of these skew values to make an intelligent decision of how far off is commonly acceptable? I would think about 5 or 10 seconds should be MORE than plenty but I'm confident my machines talk to a decent NTP server on a regular basis :)
Turhan Bey
EVE University
Ivy League
#157 - 2011-12-08 23:38:59 UTC
Likewise, I use NTP so I shouldn't be much more than a second or two different. Unless of course the EVE servers don't and they are drifting. I agree that 10s or so should be sufficient to deter false data but still account for sync issues.
Grimdahl
Aliastra
Gallente Federation
#158 - 2011-12-09 15:41:37 UTC
Is there any order filtering applied eg. drop order with highest price?

Reason for my question is: I compared market order data from Eve-Central and Eve-Marketeer and found 1 order missing
(Minmatar shuttle : sell order 888.888.888 ISK in Jita).

Callean Drevus
Perkone
Caldari State
#159 - 2011-12-09 23:22:07 UTC  |  Edited by: Callean Drevus
A lot of order filtering is applied. It drops all outrageous orders.

That is, I do not think you would mind the 888.888.888 shuttle being dropped from the results Blink

Developer/Creator of EVE Marketeer

Turhan Bey
EVE University
Ivy League
#160 - 2011-12-10 20:35:27 UTC
Minor request: Upon opening the EMK Uploader window, please don't make the "EVE Marketdata Upload ID" field selected by default. Any accidental typing wipes out the field.

I've gotten into the habit of clicking in the Log field after every launch. But it would be nice if that were unnecessary.