These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information

Author
Callean Drevus
Perkone
Caldari State
#441 - 2012-02-12 14:19:42 UTC
Ok, today, fixed the manufacturing page, which had been unreliable for quite some time. I dropped recursive building for the moment, since it made the results look pretty odd. You now see what you see ingame as well. It also takes into account a few edge cases, most notably recycled items and extra materials which also show up in the base materials. This is not as much a feature, as an overdue fix.

Developer/Creator of EVE Marketeer

Vaerah Vahrokha
Vahrokh Consulting
#442 - 2012-02-13 11:12:13 UTC
Very minimal issue but I really wish to see your site to be uber super duper extra perfect pro!

Basically you put the Z-order (at least for Firefox 10.1) of the mouse over tool tips in the main page above the Z-order of the items found drop down.

Screen shot with red arrow showing it.

The tooltips that go over the drop down are not limited to the one you see, if you hover the mouse over other headings they also appear above it.
Vaerah Vahrokha
Vahrokh Consulting
#443 - 2012-02-13 11:21:14 UTC
Another, less minimal issue I just found.

I entered plex then pressed GO and the website displayed a Could not find 'plex' screen. So far so good.

But now I entered plex in the top search box again, I correctly get the drop down choice but then if I click on the "30 days blah blah blah" entry:

- I expect to go in the PLEX page like it did in the past

- Instead nothing happens and the top search box gets filled by a "29668" number, which I suspect is the item ID. And no PLEX page shown. Now I can try typing "plex" all day long and I only get the number.
6ie
The Kiwis
#444 - 2012-02-13 13:34:58 UTC
The search makes absolutely no sense.

Search: "mega af" and press enter or Go
Returns 50+ items, not in alphabetical, or in "most likely" order, most of which do not even have an "af" in their title.
"Mega Afocal Pulse Maser I" is 12 lines down, almost off the screen.

How it should work: http://eve-marketdata.com/
Search: "mega af"
Drop down of 2 items (the only 2 with "mega af" in their title), allowing you to click the one you want, and it takes you right to that items page.

Even if you get the drop down on EVE Marketeer, and click the item you want - it doesn't take you anywhere.

Oh, and GL going back to modify the text in the search box.
Scrapyard Bob
EVE University
Ivy League
#445 - 2012-02-13 14:27:41 UTC
Vaerah Vahrokha wrote:

- Instead nothing happens and the top search box gets filled by a "29668" number, which I suspect is the item ID. And no PLEX page shown. Now I can try typing "plex" all day long and I only get the number.


Yes, this seems to be a new bug that I hit last night on some other items. However, the search worked if I clicked the "go" button and the ItemID only ended up in the search box if I clicked on the "suggestions" list that shows up under the search box as you type stuff in.

Maybe the search box needs to be smart enough to let us search on item ID numbers? :)
Aineko Macx
#446 - 2012-02-13 14:32:28 UTC  |  Edited by: Aineko Macx
Having played with the API in the last days, more feedback:

- For computing buyHighest5% and sellLowest5% you should do some pre-filtering on the orders. Right now, the numbers are often biased by existing but irrelevant orders like the token region-wide buy order for 1 millions units at 0,01isk. There are many ways to do that. A simplistic approach would be to discard buy orders below say -80% of the highest buy order, or something like the median of orders whose remaining volume is smaller than the initial volume and are not older than x.

- When using the station_info API, the datetime column is missing (it's there for the otherwise identical info API).

- I'd really like to a) query up 25 typeIDs in the history API and b) specify the time range. I'd rather do 210 small queries that way instead of the current 5247 when I update the history, pulling like 300MB of csv data in the process Blink

- You are tracking more items in the DB than just the subset WHERE marketGroupID IS NOT NULL AND published = 1. Why?
Callean Drevus
Perkone
Caldari State
#447 - 2012-02-13 20:47:40 UTC  |  Edited by: Callean Drevus
- Basically you put the Z-order (at least for Firefox 10.1) of the mouse over tool tips in the main page above the Z-order of the items found drop down.

Ah! Correct, I'll fix this soon.

- I entered plex then pressed GO and the website displayed a Could not find 'plex' screen. So far so good.

Umm, yes, it seems I started working on search, then forgot I was doing so, and started implementing different things. I'll fis these issues tonight.

- Maybe the search box needs to be smart enough to let us search on item ID numbers? :)

It should be, unless I broke it. If so, will fix it now :P

- For computing buyHighest5% and sellLowest5% you should do some pre-filtering on the orders. Right now, the numbers are often biased by existing but irrelevant orders like the token region-wide buy order for 1 millions units at 0,01isk. There are many ways to do that. A simplistic approach would be to discard buy orders below say -80% of the highest buy order, or something like the median of orders whose remaining volume is smaller than the initial volume and are not older than x.

There should be a lot of filtering done even before the orders actually get into the database. I've seen indications before that this wasn't correctly happening, but since I haven't received any reports on it thus far I've let it be. Will put checking this on the agenda for asap. A mechanism much like you describe has been implemented, using the highest buy order (or lowest well) as a measure is a smart thing though, and I'll see if I can add that.

I need specifics on the items where it's going wrong though, that will help with my investigation. Can you provide me with region/type combinations?

- When using the station_info API, the datetime column is missing (it's there for the otherwise identical info API).

Hmm, yes, this seems somewhat odd. I'll see if I can't add this in an easy way (due to the calculations being done on orders it's not automatically and representatively there).

- I'd really like to a) query up 25 typeIDs in the history API and b) specify the time range. I'd rather do 210 small queries that way instead of the current 5247 when I update the history, pulling like 300MB of csv data in the process

a is probably not going to happen, as the history queries are quite intensive as it is (biggest table in emk). b is on my list for implementation (well, limit, but date range is probably even better) :) update: with daterange, the history is pretty fast, so multiple items may be possible Smile

- You are tracking more items in the DB than just the subset WHERE marketGroupID IS NOT NULL AND published = 1. Why?

I shouldn't be, it should even be a little less, as I'm not interested in tech 2 BPO's. Where are you getting this number?

The search makes absolutely no sense.

It actually does, it's just smarter than you, and tries it's best to find things with 'mega' and 'af' in it. It didn't give any preferential treatment to the first time though, nor to both items following eachother, which would be a good addition.



UPDATE: Looking back, I'd probably have to agree that the search made little sense. It was broken in that it only took into account the first part of what you were searching for, and didn't even place a high significance on an exact match.

Now, if you match an itemname exactly (something that always happens with the dropdown), you get taken to that items page immediately. It also takes into account the secondary words, and allows you to type in typeid's again.

Oh, and it automatically starts a search if you select something from the dropdown.

I've also added the history API and a filtering possibility to the API page.

Oh! And the z-order has been fixed Roll

Developer/Creator of EVE Marketeer

Vaerah Vahrokha
Vahrokh Consulting
#448 - 2012-02-13 23:42:58 UTC
1) Thanks for all of this effort you are pouring over.

2) If you ever put additional prices sanitizing please make them optional.

There are certain "price spikes" called Pin Bars that are fundamental to every market, they look like "errors" but they are just the buyers / sellers taking liquidity before pushing hard.

Here's an example of PBs of a real market (gold), I marked them with red arrows.

Here is your own website, correctly showing a prominent pin bar for the PLEX market, seen with monthly bars.

Finally, here's your website again, correctly showing another pin bar for the PLEX market, this time with weekly bars.


These elements (expecially on the daily bars) can very easily get lost if you cut data too much and then the graphs become completely useless.


3) A low priority tip: please put a 180 days option in the history candle stick graphs (EvE has it).

4) An improvement for your history graphs: the ability to specify a start (or end) date for the first (or last) bars / dots. So one can ie. analyze 90 days of the market at the end of 2011 instead of being forced to look at today.
6ie
The Kiwis
#449 - 2012-02-14 02:07:44 UTC
"Mega Af".....wait..... drops down 2 options.....click..... BAM! Right to the items page. You Rock :)
Very nice fix on the searching! Thanks a lot.
Aineko Macx
#450 - 2012-02-14 06:13:01 UTC
Callean Drevus wrote:
I need specifics on the items where it's going wrong though, that will help with my investigation. Can you provide me with region/type combinations?

For instance, the buyHighest5 here:
http://www.evemarketeer.com/api/info/26088/csv/10000002

Quote:
I shouldn't be, it should even be a little less, as I'm not interested in tech 2 BPO's. Where are you getting this number?

Here: http://www.evemarketeer.com/api/region/csv/10000002

I track this subset myself, which are all available market items minus BPs, skills, reactions and supercapitals:

SELECT typeID, g.groupID, g.categoryID, typeName FROM invTypes t, invGroups g
WHERE marketGroupID IS NOT NULL
AND t.groupID = g.groupID
AND t.published = 1
AND g.categoryID NOT IN(9, 16, 24)
AND g.groupID NOT IN (30, 659)
Callean Drevus
Perkone
Caldari State
#451 - 2012-02-14 06:56:46 UTC
Vaerah:

2) Price filtering is not optional now at this moment (though if it ain't working either I guess it makes sense I don't get complains :P), but I've long been planning to just mark the suspicious orders, instead of delete them.

3) Good thing about low prio stuff is that it generally only takes a few seconds to do it :)

4) you can still select 720 or 1800 and use the zooming feature, but I agree that setting dates might be more intuitive.

Developer/Creator of EVE Marketeer

Vaerah Vahrokha
Vahrokh Consulting
#452 - 2012-02-14 07:47:39 UTC
Callean Drevus wrote:

4) you can still select 720 or 1800 and use the zooming feature, but I agree that setting dates might be more intuitive.


I noticed that under "trends" we get a nice black empty area.

I would fill it with some thingies:

1) A legend stating "drag to zoom", because it's not exactly obvious that the feature exists.

2) (A button to zoom in)

3) A button to zoom out once you used 1)

4) A button to reset to "original" scale, the one the graph appears when you first see the page.
Maeste Madeveda
The Spawning Pool
#453 - 2012-02-14 09:11:50 UTC
I have tested the T2 reprocessing and it seems not to beat the benefits the reprocesser finder finds. Any info on the calculation method ?
Callean Drevus
Perkone
Caldari State
#454 - 2012-02-14 19:00:02 UTC  |  Edited by: Callean Drevus
Maeste Madeveda wrote:
I have tested the T2 reprocessing and it seems not to beat the benefits the reprocesser finder finds. Any info on the calculation method ?


What exactly did you try to say? Is this good or bad? I noticed that my new calculation method that makes use of the real trade value of items has some flaws :P since people really are stupid enough to buy WAY above price :P

UPDATE: Holy hell, you guys eat 1.8 MB/s by using the site :P good thing bandwidth is relatively cheap.

UPDATE 2: Good! There was indeed something going wrong with the filtering. Most important stuff of all wasn't being applied (well, it was removing all scams, but it wasn't removing ridiculous orders). It should be fixed now (though it will take some time before it has saturated the entire website).

UPDATE 3: BAM! The filtering is working again. Good to see the ridiculous **** is purged once again.

Developer/Creator of EVE Marketeer

Maeste Madeveda
The Spawning Pool
#455 - 2012-02-15 00:41:46 UTC
Uhm magnificent typo. I wanted to write bear, and I wrote beat...

Anyway it was before the new changes so you might fixd the problems. I reprocessed a cheap T2 to know what I cold get back from it and I discovered that you don't get T2 components but mostly minerals. That would be okay if the calculations done by the site were accurate but they were not. So I was wondering how you calculate the reprocessed materials : do you use our skills ? do you know which components are given back ? etc
Callean Drevus
Perkone
Caldari State
#456 - 2012-02-15 09:47:54 UTC
As far as I know, when reprocessing, the base materials, minus a percentage are returned. I've never actually tried reprocessing a carrier to see whether I could get back the capital construction components though. But this is the premise that reprocessing profits are based on.

The new calculation method takes this into account when determining the prices of the items. It also doesn't count lower than one level, which was messing up the previous calculation (if there were things in the extra materials list that were comprised of base materials, it would still count them as base materials you'd get back for reprocessing, which wouldn't happen of course).

The only problem I'm aware of regarding reprocessing stuff now, is that some prices are unrealistically high, but the idiotic part is that I'm now using the average price of the last 30 days (according to history data), which should be pretty accurate.

Developer/Creator of EVE Marketeer

Desmont McCallock
#457 - 2012-02-15 10:26:44 UTC
The data for a reprocessed item can be retrieved from SDE invTypeMaterials table (as Callean must already know).
Indeed a reprocessed carrier returns capital construction components as do all capital ships.
Vaerah Vahrokha
Vahrokh Consulting
#458 - 2012-02-15 22:24:54 UTC
Today I tried creating my own upload group.

I entered a new group name in the group name edit box (duh!)
I then entered one item name and I got a nice pull down with the grep of the item name.

I proceeded for more items till I was satisfied.

But now how do I use this list? I find no buttons to save it nor to upload it (using history price uploads).
Maeste Madeveda
The Spawning Pool
#459 - 2012-02-15 23:12:46 UTC
I second what Vaerah said. Plus I haven't found any way to delete already written lists.

On the subject of processing, you might want to check why the reprocessing values of JF are over 100B. I guess it's because of the prices of capital parts that a dumb someone is putting on the market.

Other than that, you are doing a great job. Have some encouragement.

YAY for Callean

_o/ \o/ \o_
Aineko Macx
#460 - 2012-02-16 06:39:45 UTC
Another idea for filtering the highest buy / lowest sell: Don't select 5% of the available or demanded volume, but rather of the daily traded volume.

Seconding YAY for Callean Cool