These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information

Author
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#461 - 2012-02-16 19:05:37 UTC
Anyone else have problems with the unified uploader not picking anything up, after the patch today? (I've restarted the uploader.)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Scrapyard Bob
EVE University
Ivy League
#462 - 2012-02-16 19:38:31 UTC
Steve Ronuken wrote:
Anyone else have problems with the unified uploader not picking anything up, after the patch today? (I've restarted the uploader.)


Only if you don't shut down the uploader before patching. Every time the EVE client patches, it moves the old cache directory out of the way and creates a new one.

Also, if you run multiple clients off of multiple junction points, be sure to patch all junction points by starting up a client at least once on each junction point to get the client update for that junction.
Scrapyard Bob
EVE University
Ivy League
#463 - 2012-02-17 16:54:16 UTC
Various wish lists:

1) Large POS tower fuel cost per 30-days graph. Take the input costs for the (4) fuel block BPOs, assume ME 40, assume Production Efficiency IV, then graph it out based on the historical trade volume from the 4 market hubs. Large towers consume 1 batch (40 blocks) of fuel per hour.

2) Ore value per m3 graph. Show how ISK/m3 has varied over the past few months/years and which ores are at the top of the ISK/m3 pile. Use the refined mineral costs (assuming zero-waste) and base mineral costs off of the hi-sec market hub trade histories.

3) Ice value per block graph. Show the refined value and/or show the block trading price for the different ice types. You may want to divide it up so that there is a hi-sec chart (only the hi-sec ices) and a null/low chart with the fancier ices (it's harder to get good trade values for the null-sec ice).

4) Total traded ISK volume per market hub / system / region over time for the top (8) stations/systems/regions.
Callean Drevus
Perkone
Caldari State
#464 - 2012-02-17 22:57:58 UTC  |  Edited by: Callean Drevus
Damn, damn, I knew this would happen. I'm gone for a few days, and people are cheering me on and requesting even MORE features :P

That said, I've been away for a while because I have been busy fixing my companies production server, which broke down for one reason or another, that I still haven't been told. All the while, the EMK server was running fine, which made me feel exceptionally stupid. I'd rather have you all angry than my clients, but **** happens I guess :P

As a side effect, both my new production server (at the same place where EMK is hosted) and the EMK server are now costing me about 10 dollars a month more, but also fully back upped every day.

Quote:
Only if you don't shut down the uploader before patching. Every time the EVE client patches, it moves the old cache directory out of the way and creates a new one.

I suddenly understand why EMD has this HUGE warning about this that pops up every time you open the uploader :)

Oh, and will log all feature requests, but probably not work on them for now, as I'm still firefighting Roll

Quote:
I second what Vaerah said. Plus I haven't found any way to delete already written lists.

On the subject of processing, you might want to check why the reprocessing values of JF are over 100B. I guess it's because of the prices of capital parts that a dumb someone is putting on the market.

Ehrm, yeah, the custom list mechanism is quite old, and the interface has changed a bit, plus, I haven't added all the buttons I probably should've added.

To create a custom list, you first enter a group... now wait, I just checked, and I broke it completely when rewriting the uploader interface. I didn't even modify any part of it, but somehow it is broken now. It'll give me an opportunity to make it somewhat nicer that it is now.

Quote:
4) Total traded ISK volume per market hub / system / region over time for the top (8) stations/systems/regions.

I do not think it is possible to do any more than checking this for a region, but it will still be interesting. I had the power to do it for individual orders once (thus also enabling hub/system stats) but I've disabled recording every trade a long time ago. May start it again some time though (still have the function more or less built in).

Developer/Creator of EVE Marketeer

Scrapyard Bob
EVE University
Ivy League
#465 - 2012-02-18 02:39:02 UTC
Callean Drevus wrote:

Quote:
4) Total traded ISK volume per market hub / system / region over time for the top (8) stations/systems/regions.

I do not think it is possible to do any more than checking this for a region, but it will still be interesting. I had the power to do it for individual orders once (thus also enabling hub/system stats) but I've disabled recording every trade a long time ago. May start it again some time though (still have the function more or less built in).


I envision you just summing up all of the market history information for each region into a "total ISK traded per day" table (primary key: region + date). Then graph the total ISK/day for each of those regions. Offer a 5-day moving average and 20-day moving average variant to smooth out bumps and to more clearly show the rise-and-fall of the various trade regions.

Whether you hand calculate which (4 / 8 / 10) regions to show on the graph would be up to you.
Snarf Aldes
University of Caille
Gallente Federation
#466 - 2012-02-18 14:43:10 UTC  |  Edited by: Snarf Aldes
Both on the order info on the site, and the info provided by the api, when i look at Sterile Conduits (id = 2875), i don't see any buy orders with a minimum quantity greater than 1.
There are at least 2 buy orders in in Jita 4-4 at the moment that don't show up (one with 200 and one with 1000 min. qty.)

Is it a bug or a feature?

thnx.

Creator of Eve Addicts

Callean Drevus
Perkone
Caldari State
#467 - 2012-02-18 17:25:39 UTC
Snarf Aldes wrote:
Both on the order info on the site, and the info provided by the api, when i look at Sterile Conduits (id = 2875), i don't see any buy orders with a minimum quantity greater than 1.
There are at least 2 buy orders in in Jita 4-4 at the moment that don't show up (one with 200 and one with 1000 min. qty.)

Is it a bug or a feature?

thnx.


This is a feature. Orders which have a minimum quantity higher than half of the total quantity are filtered, because they are likely scams.

Developer/Creator of EVE Marketeer

Vaerah Vahrokha
Vahrokh Consulting
#468 - 2012-02-19 13:40:59 UTC
Some more things:

Since I cannot create my own upload group (yet) I thought to just click on the relevant items in history mode, start the uploader and have it upload them.

- It did not work, the item ID never appeared in the uploads.

- I went to check for the logs, I can find plenty of error dump logs but no log about the last upload (the stuff you see scrolling in the status tab. Is there a way to have it saved?

- I copy pasted the status window text and then searched for the item ID above but still nothing.


How do I force the uploader to upload history I have for sure? My own cache scraping script based on the same engine gets the updated history.
Callean Drevus
Perkone
Caldari State
#469 - 2012-02-19 14:30:01 UTC  |  Edited by: Callean Drevus
This seems like an issue with updating and keeping the uploader running while doing so. In that case it might not detect the items you are uploading. Otherwise, I have no idea why it is negating your manual items.

Just to make sure, you have selected the history tab in the market view (that is, not the website page, but the ingame market browser) have you?

UPDATE:

For all those of you whom have their own publically accessible endpoints, you can now add these to the EMK website, to allow people to add these preconfigured to the Unified Uploader.

This also means that the 0.6 version of the unified uploader is finally here, with support for automatic update checking, and deleting after x minutes in addition to the previously mentioned feature.

Developer/Creator of EVE Marketeer

Vaerah Vahrokha
Vahrokh Consulting
#470 - 2012-02-19 16:37:11 UTC  |  Edited by: Vaerah Vahrokha
Callean Drevus wrote:
This seems like an issue with updating and keeping the uploader running while doing so. In that case it might not detect the items you are uploading. Otherwise, I have no idea why it is negating your manual items.

Just to make sure, you have selected the history tab in the market view (that is, not the website page, but the ingame market browser) have you?



I could not use manual items, because of the issue I posted some days ago: once created the list I found no way to save it or make it execute.

But since I noticed the uploader sent stuff even with no game running (thus no in game browser open), I tried starting it alone with no game. It uploaded some things but not the items I wanted. They were surely in the cache because I clicked on yearly history in game on each of those items and my scraping script sees these cached items updated.
Callean Drevus
Perkone
Caldari State
#471 - 2012-02-19 16:59:56 UTC
Is it possible your own caching script is deleting files right after they are read? Or locking the files and not freeing them again?

Basically, if you do this:

1. Start uploader
2. Start game
3. Click on item in market window (history or orders)

The uploader should upload. If it isn't doing something along that path, something is wrong.

Version 0.5 (fixed in 0.6) read all already existing cache files on starting (the uploads you saw) and tried to upload those, but 0.6 ignores them, because it led to duplicate uploads.

Developer/Creator of EVE Marketeer

Maeste Madeveda
The Spawning Pool
#472 - 2012-02-19 20:30:09 UTC
The new 0.6 version seems to run fine on my computer.
Vaerah Vahrokha
Vahrokh Consulting
#473 - 2012-02-20 03:05:22 UTC
Callean Drevus wrote:
Is it possible your own caching script is deleting files right after they are read? Or locking the files and not freeing them again?

Basically, if you do this:

1. Start uploader
2. Start game
3. Click on item in market window (history or orders)

The uploader should upload. If it isn't doing something along that path, something is wrong.

Version 0.5 (fixed in 0.6) read all already existing cache files on starting (the uploads you saw) and tried to upload those, but 0.6 ignores them, because it led to duplicate uploads.


My script does not delete any cache file and I only started it *after* I noticed the lack of updates.
I'll keep you informed about how it goes with the new uploader.
Vaerah Vahrokha
Vahrokh Consulting
#474 - 2012-02-20 03:14:10 UTC
Callean, you know we have brainstormed the OHLC graph and export some time ago.

Now it's time to see the fruits.

I have posted a brief intro and tutorial about your web site. It shows how to prepare OHLC data good to be imported on a RL trading platform.

The article is: EvE Marketeer – powerful EvE trading oriented website

There is also a step by step tutorial on how to import the generated files into a standard RL trading platform (so you can physically see the results of your efforts): How to import EvE market data in a trading platform

Finally, there's an actual market analysis and trade example available here: EvE Online – Nitrogen Isotopes – 2012-01-18 – A very profitable trade

A 10B investment resulted in about 2.4B net profit.

This can also be done with my script (almost identical output) but your website makes it way much easier and quicker.
Vaerah Vahrokha
Vahrokh Consulting
#475 - 2012-02-20 16:07:36 UTC  |  Edited by: Vaerah Vahrokha
Issue with uploader 6: it uploaded data once (345 files) then stopped. Now regardless whether I start before or after EvE, nothing gets sent. I have (and always will have) the "delete cache files" option disabled.

Here's a screenshot of history market (I did select "history mode" in the IGB browser) running with the uploader at the top.
As you see no log nothing despite the items I am uploading were last updated yesterday.

If I go in "recent uploads" I see nothing have been uploaded. It worked with uploader 5 (with some issues I posted in the past).
I put the upload key in the Endpoints as well.

All I get in the log files is "Logger started" repeated some times.
The auto detected cache location is correct.
Snarf Aldes
University of Caille
Gallente Federation
#476 - 2012-02-20 18:01:55 UTC
Callean Drevus wrote:

For all those of you whom have their own publically accessible endpoints, you can now add these to the EMK website, to allow people to add these preconfigured to the Unified Uploader.

Cool addition, works like a charm.

Creator of Eve Addicts

Callean Drevus
Perkone
Caldari State
#477 - 2012-02-20 20:41:32 UTC  |  Edited by: Callean Drevus
Vaerah Vahrokha wrote:
As you see no log nothing despite the items I am uploading were last updated yesterday.

If I go in "recent uploads" I see nothing have been uploaded. It worked with uploader 5 (with some issues I posted in the past).
I put the upload key in the Endpoints as well.


Starting from version 0.6, it will not upload anything that is already in the cache when the uploader is started. In addition, the default EVE problem of only updating the item only once every time you start it unless something changes (for history, probably only once a day), probably applies.

Can you tell me whether the steps I outlined above work? You should follow those while you have no application (well, not EVE or the uploader anyway) open.

In addition, you should really delete some of the cache files eventually (or move them.. I forgot to add that option...) or your EVE market seems to get really slow.

I may have to write an essay about the intricacies of those damn cache files :P

Developer/Creator of EVE Marketeer

Vaerah Vahrokha
Vahrokh Consulting
#478 - 2012-02-20 23:42:26 UTC  |  Edited by: Vaerah Vahrokha
Callean Drevus wrote:
As you see no log nothing despite the items I am uploading were last updated yesterday.

Can you tell me whether the steps I outlined above work? You should follow those while you have no application (well, not EVE or the uploader anyway) open.


Tried the steps above very carefully from rebooted computer, no upload starts, the numbers stays at 345 files read like this morning.
Switched between Market data and history tab, nothing happens.

It just says "Welcome to the new EVE Marketeer uploader" and that's it.

I clicked on an item I never clicked before and THEN that item got uploaded.

Then I started the IGB and loaded the upload page.
I selected the minerals, I have not played since before DT, so something HAS to have changed in the history (or not?).

But nope.

I then selected the market orders (only minerals), then pressed the IGB play button and this time it uploaded the stuff. Notice how it uploaded the orders with the "orders in past 24 hours" pane saying 100%.

It does not update the history despite the "History in past 7 days" is 91% (i.e. data is missing)

So, data is missing but not sent? I am sending 1 year of history.
Vaerah Vahrokha
Vahrokh Consulting
#479 - 2012-02-21 16:09:26 UTC  |  Edited by: Vaerah Vahrokha
I ran the uploader again. Today it sent the history up.

This opens to 2 considerations:

1) It won't send current day data (which is ever changing, therefore would always trigger a send).

2) History mode basically is a one run now. Therefore the play button should only run through the selected category (i.e. minerals, PI...) from the first item to the last item once. Once at the end of the list, it should not restart from the beginning of the list like it is doing now and restart the cycle forever.



Ah, I just found a suggestion.

Please add the ability to permalink the graph panes. I.e. if I want to show someone a graph, I'd love to be able to show him either the whole item sheet with the candle bars already visible or maybe even have the ability to just show him the candle bars (ofc with your logo somewhere).

As you can see I am heavily linking and sponsoring your website everywhere both on the EvE forums and on my website, hope you like it.
Callean Drevus
Perkone
Caldari State
#480 - 2012-02-21 18:28:30 UTC
Quote:
1) It won't send current day data (which is ever changing, therefore would always trigger a send).

2) History mode basically is a one run now. Therefore the play button should only run through the selected category (i.e. minerals, PI...) from the first item to the last item once. Once at the end of the list, it should not restart from the beginning of the list like it is doing now and restart the cycle forever.

This seems a plausible explanation. Making the history not cycle is not really a solution though, in my opinion. People will generally leave the uploader running when they aren't doing anything else, so stopping after one cycle wouldn't make much difference, and just annoy people whom would want to keep uploading all the time (though downtime would eventually kill that anyway, I admit).

Still, for others, it might be a good idea to just make a checkbox or separate start button that will start just one cycle of the uploads.

Quote:
Please add the ability to permalink the graph panes.

Yes! This has been in my mind for a long time, but for some reason I constantly forget or postpone doing it...

Quote:
As you can see I am heavily linking and sponsoring your website everywhere both on the EvE forums and on my website, hope you like it.

I like it, but it won't get you preferential support Blink well, maybe a bit, since bad publicity is not the kind I really want :P

Developer/Creator of EVE Marketeer