These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information
Author
Previous Topic Next Topic
Scrapyard Bob
EVE University
Ivy League
#361 - 2012-01-31 04:12:27 UTC
In fact, if you run any of the "upload" tools - you can mix and match with any of the "in-game browser market scanner pages" (or write your own page).

For a very long time, I ran the EMD's uploader, but used either EMK or the EveCentral's market scanner page. Or ran the unified uploader, then used any of EMD's / EMK's or EC's market scanning page.

For fun, I tried running the unified uploader under WINE 1.2 on Ubuntu.

Traceback (most recent call last):

File "emkUploader.py", line 5, in (module)

File "zipextimporter.pyo", line 82, in load_module

File "wx\__init__.pyo", line 45, in (module)

File "zipextimporter.pyo", line 82, in load_module

File "wx\_core.pyo", line 4, in (module)

File "zipextimporter.pyo", line 98, in load_module

ImportError: MemoryLoadLibrary failed loading wx\_core_.pyd

(Since I don't run EVE on Linux, I don't plan on troubleshooting this any deeper.)
Callean Drevus
Perkone
Caldari State
#362 - 2012-01-31 07:01:03 UTC
Aw poop, and here I was thinking it should work fine :P

Back to the drawing board I guess :)

Developer/Creator of EVE Marketeer
Endeavour Starfleet
#363 - 2012-01-31 11:06:54 UTC
Any news on the LP store stuff? I am still ready to enter data.
Vaerah Vahrokha
Vahrokh Consulting
#364 - 2012-01-31 11:14:09 UTC
I found out some areas of possible usability improvement.


1) Main window => search edit box.

When you have an exact match, the "browse" button should act as "show" and directly show the commodity.
As of now when I open the website I come with an item name already filled-in (so, no drop down for completion that directly brings to the commodity sheet). So I have to either delete some part of that name to show the drop down or mistakenly press browse believing the website will look up that item but instead it shows a general search window.


2) Prices exports

By default when you export a market history you get it labelled as "xml.txt", "json", "csv.txt".

The guys using the exports, on the other hand, do it periodically and with method.

Since I clearly see the information gets passed in the URL, what about making a very "batch friendly" export file name like this:

From current

http://www.evemarketeer.com/api/ohlc/10000002/17888/csv

generating a plain "csv.txt" file name in the Save As... window (tried with Firefox only).


To

http://www.evemarketeer.com/api/ohlc/10000002/17888/csv

String splitting that path and activating the following pseudo procedure:

a) Store current date in $CURRENT_DATE. Important: date format must be YYYY-MM-DD
b) Lookup 10000002 => $REGION = "The Forge"
c) Lookup 17888 => $ITEM_NAME = "Nitrogen Isotopes"
d) $EXTENSION = "csv"

Propose end user this file name:

$CURRENT_DATE . $REGION . $ITEM_NAME . '.' . $EXTENSION

Example of resulting file name:

2012-01-31_The Forge_Nitrogen Isotopes.csv



3) Price history duration

I noticed that you can query back to a maximum of 1800 days ago.
Do you actually delete data older than that?

Otherwise I'd gladly spend credits / ISK to have the option of downloading the full stored history (hint).

EvE Online portal | EvE markets tutorials
Nebu Retski
Lead Farmers
#365 - 2012-01-31 12:26:58 UTC  |  Edited by: Nebu Retski
Scrapyard Bob wrote:
For fun, I tried running the unified uploader under WINE 1.2 on Ubuntu.


WINE 1.2 is really old. You better update to the latest wine version 1.3.3*, it is "not stable" though, but stable enough to do most things. Pretty much "most" people use one of the 1.3.2*+ versions to run EVE. So Callean Drevus, no need to get back to the drawing board yet based on the errors Scrapyard Bob got.

Now I properly tried to run the uploader again and I noticed why it didn't work at all before. After unzipping unifieduploader-0.*.zip, the contents of the lib folder remained zipped. When I unzipped the contents of the lib folder I could run the uploader.exe, however it still doesn't work properly. I get the GUI, but I also get the following error immediatly in the terminal:

Debug: src/helpers.cpp(140): 'CreateActCtx' failed with error 0x00000103 (no more data available).

I went in the config.ini file to add my upload key:

endpoint_0_uploadkey = *************

This however didn't remove the error yet. I also noticed that the path you added for the installation folder seems to be hardcoded to use the windows style (i.e. c:\bla\bla\CCP\EVE\...) while this will not work on *nix type OS's. I'm not sure if wine actually translates it properly or not, but to make code crossplatform compatible there is a function in Python that handles the path style depending on the OS -> os.path

Tonight I will check at home if I can get any further because I then I will have a running eve client. I'll let you know how it goes.
Maeste Madeveda
The Spawning Pool
#366 - 2012-01-31 15:05:07 UTC
Vaerah Vahrokha wrote:

2) Prices exports

By default when you export a market history you get it labelled as "xml.txt", "json", "csv.txt".

The guys using the exports, on the other hand, do it periodically and with method.

Since I clearly see the information gets passed in the URL, what about making a very "batch friendly" export file name like this:

From current

http://www.evemarketeer.com/api/ohlc/10000002/17888/csv

generating a plain "csv.txt" file name in the Save As... window (tried with Firefox only).


To

http://www.evemarketeer.com/api/ohlc/10000002/17888/csv

String splitting that path and activating the following pseudo procedure:

a) Store current date in $CURRENT_DATE. Important: date format must be YYYY-MM-DD
b) Lookup 10000002 => $REGION = "The Forge"
c) Lookup 17888 => $ITEM_NAME = "Nitrogen Isotopes"
d) $EXTENSION = "csv"

Propose end user this file name:

$CURRENT_DATE . $REGION . $ITEM_NAME . '.' . $EXTENSION

Example of resulting file name:

2012-01-31_The Forge_Nitrogen Isotopes.csv

I have mixed feelings on this proposition. On one hand it is easier for humans to identify the items so it's nice to have if you look after your exports, on the other hand most of the hardcore data pullers have their own database, their own pulling automaton and so on. Introducing names introduces also tedious coding as in most cases you will be subject to case related bugs.

The date could be a valuable information but you can pull it from the csv in itself as well as using your own system date to add the date, as such it's not a vital information.
Callean Drevus
Perkone
Caldari State
#367 - 2012-01-31 18:26:35 UTC  |  Edited by: Callean Drevus
@VV

1. The think you propose would not really be an usablity improvement, since there's almost always an exact match in the searchfield. There would be no option to browse anymore if I did that. That said, just clicking the search field, not deleting anything, and pressing enter should do the trick.

If anyone agrees with VV, please tell me, because this may just be my misconception as the one who built it.

2. It should be possible to give the downloaded data a different automatic filename than the URL, if that is possible, this is a very plausible and useful suggestion. Otherwise, because this API is generally used by programmers, I do not think I will change this, since it would require me to do a lot of lookups in tables (for the names) that would make the API slower, and I want that to be as fast as possible.

In that case, it might be possible to duplicate the same functionality with a different call, which shouldn't be a problem.

3. Nah, I don't delete data, but the data goes only back 2 years or so, so I should be safe for a long while yet ;) The full stored history is already available (history API call). If you know a place where I can get history that goes back further, please let me know.

@Nebu Retski: The client cache URL is actually not in the config file at all. When you start the client for the first time, the cache location is automatically determined (and saved to the config). The string you saw was thus the URL that was automatically determined.

I'm very interested to hear whether it works.

@Endeavour Starfleet: Yes! I've gotten a reply to my petition, stating that I am free to use the data (or rather, that they cannot forbid me to use it :P), but that they cannot guarantee the accuracy of it. This means that the nature of the trouble is going to change to checking whether the data is correct. If it is not, we'll have to manually enter the stuff anyway. Since the query to get the information is nearly ready, I'll have this function up soon :)

Developer/Creator of EVE Marketeer
Vaerah Vahrokha
Vahrokh Consulting
#368 - 2012-01-31 20:07:30 UTC
Callean Drevus wrote:
@VV

1. The think you propose would not really be an usablity improvement, since there's almost always an exact match in the searchfield. There would be no option to browse anymore if I did that. That said, just clicking the search field, not deleting anything, and pressing enter should do the trick.

If anyone agrees with VV, please tell me, because this may just be my misconception as the one who built it.


Steps to reproduce in Firefox latest version in Win 7 64 bit.

1) Start browser => history => reopen last session

or

1a) Add new tab, then go history => reopen the EvE Marketeer page


2) The page shows up with the item already typed in

3) Clicking on it will only select all words of the item name (no drop down)

4) Clicking again will select one word of the item name (no drop down)

EvE Online portal | EvE markets tutorials
Callean Drevus
Perkone
Caldari State
#369 - 2012-01-31 20:41:48 UTC  |  Edited by: Callean Drevus
Yes, I understand what is happening, no trouble there :P

Or was this for other people? Blink

Developer/Creator of EVE Marketeer
Nebu Retski
Lead Farmers
#370 - 2012-01-31 21:54:19 UTC  |  Edited by: Nebu Retski
Well it seems that it is working (after having the uploader detect my installation folder) when I have a client open and scanning in the IGB.

At least I see the following type of information rolling in the status tab:
EVE Central: 2012-01-31 21:47:25: Uploaded 8253 in 10000042 with 3 orders
EVE Marketdata: 2012-01-31 21:47:29: Uploaded 31802 in 10000042 with 24 orders
EVE Central: 2012-01-31 21:47:29: Uploaded 31802 in 10000042 with 24 orders
EVE Marketeer: 2012-01-31 21:47:29: Uploaded 31802 in 10000042 with 24 orders

It also lists a certain number of uploads for the various endpoints etc...

I'll check tomorrow to see if my account received credits for the uploads or not.

There is still however the Debug message I posted earlier that:
Debug: src/helpers.cpp(140): 'CreateActCtx' failed with error 0x00000103 (no more data available).

No idea yet why I get that message, but it seems that it is not necessarily related to your uploader program as I have found some posts on the eve forums with the same message but for the eve repair tool.

Edit: Ok my account received credits so it seems to be working fine
Callean Drevus
Perkone
Caldari State
#371 - 2012-01-31 22:38:45 UTC
Awesome, happy to hear it's working :)

I also added two new pages to the website. I've found that I gather a lot of data, but not nearly everything is exposed, which is why I am now trying to at least show all the API information I gather.

The first two pages are Industry Jobs, and Wallet Information for corporations (Industry Jobs for characters coming soon).

Developer/Creator of EVE Marketeer
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#372 - 2012-01-31 22:57:49 UTC
Great tool :) Appreciate the work you've put in on this.


For people who want to keep an eye on the orders in their areas, and run the unified uploader, this is a very cheap and nasty uploader script, that can be used as an endpoint in it (if you've got an php/mysql server somewhere. like XAMPP).
http://www.fuzzwork.co.uk/resources/upload.zip

Easier to manage than multiple uploaders, when you don't want them butting heads, but still want a local copy for analysis.
It's cheap and nasty, but the data goes in.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Callean Drevus
Perkone
Caldari State
#373 - 2012-01-31 23:02:53 UTC
Lol, looks quick and nasty indeed. But it works, and that's what important Blink

Developer/Creator of EVE Marketeer
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#374 - 2012-01-31 23:05:27 UTC
About the only thing that's not nasty about it is the use of binds with a prepared statement. Might not be validating the datatypes, but no need to worry about escaping.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Maeste Madeveda
The Spawning Pool
#375 - 2012-02-01 11:39:32 UTC
I stumbled upon a LP store database in my web travels : http://www.ellatha.com/eve/LP-Stores. Could prove useful.
6ie
The Kiwis
#376 - 2012-02-01 11:40:37 UTC
To get orders into your own DB - I use EOApp
http://www.eveorg.net/eoapp/index.aspx
It saves the cache files right into your MYSQL or SQL DB.
Callean Drevus
Perkone
Caldari State
#377 - 2012-02-01 16:08:30 UTC
Okey. I've added the insanely cool ability to find the items in LP stores that give you the most ISK for your LP. You can select either by sell or buy orders, but obviously the buy orders give you a more senseful result.

I've also added a feature to the item information page where you will be able to see the places where you can get that item in loyalty point stores (if it is actually possible to buy it somewhere).

Oh, and for the people who do not know yet, you can get the links for IGB features when you right click on either item or location links :)

Developer/Creator of EVE Marketeer
Nebu Retski
Lead Farmers
#378 - 2012-02-01 20:27:08 UTC
So after a bit more of testing I noticed that the uploader always crashes after 270 or 271 uploads. Yesterday I was running the uploader in a 64-bit environment, while today I ran is in a 32-bit environment, but everytime the result is the same. Manually restarting the uploader allows me to provide another 270/271 uploads.

You can find the error log here, although you might not be much with it as a lot of the output is comming from wine.

For completeness sake: I'm running 3.1.2-1-ARCH and wine 1.3.33

A possible solution could be to produce a native linux binary. This most likely requires you to have a linux distribution at hand to compile it or provide the source code to a trustworthy friend that can do it for you. In pretty much a similar way you could produce a native mac version. Anyway that is totally up to you to decide. In the meantime I will keep using it as is because I'm usually in the neighbourhood of my pc when uploading.

Keep up the good work!
Gabba Cyno
#379 - 2012-02-01 20:39:09 UTC
I am having a small amount of trouble displaying history on some PI items in my google docs XLS. I am able to pull all sorts of current info about items but when I try to pull history of the items I run into problems.

=ImportXML("http://www.evemarketeer.com/api/info/2867_2868_2869_2870_2871_2872_2875_2876/xml/10000002";"//sell_lowest5")

This is what I have in a sell that shows me the sell price of the lowest 5% of P4 PI items. That works great. But I am stock on how to display history of these items. I would love to display 30, 60, 90 day info on the following.

average : average transactions a day over this time
maximum : maximum transactions a day over this time
minimum : minimum transactions a day over this time
average_price : average price over this time
maximum_price : maximum price over this time
minimum_price : minimum price over this time

Any help on what code I need to put in my googe doc cell to make this work?

.
Nebu Retski
Lead Farmers
#380 - 2012-02-01 20:59:42 UTC
=ImportXML("http://www.evemarketeer.com/api/averages/10000043/90/34_35_36/xml";"//average")

This gives me pretty much a column of the average transactions per day for the 3 different items. Now you just make 5 more columns where you change //average with //maximum ...

There might be a better way to do it, but I don't know atm :)