These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Wanted: Market statistical analysis program
Author
Previous Topic Next Topic
Zorok
The Guardian Knights
#1 - 2012-01-20 07:26:11 UTC
Hi,
I am looking for a program out there that will analyze the sell prices for any given standard market item (to include mineral prices) in the game and have it display a graph showing a demand curve as different given prices. Basically what this would do is estimate an expected number of days (if any) that it would take to sell a product at a certain price point. The way the program would do this is it would calculate the number of days a certain sell order has been on the market and plot this onto a graph along with other sell orders for the same product. It would link the individual sell order's price and correlate that to the number of days that the market order has been sitting there. This would apply for not just one region but as many regions as the user is willing to export the data from. This would allow the user to look at all exported regions to determine where the most profitability or demand would be within a give solar system in the game as well.
The program would also allow the individual to compare the profitability of various items in the game and determine which of those items generates the most profit either shown by demand (most profit per day) or simply by max profit (profit without regard to amount of time it takes to sell item).
I've seen a lot of market programs out there for Eve but none of them give me this kind of information. Most are simply built around simply market order or wallet management. I had written an Access database program to do some of this stuff for at least the T1 items in the game however I think something like this deserves its own code (that and I ran into parsing issues after they changed the market log formatting rendering my database useless.) DOes anyone else think there is a good demand for this sort of thing?
What my database program would do is calculate an average mineral price for each region and then base the cost to build on the regions mineral price. It would also allow you to find mineral selling hubs where mineral prices were highest. After all, you don't want to sell an item in a given region if it's more profitable to sell the minerals instead.

Thanks!!
Callean Drevus
Perkone
Caldari State
#2 - 2012-01-20 17:08:09 UTC
Interesting idea, what you basically want is a graph showing average order duration versus selling/buying price. It can't be completely accurate, but it is certainly feasible.

I'll but this on my 'to be done sometime' list.

Developer/Creator of EVE Marketeer
Pinky Attor
#3 - 2012-01-24 18:40:30 UTC
Zorok wrote:
Hi,
It would link the individual sell order's price and correlate that to the number of days that the market order has been sitting there. This would apply for not just one region but as many regions as the user is willing to export the data from. This would allow the user to look at all exported regions to determine where the most profitability or demand would be within a give solar system in the game as well.



Doing this with Eve Central's data requires saving the specific order ID's daily snapshot. When you pull the Eve Central data you need an extra field in your table that it is saved to that shows this. It would look like this;

PullDate, orderid, lots of other fields....
2012-01-01, 123456, ...
2012-01-02, 123456, ...
2012-01-03, 123456, ...
2012-01-01, 987654, ...
2012-01-02, 987654, ...
2012-01-03, 987654, ...

You would then have a history of each order's daily snapshot. You can then track the rate at which volremain field decreases over time. You could even derive a field that shows the percentage remaining each day by dividing volremain by volenter. There are plenty of ways to slice and dice that stat to get what you want. You could also do a grouped query where the max and min dates for each order ID are shown, with the number of days the order existed as max(PullDate) minus min(PullDate).

Just remember that Eve Central data is not perfect and that your stats may not be fully reflective of what is really go on in the market. I find it's good enough usually. Do this in Access will be tough as the Eve central dumps are pretty large.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#4 - 2012-01-24 19:10:48 UTC
With Eve Central's data, you can use something like


DATE=`date -d "-1 day" +%Y-%m-%d`
cd /tmp/salesdata
wget http://eve-central.com/dumps/$DATE.dump.gz
gunzip $DATE.dump.gz
dos2unix $DATE.dump
chmod a+r $DATE.dump
mysql eve --execute "load data local infile '/tmp/salesdata/$DATE.dump' Replace into table salesdata Fields terminated by ',' ENCLOSED BY '\"' ignore 1 lines (orderid,regionid,systemid,stationid,typeid,bid,price,minvolume,volremain,volenter,issued,duration,rang,reportedby,@dt) set reportedtime=str_to_date(substring(@dt,1,19), '%Y-%m-%d %H:%i:%s');"


That'll load it in and update it. However, you should be able to add in a new field with now() for keeping the records separate.

That way, you can then manipulate the DB to get the figures you want.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Decimat Draconia
Brutor Tribe
Minmatar Republic
#5 - 2012-01-26 16:06:16 UTC
Steve Ronuken wrote:
With Eve Central's data, you can use something like


DATE=`date -d "-1 day" +%Y-%m-%d`
cd /tmp/salesdata
wget http://eve-central.com/dumps/$DATE.dump.gz
gunzip $DATE.dump.gz
dos2unix $DATE.dump
chmod a+r $DATE.dump
mysql eve --execute "load data local infile '/tmp/salesdata/$DATE.dump' Replace into table salesdata Fields terminated by ',' ENCLOSED BY '\"' ignore 1 lines (orderid,regionid,systemid,stationid,typeid,bid,price,minvolume,volremain,volenter,issued,duration,rang,reportedby,@dt) set reportedtime=str_to_date(substring(@dt,1,19), '%Y-%m-%d %H:%i:%s');"


That'll load it in and update it. However, you should be able to add in a new field with now() for keeping the records separate.

That way, you can then manipulate the DB to get the figures you want.


I'm new to this although I can read most of the script I don't know altogether what I'm looking at. I believe I'm correct in thinking you have Unix running on a VM with MYSQL talking to a database called eve. The SQL script will then output all the dumps in date order to the salesdata table? I'm not sure how that last section of code works without trying it?

I'm thinking of running a MSSQL server avoiding Linux and using a small amount of the dump (2012). Is there a similar script I can work with or some straightforward SQL commands I can issue to pull specific market items and data for 2012.

I'm trying to create a small table with history of complex reactions prices which I can generate a graph from.
Jenny Aakiwa
Perkone
Caldari State
#6 - 2012-01-26 22:39:39 UTC  |  Edited by: Jenny Aakiwa
I can provide a programming service to develop, customize and integrate the information to suit your needs. You can contact my previous client, "Rita Jita" about his ongoing successes with my service. I can develop an application that integrate Microsoft Office Suites, website, a new windows application, and endless possibility. I also have extensive knowledge of integrating Eve API, Eve Central API, Eve Market Data API, and EVE Cache/Exported files api to the fundamental application to produce a comprehensible report to give you up to date knowledge to engage the market and make maximum profit.

My fee is on average around 250 mil an hour of programming and I can do a short series of "On-Demand" projects to produce a desirable output to you while I proceed to work on the product. I also require that the project to be set in milestones so that I can be ensured that I am being paid for my service and satisfy your needs in the process. The estimated fee for the completed full-blown application will cost around 5 bil isk on average. Hire me as your private programmer to develop a software to suit your every needs and no longer have to learn all the complicated programs to get the result you need now!
Vaerah Vahrokha
Vahrokh Consulting
#7 - 2012-01-26 23:46:33 UTC
You basically want something like this program.

It needs an update but then it already got logic to plot market velocity, items velocity and similar.

EvE Online portal | EvE markets tutorials
Zorok
The Guardian Knights
#8 - 2012-02-06 19:27:05 UTC
Vaerah Vahrokha wrote:
You basically want something like this program.

It needs an update but then it already got logic to plot market velocity, items velocity and similar.


I tried to run the program but it didn't seem to run on my system. It looks promising from what I could tell from the tab labels. Most programs out there don't seem to do any real market analysis besides perhaps displaying a median/high/low price for an item. I saw it also had a wallet feature that I probably wouldn't use. As far as the offer of having someone build a program from the ground up, I don't make a whole lot of money in Eve. I was just looking for a pre-existing program that I could use.
Callean Drevus
Perkone
Caldari State
#9 - 2012-02-06 19:39:12 UTC  |  Edited by: Callean Drevus
Well, EMK will naturally evolve to the solution that you are looking for (I hope), but it'll take a long time :)

And that program actually looks like it's made to be transformed into about 6 or 7 EMK pages. That might be sooner than I thought it could be.

Developer/Creator of EVE Marketeer
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#10 - 2012-02-06 19:55:00 UTC
Decimat Draconia wrote:
Steve Ronuken wrote:
With Eve Central's data, you can use something like


DATE=`date -d "-1 day" +%Y-%m-%d`
cd /tmp/salesdata
wget http://eve-central.com/dumps/$DATE.dump.gz
gunzip $DATE.dump.gz
dos2unix $DATE.dump
chmod a+r $DATE.dump
mysql eve --execute "load data local infile '/tmp/salesdata/$DATE.dump' Replace into table salesdata Fields terminated by ',' ENCLOSED BY '\"' ignore 1 lines (orderid,regionid,systemid,stationid,typeid,bid,price,minvolume,volremain,volenter,issued,duration,rang,reportedby,@dt) set reportedtime=str_to_date(substring(@dt,1,19), '%Y-%m-%d %H:%i:%s');"


That'll load it in and update it. However, you should be able to add in a new field with now() for keeping the records separate.

That way, you can then manipulate the DB to get the figures you want.


I'm new to this although I can read most of the script I don't know altogether what I'm looking at. I believe I'm correct in thinking you have Unix running on a VM with MYSQL talking to a database called eve. The SQL script will then output all the dumps in date order to the salesdata table? I'm not sure how that last section of code works without trying it?

I'm thinking of running a MSSQL server avoiding Linux and using a small amount of the dump (2012). Is there a similar script I can work with or some straightforward SQL commands I can issue to pull specific market items and data for 2012.

I'm trying to create a small table with history of complex reactions prices which I can generate a graph from.


oops. didn't set this thread to alert.

Yes, this is using a mysql database running on a linux server (actual server, but VM would work.)
This will download the last days dump from eve central (using wget) unzip it, change the line endings so it doesn't break the loader, set it as readable by everyone (so mysql can get at it), then use the load data infile functionality to load in a csv file. The very last bit is just messing with the date in the file to make it load in correctly. the replace into table bit means with each file it loads, it updates the rows according to the orderid (primary key)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Zorok
The Guardian Knights
#11 - 2012-02-07 13:41:12 UTC
Callean Drevus wrote:
Well, EMK will naturally evolve to the solution that you are looking for (I hope), but it'll take a long time :)

And that program actually looks like it's made to be transformed into about 6 or 7 EMK pages. That might be sooner than I thought it could be.


I tried to go to the webpage but it appears to have some kind of SQL error (just a guess). Anyhow, I'm guessing that this is more of a website than a standalone program. I suppose a webpage would work well although I would prefer a program that could simply take in exported market logs and analyze them on your computer.