These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

Market Discussions

 
  • Topic is locked indefinitely.
 

Eve Market Data Dumps

First post
Author
Previous Topic Next Topic
Kaprer Suhr
New Eden Mining Union Incorporated
#1 - 2016-07-11 00:44:03 UTC
Hi,

I've been reading legendary Capsuleer Vaerah Varroks website. Very good stuff. This was one of the reasons I got into real life trading myself. I am looking for updated information on creating candle charts from eve market data dumps.

The steps are as follows:

Obtain daily market data in OHLC form (Open high low close)

Import into trading platform.

It is proving very difficult to obtain OHLC data. Vaerah created a program for helping with this problem, yet there remain problems with downloading it from the site, and after findings another download location, issues with the config files. (it was created as a functional tool for himself not for distribution) Anyway, I was wondering if there were any along the same path who could lead me to valid, and importable csv files of eve market data. I know that many of the sites offer this data in api formats, however, I do not know what an API is!

any and all assists is as always, much loved.

Parker
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#2 - 2016-07-11 05:21:09 UTC
If you're _just_ looking for CSVs with all the orders in new eden, https://market.fuzzwork.co.uk/api/ has the ones I'm using. (no timestamps on them, I'm afraid. ) Created every half hour.

But that's for active orders.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Vaerah Vahrokha
Vahrokh Consulting
#3 - 2016-07-11 22:50:35 UTC
Kaprer Suhr wrote:
Hi,

I've been reading legendary Capsuleer Vaerah Varroks website. Very good stuff. This was one of the reasons I got into real life trading myself. I am looking for updated information on creating candle charts from eve market data dumps.

The steps are as follows:

Obtain daily market data in OHLC form (Open high low close)

Import into trading platform.

It is proving very difficult to obtain OHLC data. Vaerah created a program for helping with this problem, yet there remain problems with downloading it from the site, and after findings another download location, issues with the config files. (it was created as a functional tool for himself not for distribution) Anyway, I was wondering if there were any along the same path who could lead me to valid, and importable csv files of eve market data. I know that many of the sites offer this data in api formats, however, I do not know what an API is!

any and all assists is as always, much loved.

Parker


Hello,

Every OHLC capable EvE website has been shut down, therefore my software would be the only way to still extract such kind of information.

However as of now, a most important component used by the software to read EvE's cache data, Reverence by Entity, is not updated since 2015.

As a consequence, I have a new version ready to go, but it throws out the errors you sent me by EvE mail.

I don't know if Entity still plays, if anybody knows, I could try and contact him in game and see if he wants to release an updated version of his work.

EvE Online portal | EvE markets tutorials
Vaerah Vahrokha
Vahrokh Consulting
#4 - 2016-07-11 22:58:10 UTC
Steve Ronuken wrote:
If you're _just_ looking for CSVs with all the orders in new eden, https://market.fuzzwork.co.uk/api/ has the ones I'm using. (no timestamps on them, I'm afraid. ) Created every half hour.

But that's for active orders.


Sadly that data won't do. There are basically two ways to create decently accurate OHLC bars. One used by my EvE market app, which uses some pre-treated data from EvE's cache and another that requires a massively huge work (including hooking up with some real time data feed with ZeroMQ, massaging the data, building some NoSQL database and so on).

Once upon a time I had large amounts of time for EvE and wanted to implement that stuff. Now I just earn money from the real markets so my time for game money has dropped a lot.

EvE Online portal | EvE markets tutorials
Hel O'Ween
Men On A Mission
#5 - 2016-07-12 16:32:21 UTC
Vaerah Vahrokha wrote:

However as of now, a most important component used by the software to read EvE's cache data, Reverence by Entity, is not updated since 2015.


IIRC, cache scraping has been deemed as illegal with the invention of CREST, which might be the reason the Entity hasn't update his lib since then.

EVEWalletAware - an offline wallet manager.
Vaerah Vahrokha
Vahrokh Consulting
#6 - 2016-07-13 02:34:41 UTC  |  Edited by: Vaerah Vahrokha
Hel O'Ween wrote:
Vaerah Vahrokha wrote:

However as of now, a most important component used by the software to read EvE's cache data, Reverence by Entity, is not updated since 2015.


IIRC, cache scraping has been deemed as illegal with the invention of CREST, which might be the reason the Entity hasn't update his lib since then.


Thank you.

It's a pity, because cache scraping would let me get done with the task with about 300 Python lines.

I have checked CREST and it looks like I am meant to process "stanzas" like:

"items": [{
"volume_str": "25735487080",
"orderCount": 1964,
"lowPrice": 5.17,
"highPrice": 5.33,
"avgPrice": 5.23,
"volume": 25735487080,
"orderCount_str": "1964",
"date": "2015-06-01T00:00:00"
}


It won't give the "open" and "close" price values as well. To get those values I have to do some nasty workarounds.
Basically it's too much work for my time for now.

EvE Online portal | EvE markets tutorials
Rhivre
TarNec
Invisible Exchequer
#7 - 2016-07-13 05:13:21 UTC  |  Edited by: Rhivre
There is someone in the lounge ingame who uses trading software in Eve, I havent had any coffee yet today, so I cannot remember his name right now, but, I will endeavour to remember once I have woken up a bit, otherwise I will try to find out when I get back home today.


Entity is still around, but doesnt really log in.

Lockefox might do some OHLC stuff as well.

Fluffy Bunny Pic!

Lockefox
EVE Prosper Market Analysis
Prosper Investment Bank
#8 - 2016-07-13 05:31:24 UTC  |  Edited by: Lockefox
o7 from the Prosper Market Show

Have a couple of tools that might be up your alley.

For a really really DIY feed, I have a (totally not ripped off from Quant) way to generate candlesticks/OHLC in R here: https://gist.github.com/lockefox/9b65be95ec05e8fae266

As for more direct feed, there are some kinks in the feed, but I do have an OHLC API endpoint we spun up for EVE Mogul. It's not super stable yet, but it's a thing.

http://api.eveprosper.com/CREST/OHLC.[csv|json]?typeID=[typeid]&regionID=[regionid]


EX:
 http://api.eveprosper.com/CREST/OHLC.csv?typeID=34&regionID=10000002

Working on getting more API feeds out by the end of the summer. There's a ton of backend work we're crunching through to get our data out to more people!
Skia Aumer
Planetary Harvesting and Processing LLC
#9 - 2016-07-15 12:47:29 UTC
Rhivre wrote:
There is someone in the lounge ingame who uses trading software in Eve, I havent had any coffee yet today, so I cannot remember his name right now, but, I will endeavour to remember once I have woken up a bit, otherwise I will try to find out when I get back home today.

Would be interesting to see how IRL trading tools are used in EVE. Did you remember the name?
Vaerah Vahrokha
Vahrokh Consulting
#10 - 2016-07-16 17:14:56 UTC
Skia Aumer wrote:
Rhivre wrote:
There is someone in the lounge ingame who uses trading software in Eve, I havent had any coffee yet today, so I cannot remember his name right now, but, I will endeavour to remember once I have woken up a bit, otherwise I will try to find out when I get back home today.

Would be interesting to see how IRL trading tools are used in EVE. Did you remember the name?


Here's an example with a step by step tutorial. On the same website there are few others as well.

In general, as long as you can extract data off the EvE markets, it's just a matter of juggling data until it's appetible to one of the RL platforms that can import data for off-line analysis.

EvE Online portal | EvE markets tutorials
Lockefox
EVE Prosper Market Analysis
Prosper Investment Bank
#11 - 2016-07-18 18:23:11 UTC
Vaerah Vahrokha wrote:
Skia Aumer wrote:
Rhivre wrote:
There is someone in the lounge ingame who uses trading software in Eve, I havent had any coffee yet today, so I cannot remember his name right now, but, I will endeavour to remember once I have woken up a bit, otherwise I will try to find out when I get back home today.

Would be interesting to see how IRL trading tools are used in EVE. Did you remember the name?


Here's an example with a step by step tutorial. On the same website there are few others as well.

In general, as long as you can extract data off the EvE markets, it's just a matter of juggling data until it's appetible to one of the RL platforms that can import data for off-line analysis.


Just to throw my 2isk in the pile. I'm not convinced traditional trading signaling is really useful inside EVE. Though you may be able to use something like MACD IRL, trying to fit those signals against EVE data tends to miss more than it hits. Getting a better hand around the data is extremely useful, ofc. Just not sure how much I really believe in Candlestickers.

Though I've found some use in signals like RSI, it's still mostly gut-tuning more than specific automation so far. But I'm also not a real quant, just an EE playing data scientist ;)
William Legrand-Marx
Nemesis Ad Astra
#12 - 2016-07-18 18:35:42 UTC
Lockefox wrote:
Vaerah Vahrokha wrote:
Skia Aumer wrote:
Rhivre wrote:
There is someone in the lounge ingame who uses trading software in Eve, I havent had any coffee yet today, so I cannot remember his name right now, but, I will endeavour to remember once I have woken up a bit, otherwise I will try to find out when I get back home today.

Would be interesting to see how IRL trading tools are used in EVE. Did you remember the name?


Here's an example with a step by step tutorial. On the same website there are few others as well.

In general, as long as you can extract data off the EvE markets, it's just a matter of juggling data until it's appetible to one of the RL platforms that can import data for off-line analysis.


Just to throw my 2isk in the pile. I'm not convinced traditional trading signaling is really useful inside EVE. Though you may be able to use something like MACD IRL, trying to fit those signals against EVE data tends to miss more than it hits. Getting a better hand around the data is extremely useful, ofc. Just not sure how much I really believe in Candlestickers.

Though I've found some use in signals like RSI, it's still mostly gut-tuning more than specific automation so far. But I'm also not a real quant, just an EE playing data scientist ;)



Agree , more so stochastics would be great but yet again hardly usable in economy as it is now.

There is nothing worthy in this world even if others think it is worth something...
Vaerah Vahrokha
Vahrokh Consulting
#13 - 2016-07-21 22:46:39 UTC
Lockefox wrote:
Vaerah Vahrokha wrote:
Skia Aumer wrote:
Rhivre wrote:
There is someone in the lounge ingame who uses trading software in Eve, I havent had any coffee yet today, so I cannot remember his name right now, but, I will endeavour to remember once I have woken up a bit, otherwise I will try to find out when I get back home today.

Would be interesting to see how IRL trading tools are used in EVE. Did you remember the name?


Here's an example with a step by step tutorial. On the same website there are few others as well.

In general, as long as you can extract data off the EvE markets, it's just a matter of juggling data until it's appetible to one of the RL platforms that can import data for off-line analysis.


Just to throw my 2isk in the pile. I'm not convinced traditional trading signaling is really useful inside EVE. Though you may be able to use something like MACD IRL, trying to fit those signals against EVE data tends to miss more than it hits. Getting a better hand around the data is extremely useful, ofc. Just not sure how much I really believe in Candlestickers.

Though I've found some use in signals like RSI, it's still mostly gut-tuning more than specific automation so far. But I'm also not a real quant, just an EE playing data scientist ;)


I have developed some quant algos, they require some decent amounts of data. In EvE's case, my data goes back to 2009 (only for few items), it might provide some little insight about market inefficiencies. EvE's markets are vastly, hugely less efficient than RL markets so it would be actually possible to exploit them.

However in order to do some statistically meaningful analysis you'd want 4-5 times more than such amount of data and this involves samping intraday data. This is plenty possible, but to be honest, if I have to pour in such kind of effort into a "fake" market, I may as well employ the knowledge to make real money in the real markets Blink

Example:
Equity curve graph
Stress testing the algo with Monte Carlo simulation


As for "indicators", "candlesticks etc.": I've never been an huge indicators fan. Indicators are laggy and used to do well enough until the middle '90s, when markets started losing auto-correlation.
Candlesticks too are fairly useless unless looked in a broader context. Looking at how market orders play with each other and the kind of reaction price shows when hitting big order blocks one may create some correlation between bars formations around those blocks and use them as price action indicators. That's neither an easy nor exact science though, takes a lot of practice.

Here's a little reference to my past studies about the matter. Here's an overview of that study on TheMittani.com

EvE Online portal | EvE markets tutorials
Lockefox
EVE Prosper Market Analysis
Prosper Investment Bank
#14 - 2016-07-21 23:35:46 UTC  |  Edited by: Lockefox
Just FYI, EVE Prosper does have (for hubs) Intraday snapshots of eve central going back about 18mo (hourly for jita, 4hr for others). Though not the entire picture, since capturing the entire orderbook would be a nightmare to store, it provides some excellent resolution into trends. We also have 8mo of zkillboard data, and should be capturing map data soon(tm).

We're currently working on API's to get them better served so we can publish public feeds for people. Basically replicate Quandl for EVE. Still need to do a good portion of work on the base layers though, so stay tuned.
Kaprer Suhr
New Eden Mining Union Incorporated
#15 - 2016-07-22 23:15:04 UTC
Stunned by the responses, thank you all. I have come to terms with candlestick data being "not so good" in eve markets. It just amazes me you all have the ability to aggregate data like that! Shocked The program evernus actually has in their market analysis tab, a candlestick viewer! Good only for price action trading and trend analysis however, as it is not a rl trading program with macd or stocastics (does have built in bollinger bands however.

again thanks, and best of luck with your projects!