These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

Market Discussions

 
  • Topic is locked indefinitely.
 

New site - Market data

First post
Author
Previous Topic Next Topic
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#1 - 2016-06-25 21:04:30 UTC
https://market.fuzzwork.co.uk/

May be of interest

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
John Volan
Imperial Shipment
Amarr Empire
#2 - 2016-06-26 02:21:42 UTC
Very cool, thanks for doing this. Looking forward to the API.
Tipa Riot
Federal Navy Academy
Gallente Federation
#3 - 2016-06-26 07:26:13 UTC  |  Edited by: Tipa Riot
Wow, just wow! This is a giant leap in transparency, though a little frightening to see the details of my orders in public. ShockedLol

Thank you for this service!

EDIT: will CCP add this extra information also in-game? It's a bit odd you can get the details of any order in the market only via API, but not in-game.

EDIT2: this intel is priceless, you can now exactly tell when people are selling/buying and in what chunks ...

I'm my own NPC alt.
Tipa Riot
Federal Navy Academy
Gallente Federation
#4 - 2016-06-26 07:46:48 UTC
What is the meaning of orderset? I see changes in this value even without a change in price or volume, sometimes 10 in a row at the same timestamp.

I'm my own NPC alt.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#5 - 2016-06-26 15:10:50 UTC
What I'm doing is retrieving the order book once every 30 minutes, and storing it all with a different orderSet value.

So when you dig into an order's history, you're seeing it snapshots every 30 minutes. (I'm still deciding how long to keep them for. As it gets bigger, performance will suffer)


I'd retrieve more often, but processing the data takes a while. I could _probably_ bring it up to 15 minutes, but that would be very tight, and load could push it into overlapping (which could cascade)


(an about page is coming. But I haven't written it yet)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Zad Murrard
Frozen Dawn Inc
Frozen Dawn Alliance
#6 - 2016-06-26 15:30:28 UTC
Very good. Looking forward to the API.

What is the intended role for the site in relation to existing market sites?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#7 - 2016-06-26 15:43:43 UTC
Zad Murrard wrote:
Very good. Looking forward to the API.

What is the intended role for the site in relation to existing market sites?



No intended role tbh. it's just another eve-central, or eve market data.

A pet project, for me to to stretch my analytics.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Tipa Riot
Federal Navy Academy
Gallente Federation
#8 - 2016-06-26 17:03:30 UTC
Steve Ronuken wrote:
What I'm doing is retrieving the order book once every 30 minutes, and storing it all with a different orderSet value.

So when you dig into an order's history, you're seeing it snapshots every 30 minutes. (I'm still deciding how long to keep them for. As it gets bigger, performance will suffer)


I'd retrieve more often, but processing the data takes a while. I could _probably_ bring it up to 15 minutes, but that would be very tight, and load could push it into overlapping (which could cascade)


(an about page is coming. But I haven't written it yet)

Maybe this is possible, but you could just store an entry only if changed from previous snapshot (per order), otherwise discard. 30m is OK for analysis but 15m would be better of course.

I'm my own NPC alt.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#9 - 2016-06-26 22:28:54 UTC
Tipa Riot wrote:
Steve Ronuken wrote:
What I'm doing is retrieving the order book once every 30 minutes, and storing it all with a different orderSet value.

So when you dig into an order's history, you're seeing it snapshots every 30 minutes. (I'm still deciding how long to keep them for. As it gets bigger, performance will suffer)


I'd retrieve more often, but processing the data takes a while. I could _probably_ bring it up to 15 minutes, but that would be very tight, and load could push it into overlapping (which could cascade)


(an about page is coming. But I haven't written it yet)

Maybe this is possible, but you could just store an entry only if changed from previous snapshot (per order), otherwise discard. 30m is OK for analysis but 15m would be better of course.



Storing only deltas complicates things, unfortunately. right now, I can store everything by just dumping it into the DB. Deltas would require a lookup per order being loaded (around 1.6 million of them)

It also means for display purposes, I'd need to pull everything ordered by the order set, then filter it down to the latest one.

I _may_ do the filtering on display of history, as that will be significantly smaller numbers being kicked around.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Vaerah Vahrokha
Vahrokh Consulting
#10 - 2016-06-27 18:40:53 UTC
Hello,

I am checking your nice website out.

I am trying to open this order.

However this triggers a bug:

Title
Bad Connection to gateway

Text
Looks like something is wrong with the database. Steve will need to kick it repeatedly until it works again

EvE Online portal | EvE markets tutorials
TheSmokingHertog
Julia's Interstellar Trade Emperium
#11 - 2016-06-27 18:48:09 UTC
Steve Ronuken wrote:
Tipa Riot wrote:
Steve Ronuken wrote:
What I'm doing is retrieving the order book once every 30 minutes, and storing it all with a different orderSet value.

So when you dig into an order's history, you're seeing it snapshots every 30 minutes. (I'm still deciding how long to keep them for. As it gets bigger, performance will suffer)


I'd retrieve more often, but processing the data takes a while. I could _probably_ bring it up to 15 minutes, but that would be very tight, and load could push it into overlapping (which could cascade)


(an about page is coming. But I haven't written it yet)

Maybe this is possible, but you could just store an entry only if changed from previous snapshot (per order), otherwise discard. 30m is OK for analysis but 15m would be better of course.



Storing only deltas complicates things, unfortunately. right now, I can store everything by just dumping it into the DB. Deltas would require a lookup per order being loaded (around 1.6 million of them)

It also means for display purposes, I'd need to pull everything ordered by the order set, then filter it down to the latest one.

I _may_ do the filtering on display of history, as that will be significantly smaller numbers being kicked around.


What if you would just start a delta on a lookup?

"Dogma is kind of like quantum physics, observing the dogma state will change it." ~ CCP Prism X

"Schrödinger's Missile. I dig it." ~ Makari Aeron

-= "Brain in a Box on Singularity" - April 2015 =-
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#12 - 2016-06-28 16:51:07 UTC  |  Edited by: Steve Ronuken
Vaerah Vahrokha wrote:
Hello,

I am checking your nice website out.

I am trying to open this order.

However this triggers a bug:

Title
Bad Connection to gateway

Text
Looks like something is wrong with the database. Steve will need to kick it repeatedly until it works again



hrm. For some reason that page is taking too long to generate. (it's probably to do with the face I'm inserting and deleting around 81 million records per day. The db doesn't like it) One of the reasons I'm considering replacing the server with one with SSDs instead. The load stage is pegging the disk for a fairly variable period of time.


If you were to look _right now_, it works.


Edit:
Digging into this, yes, it's down to the market load pegging the disk. SSDs _should_ fix this, but my options are either waiting for one company to replenish their stocks ( soyoustart ) or paying about double what I'm paying right now for that server (moving to hetzner. It's a significantly better server than the soyostart one (500GB vs 300GB for storage, 32GB vs 16GB for ram), but there's also a one off setup fee.)
While I'm not looking to make money on any of this stuff (because I'm not insane Big smile, and it's not why I do it), that's going to mean pushing it to next month. (Unless I get a run on my Patreon. Somehow, I doubt that'll happen )

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Vaerah Vahrokha
Vahrokh Consulting
#13 - 2016-07-01 07:21:46 UTC
Steve Ronuken wrote:
Vaerah Vahrokha wrote:
Hello,

I am checking your nice website out.

I am trying to open this order.

However this triggers a bug:

Title
Bad Connection to gateway

Text
Looks like something is wrong with the database. Steve will need to kick it repeatedly until it works again



hrm. For some reason that page is taking too long to generate. (it's probably to do with the face I'm inserting and deleting around 81 million records per day. The db doesn't like it) One of the reasons I'm considering replacing the server with one with SSDs instead. The load stage is pegging the disk for a fairly variable period of time.


If you were to look _right now_, it works.


Edit:
Digging into this, yes, it's down to the market load pegging the disk. SSDs _should_ fix this, but my options are either waiting for one company to replenish their stocks ( soyoustart ) or paying about double what I'm paying right now for that server (moving to hetzner. It's a significantly better server than the soyostart one (500GB vs 300GB for storage, 32GB vs 16GB for ram), but there's also a one off setup fee.)
While I'm not looking to make money on any of this stuff (because I'm not insane Big smile, and it's not why I do it), that's going to mean pushing it to next month. (Unless I get a run on my Patreon. Somehow, I doubt that'll happen )


I stumbled upon this kind of issues so many times.

I fixed them by writing my own query (result set) caching classes, usually working with memchached / REDIS.
With some creativity you can even implement "cache priming" during idle time, so you don't get those huge data access peaks.

Some other times (I see you have some good RAM) I got helped by directly putting most used pages on Nginx + Varnish.

Hope some of this stuff helps.

EvE Online portal | EvE markets tutorials
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#14 - 2016-07-04 12:42:55 UTC
https://market.fuzzwork.co.uk/api/ is now up and running.

Still need to write the function for googledocs users

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
James Zealot
Science and Trade Institute
Caldari State
#15 - 2016-07-05 05:02:09 UTC
I love you for this :-)