These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Snarf Aldes
University of Caille
Gallente Federation
#181 - 2012-06-18 10:12:39 UTC  |  Edited by: Snarf Aldes
Desmont McCallock wrote:
If those are 'history' messages it's perfectly normal as Entity explained in https://forums.eveonline.com/default.aspx?g=posts&m=1265226#post1265226

Nope, they are all from orders.

edit:
Even for history the generatedAt value should be variable, its not the actual data from the 'rows' field, those are all dates with time 00:00:00.

Creator of Eve Addicts
Desmont McCallock
#182 - 2012-06-18 11:21:24 UTC
I just checked with EVEMon and it turns out to be an EMDR bug.
Otaci
Tides of Silence
#183 - 2012-06-18 11:48:38 UTC
Just confirming the 00:00 bug.

Number of occurrences of a particular {Hrs,Mins} time in a sample of 2750987 market snapshots:

[{{0,0},2749524},
{{9,57},33},
{{10,38},33},
{{10,28},30},
{{10,37},30},
{{10,30},30},
{{10,31},29},
{{10,36},27},
{{10,39},27},
{{10,40},25},
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#184 - 2012-06-18 14:03:53 UTC
Is this a new behavior, or has it been this way for a while? If anyone with historical records could comment with certainty, that'd be awesome. I'll start seeing if I can reproduce it, in the meantime.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Barl Rathbone
Corel Industries
#185 - 2012-06-18 22:17:17 UTC
Ilyk Halibut wrote:
Is this a new behavior, or has it been this way for a while? If anyone with historical records could comment with certainty, that'd be awesome. I'll start seeing if I can reproduce it, in the meantime.


This is the first 30 rows from my DB so a few days old since I expunged:

+-----------+---------------------+
| orderID | generatedAt |
+-----------+---------------------+
| 468766023 | 2012-06-16 00:00:43 |
| 468840171 | 2012-06-17 00:00:52 |
| 547489123 | 2012-06-18 00:00:47 |
| 571460595 | 2012-06-17 00:00:50 |
| 574327332 | 2012-06-16 00:00:25 |
| 584959892 | 2012-06-16 00:00:30 |
| 602992867 | 2012-06-02 00:00:44 |
| 608251359 | 2012-05-29 00:00:17 |
| 625302900 | 2012-06-02 00:00:06 |
| 628410159 | 2012-06-15 00:00:49 |
| 632715417 | 2012-06-09 00:00:08 |
| 685886645 | 2012-06-10 00:00:04 |
| 693936276 | 2012-06-17 00:00:52 |
| 693943346 | 2012-06-16 00:00:01 |
| 701055227 | 2012-06-14 00:00:35 |
| 718577482 | 2012-06-04 00:00:48 |
| 742684127 | 2012-06-15 00:00:10 |
| 742684363 | 2012-06-17 00:00:28 |
| 742688256 | 2012-06-17 00:00:08 |
| 765443947 | 2012-05-29 00:00:03 |
| 785285268 | 2012-06-11 00:00:58 |
| 795473049 | 2012-06-14 00:00:23 |
| 809346419 | 2012-06-13 00:00:37 |
| 809347982 | 2012-06-10 00:00:30 |
| 810868383 | 2012-06-15 00:00:57 |
| 816612483 | 2012-06-16 00:00:08 |
| 822470501 | 2012-06-17 00:00:26 |
| 822877900 | 2012-06-13 00:00:28 |
| 823070102 | 2012-06-16 00:00:49 |
| 823071375 | 2012-06-16 00:00:35 |
+-----------+---------------------+
Desmont McCallock
#186 - 2012-06-19 10:13:54 UTC
Problem solved. EMDR back in business.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#187 - 2012-06-20 02:38:59 UTC
A european relay has been added. If you're across the pond, make sure you use that.

http://www.eve-emdr.com/en/latest/access.html

p.s., we'd love to have some more relays!

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#188 - 2012-06-26 08:37:24 UTC
Just wanted to say that since the generatedAt issue has been solved, everything has been running very smoothly.
The EVEMon / EMDR combination keeps Eve Addicts supplied with very up to date data.

Thnx to both, it works really well.

Creator of Eve Addicts
Barl Rathbone
Corel Industries
#189 - 2012-06-26 13:29:56 UTC
I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#190 - 2012-06-26 14:38:00 UTC
Snarf Aldes wrote:
Just wanted to say that since the generatedAt issue has been solved, everything has been running very smoothly.
The EVEMon / EMDR combination keeps Eve Addicts supplied with very up to date data.

Thnx to both, it works really well.

Thanks, glad to hear it's working well. The community has provided all kinds of useful feedback, help, and relays, which has made things go really smoothly. I hope we continue to see people share ideas about how to use EMDR, and that we can all find new and exciting things to do with the data!

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#191 - 2012-06-26 14:39:15 UTC
Barl Rathbone wrote:
I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier.

That would be really nice, though, CCP has repeatedly expressed their opposition to providing real-time market data. I'm not sure they'd do anything to help our cause like that, though, it would be nice :)

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#192 - 2012-06-26 14:40:40 UTC
Those on the mailing list have already seen this a while back, but I thought I'd also mention here that we now have four relays on two continents. Europe and the US each have two:

http://www.eve-emdr.com/en/latest/access.html

For those who value reliability as a chief concern, make sure you're connecting to more than one relay, and de-duplicating the messages that come in.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Tseehn Marhn
Rat Duck
#193 - 2012-06-30 21:25:54 UTC
Barl Rathbone wrote:
I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier.


I've been trying to think of a way to find completed orders as well. An idea I had was this.

Consider that the market order data is uploaded as lists of orders, and these lists are grouped by items in a region. If there is an order id in the database for some item in some region, and that order id is no longer appearing in the uploaded lists of orders for those items in that region, likely the order has been filled.

This assumes that every uploaded list of market data for an item in a region contains every order available in that region; I'm not 100% sure it does.
Barl Rathbone
Corel Industries
#194 - 2012-07-01 00:28:02 UTC
Tseehn Marhn wrote:
Barl Rathbone wrote:
I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier.


I've been trying to think of a way to find completed orders as well. An idea I had was this.

Consider that the market order data is uploaded as lists of orders, and these lists are grouped by items in a region. If there is an order id in the database for some item in some region, and that order id is no longer appearing in the uploaded lists of orders for those items in that region, likely the order has been filled.

This assumes that every uploaded list of market data for an item in a region contains every order available in that region; I'm not 100% sure it does.


Which is what I do - but it would be nicer if a message was generated that I could trust a bit more. :) And your data is only good as long as people check that item in that region with the uploader running. Not a problem for trade hubs but you end up with stale data at times in less frequented region/item combinations which can be slightly annoying.
Desmont McCallock
#195 - 2012-07-01 08:11:00 UTC
Absence of order id can also mean that that order was canceled or expired.
Tseehn Marhn
Rat Duck
#196 - 2012-07-01 17:43:23 UTC
Barl Rathbone wrote:
Tseehn Marhn wrote:
Barl Rathbone wrote:
I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier.


I've been trying to think of a way to find completed orders as well. An idea I had was this.

Consider that the market order data is uploaded as lists of orders, and these lists are grouped by items in a region. If there is an order id in the database for some item in some region, and that order id is no longer appearing in the uploaded lists of orders for those items in that region, likely the order has been filled.

This assumes that every uploaded list of market data for an item in a region contains every order available in that region; I'm not 100% sure it does.


Which is what I do - but it would be nicer if a message was generated that I could trust a bit more. :) And your data is only good as long as people check that item in that region with the uploader running. Not a problem for trade hubs but you end up with stale data at times in less frequented region/item combinations which can be slightly annoying.


Ah, well, good to know it should work. Also good to know no order ID means dead order.
Barl Rathbone
Corel Industries
#197 - 2012-07-01 19:46:22 UTC
Tseehn Marhn wrote:


Ah, well, good to know it should work. Also good to know no order ID means dead order.


Mind you I wouldn't trust a single order message -- I aggregate orders over time and then process through so I have a better shot of getting a valid "hit" rather than invalid data. It is all user uploaded stuff so you have to take it all with a grain of salt. :)
Barl Rathbone
Corel Industries
#198 - 2012-07-01 19:47:13 UTC
Desmont McCallock wrote:
Absence of order id can also mean that that order was canceled or expired.


True - but the point is still valid - that it's not an active buy/sell order any longer. The reason is opaque but the net effect is the same.
Ydnari
Estrale Frontiers
#199 - 2012-07-02 17:41:25 UTC
EMDR plus EveMon's uploader is good stuff.

I used to parse exported market order files to keep prices up to date in a private industry calculator and order tracking site I wrote, now I have that connected to EMDR and just have to look at items on market in game for it to update prices a few seconds later.

And now having access to price history is icing on the cake Cool

Thanks to all the developers involved.

--
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#200 - 2012-07-03 18:53:40 UTC
Ydnari wrote:
EMDR plus EveMon's uploader is good stuff.

I used to parse exported market order files to keep prices up to date in a private industry calculator and order tracking site I wrote, now I have that connected to EMDR and just have to look at items on market in game for it to update prices a few seconds later.

And now having access to price history is icing on the cake Cool

Thanks to all the developers involved.

We're glad you like it. Feel free to stop by #emdr on irc.coldfront.net anytime!

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com