These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Desmont McCallock
#141 - 2012-06-06 06:39:33 UTC  |  Edited by: Desmont McCallock
Tseehn Marhn wrote:
I see. So the EMDR is a stream of all the individual player uploads, collected from the various websites.

Currently it collects from player uploaders, EVE Market Data and EVE Central but the creator said it may stop collecting from market sites when EVEMon 1.7.0 gets released and player uploads cover the data EMDR is collecting from the market sites.
Barl Rathbone
Corel Industries
#142 - 2012-06-06 11:44:29 UTC
Tseehn Marhn wrote:
I see. So the EMDR is a stream of all the individual player uploads, collected from the various websites.


You got it. That's it exactly. Think of it is as a live ticker-tape of all the market data collected by players and uploaded via unified uploader clients.
Barl Rathbone
Corel Industries
#143 - 2012-06-06 16:04:34 UTC
Otaci wrote:
The market data being discussed in this thread are copies of what you see in-game when you look at the market for a particular item. They are the buy and sell orders for a particular item in a particular region.

An example of an EMDR message is here: http://www.eve-markets.com/msgrecvd/view/0000902f-a8a0-4958-b028-253ea6495663

Its for typeid 8585 (Medium S95a Partial Shield Transporter), in region 10000042 (at the bottom).

You can see the sell orders (7th row is false) and buy orders (7th row is true).

There are thousands of these messages per hour.


Otaci - are you storing all the JSON messages and breaking them down into separate databse fields? Just curious as to what you're doing on the backend with the EMDR messages and what you're running to do it.
Otaci
Tides of Silence
#144 - 2012-06-06 18:51:35 UTC
I am storing the JSON messages for now, but thats only because I'm still at the beginning with this web site. I need the originals to be able to track down problems and perfect my algorithms. One day I'll get around to writing a data pruning job, hopefully before the volume starts causing problems.

I am extracting the data, but I'm only keeping it in memory long enough to calculate statistics data, basically the data you see on my web-site. I'm not storing every row in a relational database. I'm using the Erlang language (http://erlang.org/).

To be honest, I'm a bit nervous about talking too much about how I am doing stuff because I've only just started with this web site and I'm concerned about competition.
Dream Five
Renegade Pleasure Androids
#145 - 2012-06-07 06:48:25 UTC
sounds incredibly cool.
Leolie Aylet
The Cybernetic Consciousness
#146 - 2012-06-07 19:54:11 UTC
Great job, awesome work !

Anyway it's possible to get buy/sell orders of a specific items in a Region/Station (a sort of filter by itemID,regionID, etc ... ) ?
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#147 - 2012-06-07 21:47:43 UTC
Leolie Aylet wrote:
Great job, awesome work !

Anyway it's possible to get buy/sell orders of a specific items in a Region/Station (a sort of filter by itemID,regionID, etc ... ) ?

No, that's not what EMDR is. EMDR is a dumb firehose of near-realtime data uploads from players. It aspires to be nothing more. Down the road, we may filter out obviously malicious payloads, or combine multiple payloads to save bandwidth/processing, but we'll always just be a dumb feed.

You'll want to hit one of the many market sites (that would sit on the receiving end of EMDR) for historical data, or record your own. EVE Central, EVE Marketdata, EVE Marketeer, EVE Addicts, and other sites all have historical data APIs. I'd consider using those if you don't want to record your own.

Hope that helps!

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#148 - 2012-06-07 21:49:52 UTC
I posted this to the mailing list, but figured I'd mention it here too. I've been away on vacation since last Friday, and will be gone until Sunday night. Normally I'd be a lot more responsive and involved in the discussion here, but this is a much-needed break from the intarwebs.

I hope all is going well, and will return in full force to push to "1.0" next week!

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#149 - 2012-06-10 15:44:13 UTC
I'l be damned. Big smile
The EMDR map is lighting up like a Christmas tree.
Stats are sky rocketing.
Otaci
Tides of Silence
#150 - 2012-06-10 16:36:54 UTC
Holy Smokes! Server load has rocketed.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#151 - 2012-06-10 21:59:30 UTC  |  Edited by: Ilyk Halibut
EMDR is holding up good so far. For VM with four virtual processors, we're sitting right around 1.17 load average on both gateways. This is excellent given the amount of traffic we're pushing.

I'm going to need a third Linode gateway volunteer, if anyone would like to donate one ($20/month). Same details apply as outlined earlier, but it'd probably need to be the only thing running on the VM. Please EVEMail me or punt gtaylor on #emdr on ColdFront IRC if you'd like to help.

For anyone comfortable with passing sudo access to me to install/maintain a relay, we also need relays. I understand granting access to some stranger is not appealing, so you're also welcome to wait until we're able to get EMDR packaged into a DEB (Ubuntu for now), but when that happens is dependent upon pyzmq's next stable release date. Until then, the setup process is non-trivial enough to where I don't want to have to walk people through it.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#152 - 2012-06-11 03:13:56 UTC
An end-of-first-night status update for any curious Georges. The gateways held up marvelously, and the announcers didn't break a sweat. Bandwidth usage on our gateways was a little higher than we'd prefer, but we'll be taking steps to improve on that shortly. We're in need of relays now, so EVEMail me, or hop in #emdr on ColdFront and get my attention there if you'd like to help.

As far as new changes tonight:

  • Dis-continued the EVE Marketdata and EVE Central feeder daemons. These were pulling market data from those two sites and stuffing it back into EMDR. Due to the large volume of data we're now getting directly, there's just no point, as we end up getting huge messages containing the same stuff (which is bad). This should reduce bandwidth and the number of dupes consumers are seeing.
  • Made some finishing touches to the relays. They now support some simple message-level de-duping, which means relays can now connect to one or more upstream announcers (or other relays) to allow for the self-healing EMDR hangs its hat on. For example, if your relay is connected to Announcer1 and Announcer2, the message streams are de-duped, so your consumers only see one of each upload. If Announcer1 goes down, everything will continue to work fine.
  • An option has been added to make the relay de-compress all outbound messages. No public relays should set this. This is only for private relays. For example, if you have two applications running on your server that need the data, instead of connecting to EMDR twice, you'd run your own relay with the decompression option enabled. This would save both of us bandwidth, and probably save yourself some CPU util (you'd only be de-compressing each message once, instead of twice).

Before I go quiet for some sleep, I'd love to see some people share their consumer code. The large volume of data we see now has brought some sites down due to engineering issues on their end, so we'll want to try to share some best practices.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#153 - 2012-06-11 04:57:55 UTC
I thought I was done for the night, but one more really important one. Make sure to read if you're currently consuming, or you may find yourself suddenly disconnected! We're shifting to what I consider a "1.0" deployment, and there's one last bit of breakage leading up to that. Read this:

https://groups.google.com/forum/?fromgroups#!topic/eve-emdr/BxpwU49PCds

We have our first proper relay up and running. We can now survive either of our two gateways/announcers failing without service interruptions. Thanks goes to udsaxman for volunteering the relay.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#154 - 2012-06-12 11:23:36 UTC
I was seeing a lot of old data being relayed by EMDR. With old i mean generatedAt times of 2 weeks in the past.
I am now filtering on generatedAt times, anything older than an hour is discarded (and doesn't have to be processed later).
This reduced the server load quite a bit.

Would be great if EMDR could filter on data freshness, that would reduce the bandwidth a bit. You probably want to have a more relaxed filter, like max age 2 days or something.

Creator of Eve Addicts
Desmont McCallock
#155 - 2012-06-12 11:36:13 UTC  |  Edited by: Desmont McCallock
This has nothing to do with EMDR and EMDR should not do any filtering as it's just a relay service.
The reason you are seeing old data is due to EVEMon 1.7.0 activation. Upon activation EVEMon is sending any data it finds in the EVE client cache (and deletes the files once it's done). I thought about filtering data in EVEMon before sending, but ended up that that's not EVEMon's responsibility as an uploader.
So this leaves the responsibility with the receiver, meaning the market aggregator.
Snarf Aldes
University of Caille
Gallente Federation
#156 - 2012-06-12 13:41:28 UTC
Desmont McCallock wrote:
This has nothing to do with EMDR and EMDR should not do any filtering as it's just a relay service.
The reason you are seeing old data is due to EVEMon 1.7.0 activation. Upon activation EVEMon is sending any data it finds in the EVE client cache (and deletes the files once it's done). I thought about filtering data in EVEMon before sending, but ended up that that's not EVEMon's responsibility as an uploader.
So this leaves the responsibility with the receiver, meaning the market aggregator.

I guessed that was the case, so it should solve itself in time.

Creator of Eve Addicts
Tseehn Marhn
Rat Duck
#157 - 2012-06-12 13:54:53 UTC
Finally got a VB application storing data to a local DB after screwing around with pathfinding algorithms for way too long (damn you D* Lite...).

I actually came to ask about the old records, but I see that that's on purpose.

I also had another question about the time format. Is there any chance of switching to something more computer friendly - like a timestamp or something? I'm sure I can convert the ISO 8601, but I'm not looking forward to it. VB is impressing me less and less the more I learn about it.

Also the price values. Sometimes they're coming through as ints, other times as floats. Is this on purpose, or is something on my end casting things funny?

Finally, I'd also like to volunteer for any coding you might need. I'm not a 1337 programmer by any means - but I have been doing it off and on as a hobby for nearly 20 years. You all seem to have it covered, and I appear to be out of your technology stack anyway, but just the same. I'm familiar with:

PHP/MySQL
HTML/CSS
JavaScript
Java
VB
More or less understand various flavors of C, C++, C#. I can learn. Once you know one language, the rest is pretty much syntax.

In any case, awesome show. Great job.
Desmont McCallock
#158 - 2012-06-12 14:05:17 UTC
Tseehn Marhn wrote:
I also had another question about the time format. Is there any chance of switching to something more computer friendly - like a timestamp or something? I'm sure I can convert the ISO 8601, but I'm not looking forward to it. VB is impressing me less and less the more I learn about it.
https://forums.eveonline.com/default.aspx?g=posts&m=974200#post974200
Tseehn Marhn wrote:
Also the price values. Sometimes they're coming through as ints, other times as floats. Is this on purpose, or is something on my end casting things funny?
All should be converted to decimal.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#159 - 2012-06-12 14:15:59 UTC
Tseehn Marhn wrote:

Also the price values. Sometimes they're coming through as ints, other times as floats. Is this on purpose, or is something on my end casting things funny?

This will be addressed in an upcoming update. Should be this week.
Tseehn Marhn wrote:

Finally, I'd also like to volunteer for any coding you might need.

The EMDR source is as at: https://github.com/gtaylor/EVE-Market-Data-Relay
the upcoming data structures and serialization module is at: https://github.com/gtaylor/EVE-Market-Data-Structures

Those can always use some eyeballs and an eye for refinement.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#160 - 2012-06-12 16:16:08 UTC
To better cope with the higher CPU demands that resulted from adding de-duplication to the relay daemons, I've written an alternative relay daemon in Go:

https://github.com/gtaylor/emdr-relay-go

This is currently completely unconfigurable and hasn't been prettied up, but it's what we're currently using on our relay. I have found that it drops CPU util by 50%, and significantly reduces memory consumption (something like 500% or more, depending on architecture and other factors).

If anyone is decent with Go, I could really use some eyes on this. I'm probably not doing everything as efficiently as I could be.

For everyone else, this is probably going to be the standard relay. It's faster, more efficient, and most importantly, it's easier to build/install.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com