EVE Technology Lab

EVE Market Data Relay is a publicly accessible, highly-available, highly-scalable, real-time market data feed. Players point their uploads at EMDR, and we relay it to anyone who would like to receive the data. All data is validated and relayed in Unified Uploader and Data Interchange Format to all connected consumers.

The idea is to lower the barrier to entry for developers who are working on, or thinking about working on market data-driven sites or applications (this is not limited to being useful on just websites). Instead of having to write boring pollers/scrapers/listeners for the various market sites, and/or get players to point their uploaders at you, EMDR takes care of that, giving you a firehose of data to subscribe to. Over time, we can point our users' uploaders at EMDR, and consolidate data, without some of the past concerns about stability and availability (our only single point of failure is Amazon's DNS service).

The "Using Data from the EMDR network" page shows how easy it is to get started consuming the real-time feed in a number of different languages.

For the curious/nerdy sorts, here's a High Level Overview of the project, as well as a glimpse into its innards and design.

If you are interested in, or are using EMDR, I highly recommend hopping on the mailing list. We are still in testing, things may change, and that's where you're going to get the heads-up.

Even better, join the #emdr IRC room on irc.coldfront.net and bark at me/us directly. Or just hang out, so I don't have to mutter to myself.

Project Status
EMDR has transitioned into what I would call very late beta. The only reason I haven't stuck a "1.0" label on it yet is that I'm being paranoid. We have over a dozen different regular consumers to the feed, and many more that appear and disappear throughout the course of the day.

Data Volume
Depending on activity and the day of the week, we receive anywhere from 1,000,000 to 3,000,000 uploads a day. Up-to-date traffic stats can be seen on eve-markets.com. The vast majority of these come from EVEMon's built-in uploader. We bear the brunt of the HTTP uploads so you don't have to, and instead feed you over a single, persistent, compressed connection. This makes that 1-3 million number a whole lot less of an issue, since you aren't being hammered by dozens of TCP connections every second (it adds up very quickly).

FAQ

• A centralized data source sounds neat, but what about reliability?
• EMDR is built in a way that eliminates all single points of failure (aside from Amazon's DNS servers). Here's a chart and a run-down showing a dual-stack model.


• Can I get historical data from EMDR?
• No. EMDR is only a dumb data firehose. We feed the sites that offer historical data APIs. They are the ones that serve this need, not EMDR. That said, if you find yourself constantly hammering other APIs to fill your DB, you should probably just consume EMDR and stop burdening other sites.


• What does it cost to use this service?
• Nothing. Additionally, if your application is in the same data center or service as one of our relays, you may not use any of your allotted bandwidth (depends on your provider).


• Why is this better than downloading X Market Service's daily data dumps?
• Depending on your usage case, it may not be better. EMDR is more useful to those who want timely updates to market data, and those who don't want to write software that pulls data from a number of different market sites.


• Why not just use EVE Central's SMTP data feed?
• EVE Central's SMTP service is the closest equivalent to EMDR. However, SMTP is horribly suited to the volume of data that we now deal with. Mails are not compressed, can be erratic as far as delivery times, and if EVE Central's SMTP servers go down, you're without data. EMDR is built from the ground up for high availability, and lets us deliver the data to you compressed, and in an easily readable/parsed format.


• Is the source code available for this project?
• Yes, and it's BSD licensed.


• I have some resources to volunteer. What do I do to help?
• We'd love to have you! See our Volunteering computing resources page.

Wow, this looks really ambitious. The technology stack is also sexy. I'd love to play around with ZMQ a bit.
However, I'm not sure how useful such a service will be for real world applications, as you don't seem to provide any kind of query methods to access already broadcasted orders. I'm not sure what kind of applications you had in mind when creating this. But being able to pull orders from a certain point back in time would certainly help for apps that won't be online 24/7 and need to keep in sync.

Another approach would be a http feed based solution such as:
http://code.google.com/p/pubsubhubbub/
Its based on HTTP and would allow more flexible access to orders by e.g. using URLs for different market categories or regions, access to archived data and caching and load balancing methods.

Such a service would probably better be accomplished by someone who is subscribed to our firehose, who stored and offered the data up to others. Perhaps one of the other major market sites will get on board eventually. However, historical data is outside the scope of EMDR. I'm not being stubborn, EMDR is just going to do one thing and do it really well. If someone wants to take the data and make it available on their market site, I will gladly point to them in the docs as a source of historical data that at least partially came through EMDR.



I guess it does technically rule out a lot of desktop/mobile apps, since they aren't online 24/7, but my primary intended audience is those that want near real-time access to market data. For that particular usage case, EMDR is synonymous to getting uploader data over HTTP, but much more fast, reliable, and efficient. In other words, this is probably most appropriate for websites, or real-time trading applications in the case of a desktop/mobile app (with historical backfilling from EVE Central or EMK/EMD).

Excellent. I quite literally just updated the uploading documentation to include specifics for Unified Format uploading. This may be useful for your test purposes.



If you let me know when you support the Unified Format, I'll add you to our list on the instructions page as a client that supports it, and may go as far as screenshotting the config process for our users.

That would be awesome. Thanks!

Holy batman, you use the same service I use! +1 for Linode. I do have to ask two questions tho;

1. Are you doing any error detection for bogus submissions that are not valid?
2. Which linode location are you located at, and why haven't you setup a alias for;
tcp://relay-linode-atl-1.eve-emdr.com:8050

Let me know, thanks!

The idea is very good, but in RL finance the pure real time feeds are a minority and mostly used for pure buy / sell trading platforms.

All the others (I'd say 95 percent of the existing platforms) have a mechanism to query for past data up to present or from time X to time Y.

Without past data storage many of the advanced trading tecniques cannot be used.

Example:

This whole thread.

What can be achieved with data feed + history:

Tritanium, on professional trading platform

Nitrogen Isotopes, on Microsoft Excel graph

Nitrogen Isotopes, with very clear price action

Technetium, on a semi-professional trading platform

There is data available from 2009 (already digested in various formats including industry OHLC, on EvEMarketeer.com) but also older data (EvE Central stored data).

For how EvE markets are structured and because of the lack of leverage, the minimum resolution you can really work with is daily prices.

Having a continuous stream is fantastic but without the ability to go back at least 2 years (even with just daily data) it really falls short.

I agree with Vaerah, without historical data this does not seem much different than being apart of the unified data uploader tool. Specifically because it introduces a different point of failure where a developer (like me) does not necessarily need a feed as much as snapshot of transaction history for the market.

Again, EMDR is a replacement for having uploaders pointed at your site directly. It is for market sites to get their users pointed at a central place in order to combine incoming data feeds. Instead of X player's uploads just going to EVE Central or EVE Marketeer, their uploads go to everyone. That is the point, not historical data.

If you need historical data and you're not storing any of your own, EMDR is not for you. There is an entry in the FAQ above that says as much. EMDR is more appropriate for the sites that store the feed. For sites that are just getting started out, I recommend connecting to EMDR, and backfilling data from EVE Central, EVE Marketeer, and EVE Market data as needed.

Hi,
I have been playing around with emdr over the weekend, and I like the idea.

I do however have some questions/remarks.

In an ideal world, everyone would upload only to emdr, and not to any of the market data sites. The sites would all get their data from emdr and would not have to supply any data to emdr.

Unfortunately, we do not live in an ideal world, so the situation will be something like this:

Users will upload to (multiple) market data sites and emdr, and market data sites will supply data to emdr (like eve-central is doing now).

Now consider the following situation, a user is uploading to a total of 4 endpoints: eve-central, eve-marketeer, emdr and eve-addicts.
Eve-central, eve-marketeer and eve addicts all push the uploads they are getting to emdr. Lets say they are smart and don’t push anything they get from emdr back into the relay system.

Eve-central, eve-marketeer and eve addicts all also subscribe to emdr.

Now the user uploads data for item X, I get it directly, and I get it an additional 3 times from emdr (once from the direct user upload to emdr and twice from the 2 other sites).

Currently there is no way for me to see that I have already received that data directly. So I have to accept and process all copies I get from emdr.

This can be resolved, but it requires a change in the json layout. If the uploading client software sends its destinations, for instance the urls it’s uploading to, then I as an endpoint can see if the data was already sent to me directly, and just discard it if it comes in again via emdr.

For example:
“destinations” : [
{“url”:”http://eve.addicts.nl/api/upload.php”}
{“url”:”http://www.evemarketeer.com/api/upload”}
{“url”:”http://master.eve-emdr.com/upload/eve_marketeer/”}
{“url”:”http://eve-central.com/datainput.py/inputdata”}
]

Destinations should not be deleted, only added to by intermediate sites if they upload it somewhere else.
If I would see that emdr is in the list, I don’t have to push it into the relay system, if it’s not I can decide to upload it after adding the url for emdr to the list.

This would ofcource only work for uploaders using the unified format.

So, the bottom line, I like the idea, but at the moment I’m a bit concerned about all the duplicate data that I will have to process if I scubscribe to the stream. Without a way to tell if you have already seen the data before, it will require a significant amount of processing to work through all the duplicates.

Snarf