These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Snarf Aldes
University of Caille
Gallente Federation
#21 - 2012-04-16 16:14:20 UTC
Also, users like statistics.
With emdr there is no way to identify users, so no way to count their uploads.
For my site there is just a list of the top uploaders, but for eve-marketeer there is a point system attached to the number of uploads that let you do extra stuff on the site.

No idea how to fix that... personally i don't care about stats, but i do catch myself looking at the ranking every now and then.

Creator of Eve Addicts
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#22 - 2012-04-16 19:08:14 UTC
Snarf Aldes wrote:
Also, users like statistics.
With emdr there is no way to identify users, so no way to count their uploads.
For my site there is just a list of the top uploaders, but for eve-marketeer there is a point system attached to the number of uploads that let you do extra stuff on the site.

No idea how to fix that... personally i don't care about stats, but i do catch myself looking at the ranking every now and then.


EMDR preserves the whole Unified Uploader message. Unified Uploader Interchange format supports uploaderKeys, which are used for this exact purpose. It's a matter of the upload clients using it.

But we are prepared, and I wholeheartedly agree that we need to preserve and relay such information to you. I'd love to see a breakdown of which site has the most people providing data to EMDR, as well. I've got some preliminary tracking in, but we're currently only being constantly fed by EVE Central.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#23 - 2012-04-16 19:48:38 UTC
How does eve-central upload data?

When i was looking what was coming out of emdr, i basically saw only the data that i was uploading to you myself.
Until at some point i got a burst of some MB's in data.
Since i was looking at it on stdout i decided to kill the script :)

Looks like they don't send it through as soon as they get it, but batch it and send it at a regular interval?

Creator of Eve Addicts
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#24 - 2012-04-17 03:32:45 UTC
We're subscribed to their SMTP-based batch service. This is currently the only option for real-time data feeds out there, which is one of the reasons EMDR came into being. The emails with market data come from EVE Central every 1-3 minutes, depending on some variability on their side. On our end, we parse the email and feed it back into EMDR.

This is probably why you saw the huge burst, and only the stuff you were uploading. I'm in discussion with some other market sites to get them to relay their data as well, but these things take time, especially when we're talking about a new service.

For those with market-driven sites, I'd love to help make it easy for you to share data! Please EVE Mail me and I'll work with you on it. It's very simple, and we all win.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#25 - 2012-04-17 22:34:36 UTC  |  Edited by: Desmont McCallock
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#26 - 2012-04-18 01:17:01 UTC
Desmont McCallock wrote:
EMDR Client in C#. Please add in "Using Data from the EMDR network" page.


Much appreciated! This has been added.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Nomad I
University of Caille
Gallente Federation
#27 - 2012-04-18 09:29:13 UTC
The idea to gather several feeds into one network is awesome.

1 question: Do you want to gather data for the network or will the markeddata sites push data into this network? The 2nd approach would be like a torrent network for data.
Snarf Aldes
University of Caille
Gallente Federation
#28 - 2012-04-18 15:33:35 UTC  |  Edited by: Snarf Aldes
Nevermind, i'm an idiot.... Typo's are not my friend.. Shocked

Creator of Eve Addicts
Desmont McCallock
#29 - 2012-04-18 15:45:45 UTC  |  Edited by: Desmont McCallock
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#30 - 2012-04-18 15:50:35 UTC
Nomad I wrote:
The idea to gather several feeds into one network is awesome.

1 question: Do you want to gather data for the network or will the markeddata sites push data into this network? The 2nd approach would be like a torrent network for data.


I'll be trying to convince market sites to relay their uploads to EMDR. Right now, the project is very young, so it's somewhat of an uphill battle. I welcome all the help I can get convincing the players that be that it's worth their time.

Ideally, each market site would re-POST every unique upload that comes to them via a player to EMDR, or hook into the top end of the feed with ZeroMQ and do it in real-time. The first method is the simplest, the second one is the most efficient. I've been in contact with EVE Marketdata and EVE Marketeer, so let's hope they'll come around!

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#31 - 2012-04-18 18:44:56 UTC
Desmont McCallock wrote:
EMDR Client in VB. Please also add in "Using Data from the EMDR network" page.


Thanks again, this has been added. We've got quite a few of the major languages covered.

Anyone up for Java? Scala? Anything on here http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#32 - 2012-04-18 20:16:49 UTC
Ilyk Halibut wrote:
Nomad I wrote:
The idea to gather several feeds into one network is awesome.

1 question: Do you want to gather data for the network or will the markeddata sites push data into this network? The 2nd approach would be like a torrent network for data.


I'll be trying to convince market sites to relay their uploads to EMDR. Right now, the project is very young, so it's somewhat of an uphill battle. I welcome all the help I can get convincing the players that be that it's worth their time.

Ideally, each market site would re-POST every unique upload that comes to them via a player to EMDR, or hook into the top end of the feed with ZeroMQ and do it in real-time. The first method is the simplest, the second one is the most efficient. I've been in contact with EVE Marketdata and EVE Marketeer, so let's hope they'll come around!

Do you have an example of how to accomplish it with ZMQ (in PHP if possible)?

Eve Addicts is now integrating the data recieved from EMDR.
I have an uploader ready, just need to tweak it a little, so soon I will be forwarding all direct uploads.

Creator of Eve Addicts
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#33 - 2012-04-18 20:33:44 UTC
Snarf Aldes wrote:

Do you have an example of how to accomplish it with ZMQ (in PHP if possible)?


I don't just yet, and your data would need to be in Unified Uploader format already (instead of EMK). That means converting everything you've got coming in to Unified format, which is kind of tedious. I didn't think about this before mentioning the ZMQ possibility, which was foolish.

There's enough of an efficiency win with ZMQ to where I probably do need to make a ZMQ gateway to complement the HTTP gateway. This would do all of the conversion so you don't have to. I can put this on the list of things to work on and let you know when I have something to show. This may be a post-launch task, though, as I've got a good bit more to do leading up to "1.0".

For now we'll probably just need to go the route of re-POST'ing to the HTTP gateway. In 99% of the cases, it should be as easy as taking the request you received and just relaying it to EMDR's HTTP gateway. If there's anything I can do to provide additional details on this, don't hesitate to ask. Here is what the clients get for uploader instructions, your relay would basically work the same way:

http://www.eve-emdr.com/en/latest/uploading.html

Snarf Aldes wrote:

Eve Addicts is now integrating the data recieved from EMDR.
I have an uploader ready, just need to tweak it a little, so soon I will be forwarding all direct uploads.


Awesome on both accounts. Let me know if you have any questions that I can help with. I definitely want to make relaying the data as easy as possible.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#34 - 2012-04-18 22:47:45 UTC
For the other Pythonistas, I've got an example of a Python greenlet consumer here:

https://github.com/gtaylor/EVE-Market-Data-Relay/tree/master/examples/python/greenlet_consumer

This is more advanced than the consumer in the docs, as it handles each incoming message in a greenlet. This allows your consumer to handle large amounts of market data without blocking while saving to the DB backend of your choice. You can still choke your consumer with CPU-intensive tasks in the worker, so watch that.

I've created an examples dir in the repo where people are welcome to contribute code in their choice of language for whatever purpose. If you've got something that you'd like to contribute, let me know and I'll work it in.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Dragonaire
Here there be Dragons
#35 - 2012-04-18 23:13:25 UTC
Just thought someone here might also be interested in some code I started on for the unified format stuff done in PHP. https://sourceforge.net/projects/evemarketfeed/

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#36 - 2012-04-18 23:26:49 UTC
Dragonaire wrote:
Just thought someone here might also be interested in some code I started on for the unified format stuff done in PHP. https://sourceforge.net/projects/evemarketfeed/


Awesome, we'll have to find somewhere for stuff like this in the EMDR documentation. Your timing is perfect, in that I just put this together:

https://github.com/gtaylor/EVE-Market-Data-Relay/tree/master/examples

The basic gist is that I'd love to collect code examples of EMDR-related things in various languages. If you write anything neat using EMDR, we can toss it in examples for others to re-use. Past a certain size (like with Dragonaire's evemarketfeed), it makes sense to have separate projects, but for anything smaller and self-contained, send a pull request or post to our issue tracker.

Are there any other resources people would like to mention as potentially useful for use with EMDR? Perhaps I can cull some of the stuff from EVE-Dev that would be appropriate.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#37 - 2012-04-18 23:31:10 UTC
I have also added some hints for those with market sites who wish to share their data over EMDR:

http://www.eve-emdr.com/en/latest/uploading.html#syndicating-your-market-site-s-upload-data-to-emdr

The easiest route is to just get in touch with me about running a gateway on your infrastructure in place of your current upload processing code, then subscribe to the EMDR feed. Since EMDR is still young, I understand that most people probably won't be comfortable with this just yet, so I do also briefly outline the steps needed to re-POST the data to our HTTP gateway.

If anyone ends up syndicating data, let us know of any trials and tribulations you run into. Code samples are also appreciated.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Rosaki
Blackman Industries
#38 - 2012-04-19 05:49:08 UTC
So, I bastardized a three.js example to make a WebGL powered EMDR upload activity map

http://map.eve-emdr.com

Systems will light up whenever market data is reported. Upload some data and watch your region flash, or just wait for the eve-central data to spam.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#39 - 2012-04-19 05:53:59 UTC
Rosaki wrote:
So, I bastardized a three.js example to make a WebGL powered EMDR upload activity map

http://map.eve-emdr.com

Systems will light up whenever market data is reported. Upload some data and watch your region flash, or just wait for the eve-central data to spam.


Looks awesome. Apologies for there being such large gaps between data bursts. We currently only have EVE Central's SMTP service feeding in 24/7, and that bursts every 1-5 minutes. I'll try to start pulling data from EVE Marketeer and EVE Market Data's API soon to get it lit up. If anyone else wants to syndicate uploads, shoot me a mail!

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#40 - 2012-04-19 07:11:02 UTC  |  Edited by: Snarf Aldes
I am already converting to unified format, so that's not a problem.
I do however see some significant overhead with posting it to EMDR. If i can do it with ZMQ, that should be more efficient.

I had to disable the EMDR integration for now,i'm getting weird data, probably because I made a mistake somewhere with the parsing. I have to look into that.

EDIT:
I have looked at my code, but can't find anything wrong with it. I haven't saved the raw json i'm getting from emdr, so i can't look at that. I am seeing mixed up data, like someone tried to upload 2 rowsets in a single message, but instead of making 2 rowsets the rows of both items got merged.

Creator of Eve Addicts