These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Barl Rathbone
Corel Industries
#121 - 2012-05-18 17:28:46 UTC  |  Edited by: Barl Rathbone
Ilyk Halibut wrote:
Yikes, that is probably a silly mistake in my EVE Central SMTP feeder. I've tweaked this, can you see if it's acting as expected for you now?


Nope - still seeing it as of right now. (just emptied my DB and grabbed fresh data)

E.g:

Eve Market Data:

[58999.03,2,32767,2539306093,3,1,false,"2012-05-16T05:54:48+00:00",90,60005686,30002053]

Eve Central:

[58999.03,2,32767,2539306093,3,1,true,"2012-05-16T05:54:48+00:00",90,60005686,30002053]
Desmont McCallock
#122 - 2012-05-18 17:59:20 UTC  |  Edited by: Desmont McCallock
Just a heads up. EVEMon 1.7.0 which will contain the EMUU will go beta on the 26th. So get prepared.
Release is planned for June 10th if things go well.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#123 - 2012-05-18 18:00:59 UTC
Desmont McCallock wrote:
Just a heads up. EVEMon 1.7.0 which will contain the EMUU will go beta on the 26th. So get prepared.

Cool, looking forward to it. We'll be ready.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#124 - 2012-05-18 19:10:40 UTC
Barl helped me work out the aforementioned issue, so everyone can rest easy.

  • Correctly parse the is_bid column of the EVE Central SMTP feed. Was previously matching to 'True', whereas it should be '1'. I'm not sure if I got this wrong from day 1, or if it was a change introduced with the recent addition of columns to EC's SMTP feed. It's most likely my own mistake.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#125 - 2012-05-22 00:47:34 UTC
Linode is conducting some short-notice maintenance. We haven't grown past our single-node setup yet, so this will result in a short (less than 30 minute) outage. We'll be moving to a multi-node setup this week, so this will not be a problem in the future. Here's the text from Linode:

Quote:

Hello,

This is an update to remind you that scheduled maintenance will be occurring for the host your Linode resides on shortly. The time window for the start of the maintenance is:

Monday, May 21st 2012 @ 21:00 - 23:59 (9 PM - midnight) EDT
(Tuesday, May 22st, 2012 @ 01:00 - 04:00 UTC/GMT)

When the maintenance begins, your Linode will be gracefully powered down. We're expecting the maintenance to be completed in under 30 minutes, at which time your Linode will be queued to return to its previous state (booted or powered off).

There is no need to issue any jobs for your Linode as part of the maintenance. We will update you via this ticket once the maintenance has been completed.

We appreciate your patience and understanding as we work to maintain our infrastructure. Please let us know if you have any questions or concerns.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#126 - 2012-05-22 02:43:59 UTC
Went over without a hitch. Let me know if anyone's consumers didn't re-connect automatically, it should have all recovered by itself.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Barl Rathbone
Corel Industries
#127 - 2012-05-22 14:11:36 UTC
Ilyk Halibut wrote:
Went over without a hitch. Let me know if anyone's consumers didn't re-connect automatically, it should have all recovered by itself.


Ironically my node went through maint at the same time so I can't comment.
Otaci
Tides of Silence
#128 - 2012-05-24 11:06:22 UTC
Ilyk Halibut wrote:
Went over without a hitch. Let me know if anyone's consumers didn't re-connect automatically, it should have all recovered by itself.

My consumers connected back automatically with no problems.

I have some stats on EMDR messages here: http://www.eve-markets.com/emdr/
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#129 - 2012-05-24 19:36:57 UTC
Otaci wrote:
I have some stats on EMDR messages here: http://www.eve-markets.com/emdr/

Very cool, thanks for sharing.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#130 - 2012-05-24 20:34:35 UTC  |  Edited by: Desmont McCallock
Otaci wrote:
Ilyk Halibut wrote:
Went over without a hitch. Let me know if anyone's consumers didn't re-connect automatically, it should have all recovered by itself.

My consumers connected back automatically with no problems.

I have some stats on EMDR messages here: http://www.eve-markets.com/emdr/

Pretty nice work there, Otaci. I'm sure that you will iron out any wrinkles the graph generation is making. Ilyk should really add a link to that page in EMDR home page.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#131 - 2012-05-25 03:55:22 UTC  |  Edited by: Ilyk Halibut
I'm finishing up the heavy lifting for the last major task left with EMDR. We need to support order lists that lack entries for an item in a particular region. IE: The item is not available in said region.

While I was tackling this, I decided to break my data structures and serialization/deserialization classes and functions out. I'll end up using these for some of my own projects, and figured other Pythonistas may as well.

I could really use some eyes on this: https://github.com/gtaylor/EVE-Market-Data-Structures

I'm really only interested in supporting Python 2.6 and 2.7 for now. I'll be writing documentation sometime soon, then replacing a good bit of the guts in EMDR with this. It's mostly just a ripping of the guts out of EMDR into its own package, so there is very little change, aside from supporting empty region+item combos.

Here's an example of an EMDR consumer that connects to the feed and parses every incoming message:

https://github.com/gtaylor/EVE-Market-Data-Structures/blob/master/examples/emdr_parser.py

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#132 - 2012-05-29 03:35:51 UTC
Leading up to our full-fledged launch, I have created an IRC room. This is mostly aimed at helping people set up, develop, and use consumers, gateways, and relays. I'll make a point of hanging out in here every day while working. I may not always be able to answer questions quickly, but if you speak up and hang around, I'll get you taken care of.

IRC Room:
irc.coldfront.net
#emdr

Also, if you'd like a more persistent form of communication, use the mailing list:

https://groups.google.com/forum/?fromgroups#!forum/eve-emdr

I'll be asking for volunteers over the mailing list shortly.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#133 - 2012-05-29 05:34:36 UTC
The EVE Market Data Structrures project (which EMDR will be using soon) saw a lot of work tonight:

  • Will use ujson, simplejson, or json module (in that order) if present.
  • Improved unit test coverage.
  • Added iterator methods for all data structures.
  • Added length calculation for all data structures.
  • Added __contains__ to all data structures.

This is all mostly basic stuff, made slightly less trivial due to internal accounting within each data structure. Rather than leaving you hanging on figuring it out, we'll just provide these for you.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#134 - 2012-05-29 07:54:24 UTC  |  Edited by: Desmont McCallock
One thing I haven't understood, from the beginning in your code logic, is why are you decoding and then encoding again a UF message.

In your shoes I would just take the message, compare it to the specs to see if it satisfies all fields (validation), and if it does send the same message to the announcer. If the "currentTime" field is your concern, I don't thing that the validation logic takes more than a few millisecs, so that field is not necessary to be re-generated.

The validation logic should consist of checking that all fields are of the specs type, like

Field | Type | Value
result | string | orders/history
version | float | indifferent
uploadKeys | dict | contains name and key and are of type string
etc.

Edit: The above suggestion comes from the fact that I have changed the spec version number to "0.1" but I get back the message with version "0.1alpha" which is a violation of the specs (as the forwarder has altered a field it shouldn't).
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#135 - 2012-05-29 14:25:14 UTC
Desmont McCallock wrote:
One thing I haven't understood, from the beginning in your code logic, is why are you decoding and then encoding again a UF message.

We've discussed this before, and I understand you don't like the approach, but I think it's a better fit in the long run. It'll be staying that way until a reason presents itself to act otherwise.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#136 - 2012-05-29 20:56:10 UTC
I'm going to go on ahead and open this up in a super limited capacity, but EMDR is in need of one trusted volunteer to run a European gateway. Since EMDR is not yet packaged up in an easy-to-install form, I will need to be able to set it up and have a means of updating it without harassing the owner of the machine.

Here are some more specifics on what I'm looking for:

  • While EMDR doesn't currently use much bandwidth at all (about 2-3 GB a day, in/out combined), this will change as we grow. A gateway host can't be on a tight bandwidth budget.
  • A stable, speedy connection is a must-have.
  • The box does not need to be a behemoth, but I'd like to have at least 512 MB of RAM available (712+ preferred) for EMDR's use. This is a good bit more than we currently use, but again, data growth, and additional features may bring this total up.
  • You must be OK with the fact that running a gateway does cost money, and you will be harassed if the machine goes down.
  • I'm currently only interested in running these on Ubuntu 11.10 and higher (12.04 would be preferred, especially in the case of a VM). This will eventually be extended to Debian, but I'm not ready to deal with that yet.

I've had a ton of people asking about helping out, which is awesome. For now, I need just one super-stout go-to-guy/machine to help test distributing the uploads across multiple gateways. If you are serious about volunteering, send me an EVEMail with the following details:

  • The geographic location of your machine.
  • The name of the service provider.
  • A brief description of the machine's net connection.
  • Is this a VM or a bare metal machine?
  • The version of Ubuntu it is running.
  • Is anything else running on the machine that would cause resource contention?
  • Are you OK with handing over sudo access?

Thanks in advance, and I'm super psyched to see us start to weave our web of high availability.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#137 - 2012-05-31 02:37:53 UTC
To prepare for my absence next week, we have ramped up to add a second HTTP gateway for incoming uploads. If I have time tomorrow, we'll bring up the redundant relay for you to connect to, for those that are availability-conscious.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Tseehn Marhn
Rat Duck
#138 - 2012-06-05 05:46:59 UTC
Just started looking into building a real-time market analyzer, and this caught my interest.

I've done zero work with any EVE market apis before, so just a quick newbie question:

What, exactly, is in the data dumps? Current universe-wide buy and sell orders? I'm not too worried about historical data just yet, but a list of all current buy/sells would be perfect.
Otaci
Tides of Silence
#139 - 2012-06-05 21:46:03 UTC
The market data being discussed in this thread are copies of what you see in-game when you look at the market for a particular item. They are the buy and sell orders for a particular item in a particular region.

An example of an EMDR message is here: http://www.eve-markets.com/msgrecvd/view/0000902f-a8a0-4958-b028-253ea6495663

Its for typeid 8585 (Medium S95a Partial Shield Transporter), in region 10000042 (at the bottom).

You can see the sell orders (7th row is false) and buy orders (7th row is true).

There are thousands of these messages per hour.
Tseehn Marhn
Rat Duck
#140 - 2012-06-06 06:33:52 UTC  |  Edited by: Tseehn Marhn
I see. So the EMDR is a stream of all the individual player uploads, collected from the various websites.