These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay

Author
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#101 - 2012-05-14 04:11:00 UTC
I'm hoping we're reaching the end of the line as far as disruptive changes goes. I definitely need some eyes on things after this round of changes, as we have changed how we handle decompression and parsing.

  • Decompression should now work properly (for real this time) for form-encoded POSTs. I'd strongly recommend against using form-encoded POSTs, as it's additional complexity, but the option is there for people with bad tools, or can't be bothered to read docs for their tools.
  • Added more error handling for providing feedback when badly formed form-encoded POSTs are encountered.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Desmont McCallock
#102 - 2012-05-14 06:08:50 UTC  |  Edited by: Desmont McCallock
After running some testing:

  • Method: POST (form-encoded) Compression: None Result: OK
  • Method: POST (form-encoded) Compression: GZip Result: OK
  • Method: POST (form-encoded) Compression: Deflate Result: OK
  • Method: POST (entity) Compression: None Result: OK
  • Method: POST (entity) Compression: GZip Result: OK
  • Method: POST (entity) Compression: Deflate Result: OK

EMDR will be added in EMUU endpoints in few.
Desmont McCallock
#103 - 2012-05-14 06:23:48 UTC
Now last thing that needs to be fixed is when uploading empty orders messages.
I tried to setup EMDR locally (on Windows machine) but after a days headbanging I gave up. Didn't expected that setting up a Python client would be so hard. What really got on my nerves was the fact that I had to download, build, install all those external libraries a Python project is using and then finding out that it still doesn't run.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#104 - 2012-05-14 14:26:33 UTC
Desmont McCallock wrote:
Now last thing that needs to be fixed is when uploading empty orders messages.
I tried to setup EMDR locally (on Windows machine) but after a days headbanging I gave up. Didn't expected that setting up a Python client would be so hard. What really got on my nerves was the fact that I had to download, build, install all those external libraries a Python project is using and then finding out that it still doesn't run.

Yeah, it can run on Windows, but considering that you've got a lot of extra work to do to install a compiler on Windows, it's probably not going to be something I want to officially support, even though it should be doable (in theory). As you saw, there are quite a few dependencies, gevent and libzmq being the ones needing a compiler.

I'll eventually write detailed instructions for Linux (which would be generic enough for the Mac OS users to follow), though. Once I've got the last few code rough spots fixed up, documentation will become my focus for the rest of the week. This will help make it possible to recruit people to run relays.

Something I have been looking into pyzmq-static:

http://pypi.python.org/pypi/pyzmq-static/

This is currently running an older version of pyzmq that isn't compatible with EMDR, but that bundles libzmq, which would reduce some of the annoyances you ran into.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Desmont McCallock
#105 - 2012-05-14 14:48:54 UTC
Actually it's zmq.green that's giving me the issues. It's not officially included in the distribution and I have to build, install it manually. Although I managed to do this at some time, still It through an exception on me with "Name _Socket not defined" in zmq\green\core.py
Barl Rathbone
Corel Industries
#106 - 2012-05-14 16:05:13 UTC
Ilyk Halibut wrote:
Barl Rathbone wrote:
Ilyk, if you still want/need another relay I can set up a new node for you on my rackspacecloud account. I assume it doesn't have to be a large one.

My rackspace stuff is all sourced out of the Chicago datacenter. Or if needed I could bring up a Linode, looks like that's what you're using currently.

I'll definitely be in need of relays in the near future. Been dealing with moving into a new house unexpectedly, which slowed things down for a while, but I'm back in action now. Aiming to get one or two more issues worked through this week, then do a fial documentation push, then look to get some relays set up.

Rackspace or Linode are both just fine. It just can't be Atlanta+Linode, since that wouldn't give us any useful additional redundancy. Data usage from Atlanta to another Linode location may be free, though.

I'll be sending the call out for relays on the forums, and on the mailing list: https://groups.google.com/forum/#!forum/eve-emdr

Keep an eye on whichever one works best for you. The mailing list is super, super low volume, as I've mostly been doing things here. That'll probably be more for important announcements once we're up and running.


Nah - my stuff is in ORD for Rackspace and EWR (newark) for Linode so you'd have physical redundancy.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#107 - 2012-05-14 17:41:49 UTC
Desmont McCallock wrote:
Actually it's zmq.green that's giving me the issues. It's not officially included in the distribution and I have to build, install it manually. Although I managed to do this at some time, still It through an exception on me with "Name _Socket not defined" in zmq\green\core.py


zmq.green is included as of pyzmq 2.2.0, make sure you upgrade to that. You'll need gevent as well.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Desmont McCallock
#108 - 2012-05-14 18:11:29 UTC  |  Edited by: Desmont McCallock
Unfortunately my friend, although it's in the repo, it's not included in the distributed msi or source zip files.
Never the less I managed to get the source code from the repo, build it, install it but I can't make the EMDR source code to see the package. Now I get "ImportError: No module named green". Every other requirement is installed and working.
Barl Rathbone
Corel Industries
#109 - 2012-05-15 16:51:20 UTC  |  Edited by: Barl Rathbone
Desmont McCallock wrote:
Unfortunately my friend, although it's in the repo, it's not included in the distributed msi or source zip files.
Never the less I managed to get the source code from the repo, build it, install it but I can't make the EMDR source code to see the package. Now I get "ImportError: No module named green". Every other requirement is installed and working.


I'm in the same boat with Desmont. I actually got the 2.2.0 egg built and installed but it broke pretty much everything in the process (e.g. now I'm getting undefined symbols on zmq_sendmsg when it tries to import the zmq.green lib or just zmq).

Frustrating to say the least. I'm trying to back everything out now and see if I can get my test harness working again before proceeding down the green path for a second try.

**EDIT**

Same error now as Desmont - no matter what it's not finding green.

** FINAL EDIT **

Got it working - I had the wrong tarball - cloned 2.2.0 from the git repo and rebuilt against zmq2.2.0 and it's now working. Jeeze!
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#110 - 2012-05-15 18:55:37 UTC
I think they're approaching a release, so hopefully it won't be as painful for much longer.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Barl Rathbone
Corel Industries
#111 - 2012-05-15 19:25:11 UTC  |  Edited by: Barl Rathbone
Ilyk Halibut wrote:
I think they're approaching a release, so hopefully it won't be as painful for much longer.


The problem is that you need the dev branch of 2.2.0, not the release tarball. If you clone it out of git it's fine, it includes the green files. If you use the release 2.2.0 tarball it doesn't contain green and (obviously) fails.

However, I'm now pulling data with worker threads. Next up, database insertion.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#112 - 2012-05-16 04:10:09 UTC
This will eventually be posted to the unified upload interchange format thread when it's in better shape, but I'm breaking all of the market data structures, along with serializers and de-serializers, out into their own project. Initially, this will just be for the unified format, but it'll be maintained in a manner to make it easy to drop others in if something new and better comes along.

https://github.com/gtaylor/EVE-Market-Data-Structures

There's going to be some pretty heavy lifting going on with this over the week, so I wouldn't use it for anything yet. It doesn't support the absence of orders/history for a given item+region ID combo yet, but that will be addressed shortly.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Barl Rathbone
Corel Industries
#113 - 2012-05-16 13:16:03 UTC
Ilyk Halibut wrote:
This will eventually be posted to the unified upload interchange format thread when it's in better shape, but I'm breaking all of the market data structures, along with serializers and de-serializers, out into their own project. Initially, this will just be for the unified format, but it'll be maintained in a manner to make it easy to drop others in if something new and better comes along.

https://github.com/gtaylor/EVE-Market-Data-Structures

There's going to be some pretty heavy lifting going on with this over the week, so I wouldn't use it for anything yet. It doesn't support the absence of orders/history for a given item+region ID combo yet, but that will be addressed shortly.


Nice! I was just working on my de-serializer but if you're going to publish one, hey why reinvent the wheel? :)
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#114 - 2012-05-16 14:25:25 UTC
I'll probably be doing some pretty heavy re-working of what I have so far, because it made some bad assumptions. However, I'm hoping to have it in good shape and unit tested this week.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Barl Rathbone
Corel Industries
#115 - 2012-05-16 20:19:06 UTC
Well in a very quick and dirty method I've got a process running pulling the full feed and stuffing it into my database. Very quick, very dirty but I wanted to load some data so I'll let it run for a while.

Nice work Ilyk. Very nice work!
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#116 - 2012-05-17 14:56:21 UTC
Barl Rathbone wrote:
Well in a very quick and dirty method I've got a process running pulling the full feed and stuffing it into my database. Very quick, very dirty but I wanted to load some data so I'll let it run for a while.

Nice work Ilyk. Very nice work!


Excellent, let me know if you see anything wrong. I'd like to have the data as scrutinized as humanly possible so we can catch anything before reaching "1.0".

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Barl Rathbone
Corel Industries
#117 - 2012-05-17 16:56:38 UTC
Ilyk Halibut wrote:
Barl Rathbone wrote:
Well in a very quick and dirty method I've got a process running pulling the full feed and stuffing it into my database. Very quick, very dirty but I wanted to load some data so I'll let it run for a while.

Nice work Ilyk. Very nice work!


Excellent, let me know if you see anything wrong. I'd like to have the data as scrutinized as humanly possible so we can catch anything before reaching "1.0".


It's a lot of data over time. My initial script to take in the data is pretty inefficient (straight up loop through the rows in each JSON message for inserts) and needs to be rewritten so it inserts all the rows in a JSON message at once rather than all the database connections being opened and closed.

So far the data base been pretty clean from what I've seen. I let the collector run for a few hours yesterday and didn't notice any problems that weren't on my side.

Let me know if you need another relay - I can have another linode or rackspace cloud box up and running quickly if needed.
Barl Rathbone
Corel Industries
#118 - 2012-05-18 16:11:51 UTC
I need to do some more research but I'm seeing some inconsistencies in the data. Unfortunately I didn't have my script save off the original data (a mistake soon rectified) for comparison but here's an example:

mysql> select * from marketdata where orderID = 2532530253;
+-------+-----------------+---------------------+----------+--------+-------------+-----------------+---------------+---------------+-------+------------+-----+---------------------+----------+-----------+---------------+------------+---------------------+
| id | uploadKeys | generatedAt | regionID | typeID | price | volumeRemaining | volumeEntered | minimumVolume | range | orderID | bid | issueDate | duration | stationID | solarSystemID | suspicious | dateAdded |
+-------+-----------------+---------------------+----------+--------+-------------+-----------------+---------------+---------------+-------+------------+-----+---------------------+----------+-----------+---------------+------------+---------------------+
| 27452 | Eve Addicts | 2012-05-18 00:00:52 | 10000043 | 30050 | 50001057.01 | 2 | 5 | 1 | -1 | 2532530253 | 1 | 2012-05-18 00:00:41 | 90 | 60008494 | 30002187 | NULL | 2012-05-18 10:33:50 |
| 31771 | EVE Market Data | 2012-05-18 15:34:17 | 10000043 | 30050 | 50001057.01 | 2 | 5 | 1 | -1 | 2532530253 | 1 | 2012-05-09 18:14:10 | 90 | 60008494 | 30002187 | NULL | 2012-05-18 10:35:14 |
| 45968 | EVE Central | 2012-05-18 15:34:55 | 10000043 | 30050 | 50001057.01 | 2 | 5 | 1 | -1 | 2532530253 | 0 | 2012-05-18 04:43:41 | 90 | 60008494 | 30002187 | NULL | 2012-05-18 10:35:52 |
| 48411 | EVE Central | 2012-05-18 15:34:56 | 10000043 | 30050 | 50001057.01 | 2 | 5 | 1 | 65535 | 2532530253 | 0 | 2012-05-18 04:43:41 | 90 | 60008494 | 30002187 | NULL | 2012-05-18 10:35:52 |
+-------+-----------------+---------------------+----------+--------+-------------+-----------------+---------------+---------------+-------+------------+-----+---------------------+----------+-----------+---------------+------------+---------------------+
4 rows in set (0.09 sec)

The problem is the bid flag - this is just a random example of the problem - but the first two orders are showing as bids but the latter two from Eve Central are not. I'm less concerned about the inconsistency on the range field since i'm not sure what use it is at this time. And yes, it's harder to read that table without a fixed-width font but [code] doesn't seem to work here.
Barl Rathbone
Corel Industries
#119 - 2012-05-18 17:05:16 UTC
I just went through a data dump comparing the raw JSON messages and either Eve Central has their export backwards or the other two sites do. I'm showing identical orders but the JSON has false for Eve Central and true elsewhere. For example, from Eve Central:

[11.18,100000,4,2241318813,100000,1,false,"2012-04-04T04:44:43+00:00",90,60009217,30003451]

and from Eve Market Data:

[11.18,100000,4,2241318813,100000,1,true,"2012-04-04T04:44:43+00:00",90,60009217,30003451]

that's kind of a problem. :)

On the flip side I got my feeder to use half the disk I/O it was doing before thankfully. Under 500 blocks/second now.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#120 - 2012-05-18 17:17:18 UTC
Yikes, that is probably a silly mistake in my EVE Central SMTP feeder. I've tweaked this, can you see if it's acting as expected for you now?

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com