These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#41 - 2012-04-19 14:18:29 UTC
Snarf Aldes wrote:
I am already converting to unified format, so that's not a problem.
I do however see some significant overhead with posting it to EMDR. If i can do it with ZMQ, that should be more efficient.

I had to disable the EMDR integration for now,i'm getting weird data, probably because I made a mistake somewhere with the parsing. I have to look into that.

EDIT:
I have looked at my code, but can't find anything wrong with it. I haven't saved the raw json i'm getting from emdr, so i can't look at that. I am seeing mixed up data, like someone tried to upload 2 rowsets in a single message, but instead of making 2 rowsets the rows of both items got merged.


I'd suggest one of the two solutions:

  • We can replace your current upload processor script with a standard EMDR Gateway. I'd need to add a config flag to allow you to change uploader paths to mirror your current one, but that's not a huge deal.This would accept HTTP uploads, and relay them to EMDR's brokers.
  • You can send me the JSON that you are relaying so I can put it through its paces in my test environment.


The first one requires some setup work, but is the most likely to work indefinitely. My aim is to eventually make this as easy as possible, so let me know what I can do. Walking you through finishing your uploader relay off with ZeroMQ isn't a big deal, as long as we're sure you're passing in good data.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#42 - 2012-04-19 15:36:56 UTC
Someone is sending malformed json data into the system. Their data has merged the rows section of a bunch of items under the typeID of a single item.
I have put the data online: bad_json.txt
The first item, typeID 3396 ('Industrial Construction' skillbook) has only 20 sell order and 11 buy orders in The Forge at the moment, the json contains 456 orders total.



Creator of Eve Addicts
Desmont McCallock
#43 - 2012-04-19 16:03:28 UTC  |  Edited by: Desmont McCallock
Why is the 'generator' unknown? Who is feeding data without stating who he is?
I would suggest those to be discarded.
Desmont McCallock
#44 - 2012-04-19 16:13:57 UTC
EMDR Client in Java. Please also add in "Using Data from the EMDR network" page.

Don't expect me to write the client in 'Scala' too. I have enough with all these programming languages I already know. Although I'll try to write it in C++ but I'm not promising anything.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#45 - 2012-04-19 16:20:16 UTC
Snarf Aldes wrote:
Someone is sending malformed json data into the system. Their data has merged the rows section of a bunch of items under the typeID of a single item.
I have put the data online: bad_json.txt
The first item, typeID 3396 ('Industrial Construction' skillbook) has only 20 sell order and 11 buy orders in The Forge at the moment, the json contains 456 orders total.


Yuck. I'll take a look at this as soon as I'm done at the day job.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#46 - 2012-04-19 16:20:36 UTC
Desmont McCallock wrote:
EMDR Client in Java. Please also add in "Using Data from the EMDR network" page.

Don't expect me to write the client in 'Scala' too. I have enough with all these programming languages I already know. Although I'll try to write it in C++ but I'm not promising anything.


Thanks! I'll get this added to the docs after work.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#47 - 2012-04-19 19:36:50 UTC
Snarf Aldes wrote:
Someone is sending malformed json data into the system. Their data has merged the rows section of a bunch of items under the typeID of a single item.
I have put the data online: bad_json.txt
The first item, typeID 3396 ('Industrial Construction' skillbook) has only 20 sell order and 11 buy orders in The Forge at the moment, the json contains 456 orders total.


There's actually good news on this front. This was a bug that I introduced with a late-night coding session, rather than something malicious: https://github.com/gtaylor/EVE-Market-Data-Relay/commit/62b49b293062d14999cb97ecf3127afb7d80e8ff#L0L33

I need another set of eyes on the data to make sure this fix is good. I'll write proper unit tests soon, had to get this in quickly between tasks at work.

Sorry for the hassle. Thanks for noticing this and letting me know, I greatly appreciate the help in testing this out.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#48 - 2012-04-19 22:26:12 UTC
Added Desmont's Java client example, and godo's Perl example.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#49 - 2012-04-20 04:17:54 UTC
We are now pulling recent data from EVE Market Data. Could definitely use some extra eyes on the feed to make sure the data coming through is good. It's orders only for now, looks like there's a bug in their history querying code.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#50 - 2012-04-20 05:46:47 UTC
Ilyk Halibut wrote:
Snarf Aldes wrote:
Someone is sending malformed json data into the system. Their data has merged the rows section of a bunch of items under the typeID of a single item.
I have put the data online: bad_json.txt
The first item, typeID 3396 ('Industrial Construction' skillbook) has only 20 sell order and 11 buy orders in The Forge at the moment, the json contains 456 orders total.


There's actually good news on this front. This was a bug that I introduced with a late-night coding session, rather than something malicious: https://github.com/gtaylor/EVE-Market-Data-Relay/commit/62b49b293062d14999cb97ecf3127afb7d80e8ff#L0L33

I need another set of eyes on the data to make sure this fix is good. I'll write proper unit tests soon, had to get this in quickly between tasks at work.

Sorry for the hassle. Thanks for noticing this and letting me know, I greatly appreciate the help in testing this out.

Great that it is fixed!
I'll enable the EMDR stream again for Eve Addicts.

Creator of Eve Addicts
Snarf Aldes
University of Caille
Gallente Federation
#51 - 2012-04-20 06:44:55 UTC  |  Edited by: Snarf Aldes
Hmm, there is another problem now.
A single rowset is split in 2 rowsets, one for buy orders, and one for sell orders. This is not according to the specs. I expect to see all data (both buy and sell orders) in the same rowset.

The problem with this is, that when i process the data, i pull the uploads from the db and ordered from new to old. Then in process them in order. If i find a duplicate for an item later in the set, i discard it because a newer version was already processed.

With the split data, only half will get processed.

Edit:
Apparently the split data is comming from Eve Marketdata:
Quote:
{
"uploadKeys": [
{
"origin": "EVE Market Data"
}
],
"generator": {
"version": "0.1",
"name": "EMDR EVE Market Data API Feeder"
},
"currentTime": "2012-04-20T08:29:29",
"resultType": "orders",
"version": "0.1alpha",
"rowsets": [
{
"typeID": 4272,
"rows": [
[
3999999.99,
5,
32767,
2445885201,
5,
1,
false,
"2012-02-20T07:43:05",
90,
61000363,
30004608
]
],
"regionID": 10000058,
"generatedAt": "2012-04-20T08:29:29"
}
],
"columns": [
"price",
"volRemaining",
"range",
"orderID",
"volEntered",
"minVolume",
"bid",
"issueDate",
"duration",
"stationID",
"solarSystemID"
]
}

uploadKeys is also not formatted as in the spec with a "name" and "key" pair, but that is not causing me any problems.

Creator of Eve Addicts
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#52 - 2012-04-20 13:53:10 UTC
Yeah, I think I'm going to need to pull, queue, and re-combine data from their API after some more shuffling to get it mixed together nicely. As you noted, the orders are separated.

In the meantime, ordering by the generatedAt attribs may help, as that is when the data came to be. That may be a more reliable means than what order it came into your uploader.

I'll fix the uploadKeys, too.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#53 - 2012-04-20 14:25:07 UTC
uploadKeys issue has been corrected to match more closely to the spec. Here are the two coming in from the feeder services (EMD and EC):

EMD feeder
{'name': 'EVE Market Data', 'key': 'EMDR'},

EC feeder
{'name': 'EVE Central', 'key': 'EMDR'},

I'd like some feedback on these, as the spec is kind of vague what the key is meant for. Is this acceptable for everyone?

Also, generator is going to be off for direct uploads until I have more time tonight. You may see this in the case of a direct upload:

{'name': 'Unknown', 'version': 'Unknown'}

If you see that, it's almost certainly an uploader. I'll get the correct information plugged in tonight, I just have to read up on EMK's upload spec again.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#54 - 2012-04-20 15:30:02 UTC
I believe "key' is to be used to identify the 'uploader' (in giving credits for view in the stats pages).
For example, if you use a site's market scanner with the igb to upload orders, that key is an ID generated by the site bind to that character (and the site knows where to give credits to).
In case of an 'uploader' the key is bind to the app.

At least that's how I understand it.

I strongly believe that the 'key' should remain an integral and the use of strings should be avoided.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#55 - 2012-04-20 15:33:34 UTC
I can get that changed later tonight, though, wouldn't it be up to the site who generated the key to decide whether to use straight numeric, or alphanumeric (hash?) keys? I could see value in not restricting to numerals, though, this is more of a decision for the specification thread, I think, as the current spec say: "This can be a simple ID number, or some kind of message digest"

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#56 - 2012-04-20 15:39:50 UTC  |  Edited by: Desmont McCallock
Here is what EMDR is currently generating:
Quote:
"uploadKeys": [
{
"origin": "EVE Market Data"
}
],
"generator": {
"version": "0.1",
"name": "EMDR EVE Market Data API Feeder"
},
Which I believe doesn't correspond to the specs.

The correct format should be
Quote:
"uploadKeys": [
{
"name": "EVE Market Data", "key" : "an id that you use internally"
}
],
"generator": {
"name": "EVE Market Data Relay",
"version": "the version of production code"
},

Quote:
I can get that changed later tonight, though, wouldn't it be up to the site who generated the key to decide whether to use straight numeric, or alphanumeric (hash?) keys? I could see value in not restricting to numerals, though, this is more of a decision for the specification thread, I think, as the current spec say: "This can be a simple ID number, or some kind of message digest"

Edit: Hash could also do, just not use descriptive strings. I know it says so but my opinion is not to use it for descriptive messages.
Snarf Aldes
University of Caille
Gallente Federation
#57 - 2012-04-20 17:00:45 UTC
Ilyk Halibut wrote:
Yeah, I think I'm going to need to pull, queue, and re-combine data from their API after some more shuffling to get it mixed together nicely. As you noted, the orders are separated.

In the meantime, ordering by the generatedAt attribs may help, as that is when the data came to be. That may be a more reliable means than what order it came into your uploader.

I'll fix the uploadKeys, too.

IMHO, this should be done by Eve Market Data, they should upload according to the spec. You can't correct the uploads of everyone who decides to ignore the specs...

Creator of Eve Addicts
Snarf Aldes
University of Caille
Gallente Federation
#58 - 2012-04-20 17:06:06 UTC
The whole uploadKey section is pointless. You can't use it for a point system since you can't tell if you already counted that specific piece of data via an other route...
Like i said before, you need to know where the data was sent, so you can tell if you have already seen it or not.

Creator of Eve Addicts
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#59 - 2012-04-20 17:07:31 UTC
With EMD, I'm actually looking at their API that shows the a list of recently updated orders/history, then querying for those items in the given regions. I then construct the payload to POST to EMDR, much like a player's uploader would. So I'm fully at fault for the silly grouping you're seeing.

I've got a fix in mind, so this will be resolved tonight, fo sho.

Also, a bug with EMD's API was fixed that will allow me to start syndicating history entries, instead of the current, just-orders stream. I'll also be getting an EVE Marketeer feeder set up, meaning we'll have syndicated data from all three sites. After that point, we'll focus on hammering out any remaining issues, then work on getting this thing distributed to other volunteers' machines.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#60 - 2012-04-20 17:13:09 UTC
Snarf Aldes wrote:
The whole uploadKey section is pointless. You can't use it for a point system since you can't tell if you already counted that specific piece of data via an other route...
Like i said before, you need to know where the data was sent, so you can tell if you have already seen it or not.


This may eventually be something that we just have to add, and call it Unified Uploader Interchange+routing. I hate to diverge, but agree that it's a problem.

Unfortunately, for it to work, we'd also have to have the uploader clients follow suite. While I do plan on eventually writing a "pure" EMDR uploader client (for the sake of simplicity, and better cross-platform support), we're kind of in a rough spot without support from the rest of the people involved with the specifications.

Perhaps the need for such a thing will get more attention as we get EMDR fully deployed and market sites start using it. In the end, the market sites and uploader client developers have the most sway.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com