These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#161 - 2012-06-12 21:05:16 UTC
Tseehn Marhn wrote:

Also the price values. Sometimes they're coming through as ints, other times as floats. Is this on purpose, or is something on my end casting things funny?

I just tried to reproduce this and couldn't. Are you sure they're not being parsed by your JSON parser as ints or floats? The JSON strings look correct, from the ones I sampled. Can you send a complete JSON string with the integers instead of the floats?

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Tseehn Marhn
Rat Duck
#162 - 2012-06-12 21:46:04 UTC
Desmont McCallock wrote:
https://forums.eveonline.com/default.aspx?g=posts&m=974200#post974200



I see. Ah well.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#163 - 2012-06-12 22:05:01 UTC
I'd like to open discussion up for dropping EMK upload support. Please discuss this on the mailing list thread if you have an opinion:

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Tseehn Marhn
Rat Duck
#164 - 2012-06-12 22:56:27 UTC  |  Edited by: Tseehn Marhn
Ilyk Halibut wrote:
Tseehn Marhn wrote:

Also the price values. Sometimes they're coming through as ints, other times as floats. Is this on purpose, or is something on my end casting things funny?

I just tried to reproduce this and couldn't. Are you sure they're not being parsed by your JSON parser as ints or floats? The JSON strings look correct, from the ones I sampled. Can you send a complete JSON string with the integers instead of the floats?


The parser is at fault. Any number without a decimal portion is cast as an int. Unless all prices should have the decimal portion. I'll let it run for a bit and see if I catch any more.

In the meantime I did catch some doubles, but I'm assuming you're not rounding prices to two places.

{"uploadKeys":[{"name":"BattleClinic","key":"3C3490E986D448B1E5379147115E887E4E045554"},{"name":"EVE Central","key":"0"},{"name":"EVE Market Data Relay","key":"0"}],"generator":{"version":"1.7.0.3809","name":"EVEMon.MarketUnifiedUploader"},"currentTime":"2012-06-12T22:27:22+00:00","resultType":"orders","version":"0.1alpha","rowsets":[{"typeID":19726,"rows":[[2.499800e+09,1,32767,2530794964,1,1,false,"2012-06-10T00:00:37+00:00",90,60000916,30000162],[2.600000e+09,1,32767,2558036643,1,1,false,"2012-06-04T00:00:48+00:00",90,60003046,30000162],[2.600000e+09,1,32767,2558036234,1,1,false,"2012-06-04T00:00:21+00:00",90,60001804,30000162],

I'll let it run for a bit and see if anything other than a decimal comes up.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#165 - 2012-06-12 23:53:41 UTC
Yeah, that looks like a language/parser/toolkit issue, which I'm not able to help with. I don't remember much at all about VB.

You'll probably want to explicitly cast the numbers, though. It sounds like your parser casts whole numbers (.0) as ints/longs, and everything else as decimals. Python does it differently, in that it's looking for the presence of a period in the JSON string to denote a float, and else, an int/long.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Tseehn Marhn
Rat Duck
#166 - 2012-06-13 00:20:52 UTC
Yeah, it's easily solved.

Convert.ToDouble(yourArrayData(index))
Desmont McCallock
#167 - 2012-06-13 06:03:38 UTC
Are you using the .NET native json parser?
http://msdn.microsoft.com/en-us/library/system.web.script.serialization.javascriptserializer.aspx#Y0
Btw, I'll just repeat, prices should be cast to decimal.
Tseehn Marhn
Rat Duck
#168 - 2012-06-13 13:06:17 UTC
Desmont McCallock wrote:
Are you using the .NET native json parser?
http://msdn.microsoft.com/en-us/library/system.web.script.serialization.javascriptserializer.aspx#Y0
Btw, I'll just repeat, prices should be cast to decimal.


Yes, I am. I'm using the sample code posted on the EMDR website. Which, if I'm not mistaken, you wrote actually. So I should be OK there.

And just tonight I rewrote the little bit of an application I've completed so far. Having learned so much the last few days, I had to restructure the whole thing. Including getting all my data types straight between JSON, the application, and the DB. Turning on 'option strict' really helped with this actually.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#169 - 2012-06-14 03:57:24 UTC
I've done a pretty good round of documentation updates, and updated the main thread. If anyone sees anything mis-spelled, badly explained, or flat out wrong, please let me know. My aim is to make most of this stuff easy to understand.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Bum Badiipadaap
Federal Navy Academy
Gallente Federation
#170 - 2012-06-16 07:18:13 UTC
Has anyone done this using only javascript?
Desmont McCallock
#171 - 2012-06-16 07:50:32 UTC
Bum Badiipadaap wrote:
Has anyone done this using only javascript?

Does this suffice? http://www.zeromq.org/bindings:javascript
Vaerah Vahrokha
Vahrokh Consulting
#172 - 2012-06-17 00:21:44 UTC
What if some 3rd party data consumer website did not need all this data sent at all, but just something like:

record {

item_ID,
region_ID,
minimum_sell,
date_time_minimum_sell_last_update,
maximum_bid,
date_time_maximum_bid_last_update

}

EvE Online portal | EvE markets tutorials
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#173 - 2012-06-17 03:09:37 UTC
Vaerah Vahrokha wrote:
What if some 3rd party data consumer website did not need all this data sent at all, but just something like:

record {

item_ID,
region_ID,
minimum_sell,
date_time_minimum_sell_last_update,
maximum_bid,
date_time_maximum_bid_last_update

}



You are free to pick out whatever you need. EMDR sends data out in the unified format, which has everything you need in order to selectively pick from.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Vaerah Vahrokha
Vahrokh Consulting
#174 - 2012-06-17 06:33:57 UTC
Ilyk Halibut wrote:

You are free to pick out whatever you need. EMDR sends data out in the unified format, which has everything you need in order to selectively pick from.


Since EMDR does not have past data I will not be able to pick older orders (still well valid today) and merge them with the new ones to form a decent order book. Did I miss something?

EvE Online portal | EvE markets tutorials
Desmont McCallock
#175 - 2012-06-17 07:58:53 UTC
EMDR is not a market data aggregator but a simple relay, a hose of continuous real time market data. What people do with those data are up to them.
Demption
Yeet or Be Yeeten
#176 - 2012-06-17 12:27:44 UTC
Vaerah Vahrokha wrote:

Since EMDR does not have past data I will not be able to pick older orders (still well valid today) and merge them with the new ones to form a decent order book.

All currently active orders are exported (by say EVEMon) and uploaded, so that's older orders that are still valid too.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#177 - 2012-06-17 19:13:34 UTC
Vaerah Vahrokha wrote:

Since EMDR does not have past data I will not be able to pick older orders (still well valid today) and merge them with the new ones to form a decent order book. Did I miss something?

Yeah, I think you missed the FAQ on the top of the thread:


  • Can I get historical data from EMDR?
  • No. EMDR is only a dumb data firehose. We feed the sites that offer historical data APIs. They are the ones that serve this need, not EMDR. That said, if you find yourself constantly hammering other APIs to fill your DB, you should probably just consume EMDR and stop burdening other sites.

So, in essence, record what you can/want to, fill in your historical gaps with your favorite third-party API (or several). After that point, keep recording the new stuff and you're all caught up. You still get 'history' messages from EMDR, which are also another way to fill in the gaps (history and order are the two message types).

But the most important thing to grasp is that EMDR is a dumb firehose. It sets you up as if you were on the receiving end of a huge amount of direct uploads, and leaves you to decide what to do with all of the incoming data.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#178 - 2012-06-18 05:01:51 UTC
Tonight's updates:

  • Adjustments to the GoLang relay to figure out what error is leading to memory leakage.
  • Restored EMK uploading to the gateways, for now...
  • Added some additional useful error messages for incorrectly formatted UUIF messages.
  • Dead code removal.
  • Set up some automated alarms to notify me if there's an outage.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#179 - 2012-06-18 08:45:48 UTC
I'm seeing something weird with respect to dates.
The generatedAt value of 99% of the data has an hour field of 00, like this:
snippet of received data:

Quote:
2012-06-18T00:00:20+00:00
2012-06-15T00:00:32+00:00
2012-06-17T00:00:39+00:00
2012-06-17T00:00:05+00:00
2012-06-18T00:00:37+00:00
2012-06-18T00:00:20+00:00
2012-06-16T00:00:56+00:00
2012-06-18T00:00:38+00:00
2012-06-18T00:00:28+00:00
2012-06-17T00:00:09+00:00
2012-06-16T00:00:48+00:00
2012-06-10T00:00:46+00:00
2012-06-18T00:00:49+00:00
2012-06-18T00:00:55+00:00
2012-06-17T00:00:39+00:00
2012-06-16T00:00:42+00:00
2012-06-18T00:00:13+00:00
2012-06-18T00:00:32+00:00
2012-06-16T00:00:57+00:00
2012-06-17T00:00:20+00:00
2012-06-17T00:00:56+00:00
2012-06-18T00:00:44+00:00
2012-06-18T00:00:49+00:00
2012-06-18T00:00:14+00:00
2012-06-18T00:00:28+00:00
2012-06-11T00:00:33+00:00
2012-06-17T00:00:35+00:00
2012-06-18T00:00:14+00:00
2012-06-18T00:00:47+00:00
2012-06-18T00:00:57+00:00
2012-06-18T00:00:54+00:00
2012-06-18T00:00:46+00:00
2012-06-18T00:00:51+00:00
2012-06-17T00:00:12+00:00
2012-06-17T00:00:30+00:00
2012-06-18T00:00:26+00:00
2012-06-18T00:00:32+00:00
2012-06-10T00:00:36+00:00
2012-06-18T00:00:29+00:00
2012-06-12T00:00:33+00:00
2012-06-14T00:00:26+00:00
2012-06-17T00:00:00+00:00
2012-06-17T00:00:10+00:00
2012-06-18T00:00:37+00:00
2012-06-18T00:00:14+00:00
2012-06-18T00:00:49+00:00
2012-06-14T00:00:20+00:00
2012-06-17T00:00:35+00:00



I don't know where the problem occurs, at EVEMon or at EMDR, but this cannot be right.

Creator of Eve Addicts
Desmont McCallock
#180 - 2012-06-18 08:55:59 UTC
If those are 'history' messages it's perfectly normal as Entity explained in https://forums.eveonline.com/default.aspx?g=posts&m=1265226#post1265226