These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Tseehn Marhn
Rat Duck
#201 - 2012-07-03 19:34:36 UTC
Barl Rathbone wrote:
Tseehn Marhn wrote:


Ah, well, good to know it should work. Also good to know no order ID means dead order.


Mind you I wouldn't trust a single order message -- I aggregate orders over time and then process through so I have a better shot of getting a valid "hit" rather than invalid data. It is all user uploaded stuff so you have to take it all with a grain of salt. :)



Yeah, considering this and other things, I'm just going to scrape data from the cache for now. It sucks I have to click through the market, but I don't feel like trying to find ways to filter out the spoofed orders right now. Eventually though, I want to combine local and EMDR data - and use local data to help verify the EMDR stuff where you can.

BTW, thanks Desmont for the cache parser DLL. Saved me weeks of trying to figure out the cache on my own.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#202 - 2012-07-03 21:25:49 UTC
I think what was meant was that spoofing is obviously possible, but we don't see much evidence that it's a problem right now.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Barl Rathbone
Corel Industries
#203 - 2012-07-03 21:35:53 UTC
Ilyk Halibut wrote:
I think what was meant was that spoofing is obviously possible, but we don't see much evidence that it's a problem right now.


Correct. I haven't seen any real issues and to be honest, the volume of orders received if you identify the top and bottom oddballs tends to even out, just like any 95th percentile.
Schereau Lasomere
University of Caille
Gallente Federation
#204 - 2012-07-11 04:32:48 UTC
I'm trying to develop an EVE market data processor and I'm feeding it data from EMDR. A really great service! Big thanks to UUDIF's and EMDR's developers!

I've got a question for the UUDIF and EMDR developers:

I've read the EMDR and UUDIF specs as well as searched the related forum threads for info but I'm still confused about the currentTime and generatedAt timestamps.

You can read the reasons for my confusion below but basically I'm not sure which timestamp to use/trust for data freshness verification.


Some observations and suggestions:

currentTime: The UUDIF spec says that the time "may also be changed by any forwarders as well when the message is re-sent" which I interpret as the change being optional.

If the time is NOT changed by a forwarder (EMDR for example), then this time, as far as I can figure out, would be the generator's UTC/local time and so can't be used for verifying order freshness since all generators' clocks most likely are not synchronized.

Issue: The EMDR documentation does not (?) say whether EMDR does change currentTime or not. I'm guessing EMDR is NOT changing the time since I'm receiving messages out of order based on currentTime value (reading data from a single relay, the default European relay in Germany).


generatedAt: The UUDIF spec says this time is "The date/time...when the rowset was created". From where is this time taken/time according to whos clock?

  • Search result generation time according to CCP's server time?
  • UTC time, based on player's computer clock, when the search result is stored in the game cache?
  • Other?


According to the UUDIF spec, this time is supposed to be used for freshness verification. If it's the UTC time, based on player's local time, when the search result is stored in cache, then this timestamp can't be used for freshness verification since most players' clocks aren't perfectly synchronized.

The UUDIF spech doesn't state where this time should be taken from and so I'm vorried that uploader developers may use different sources for this timestamp. This makes me unsure if I can trust this timestamp as an accurate freshness indicator.

Suggestion: Update the UUDIF spec so it enforces a common method for determining this time, ideally search result generation time according to CCP's server time, if possible.
Snarf Aldes
University of Caille
Gallente Federation
#205 - 2012-07-11 07:20:07 UTC
generatedAt is the time when the eve client writes the data to the cache file.
currentTime is the time that the cache file gets parsed and sent by the upload program (evemon etc.)

Times should all be in UTC.

Creator of Eve Addicts
Desmont McCallock
#206 - 2012-07-11 08:08:48 UTC
As Snarf said. All times should be UTC.

As the creator of EVECacheParser (a .NET library based on Entity's Reverence Python library) let me explain where 'generatedAt' comes from. Each cache file contains the time of its creation (EVE Time which is the same as UTC) in Windows filetime format. This filetime when parsed gives as the 'generatedAt' time.

'currentTime' is generated by the uploader app to state when the UUDIF was created. EMDR is indeed changing that (as a forwarder it's allowed to) and its not the same with the 'currentTime' EVEMon generates. Although in the majority of cases that time can be the same as the round trip time can be less than a sec.

Hope this helps.
Muscaat
EVE Markets
#207 - 2012-07-11 11:40:05 UTC
Snarf Aldes wrote:
Times should all be in UTC.


I disagree. As long as the timestamps have the correct timezone on them then there's no reason why uploaders should be forced to convert them to UTC themselves. See https://groups.google.com/d/msg/eve-emdr/oc6aMya4V1Y/5GsWbmh0l9QJ for the discussion triggered by me being in UTC+1 right now Blink
Desmont McCallock
#208 - 2012-07-11 12:00:38 UTC
Because by using UTC you don't have to worry about DST.
Muscaat
EVE Markets
#209 - 2012-07-11 12:09:15 UTC
Desmont McCallock wrote:
Because by using UTC you don't have to worry about DST.


Who is "you" in that statement?

If the generators (the people who write the uploaders) are forced to convert to UTC then they do need to worry about DST, because they need to convert away from their local timezone (whatever that is right now).

The spec doesn't currently force UTC, which means that uploaders don't need to worry about DST (other than knowing what timezone they're in right now, which their date library does for them, right?) and consumers just need to parse the date properly and not worry about it (because their date library deals with the timezone for them, right?)

I agree that things would be a lot simpler if everyone just used UTC and got used to going to work at 23:00 or whatever, but I don't think that idea will catch on P
Desmont McCallock
#210 - 2012-07-11 12:47:58 UTC  |  Edited by: Desmont McCallock
Muscaat wrote:
Desmont McCallock wrote:
Because by using UTC you don't have to worry about DST.

I agree that things would be a lot simpler if everyone just used UTC
That's the point behind the suggestion.

And just to clarify something here. Both me and Snarf said 'Times should all be in UTC.' as a suggestion.
If we said 'All times have to be UTC' (which make it mandatory) then you would have a point to argue.

** Is there a Hermes recursion going on, cause there is a lot of misunderstanding around?

Edit: Btw, as we discussed (and I proved to you) in the Google group discussion, the time value has to be in UTC but you clearly have to state the TZ diff if you are using it.
Muscaat
EVE Markets
#211 - 2012-07-11 13:36:43 UTC
Desmont McCallock wrote:
Muscaat wrote:
[quote=Desmont McCallock]Because by using UTC you don't have to worry about DST.
Edit: Btw, as we discussed (and I proved to you) in the Google group discussion, the time value has to be in UTC but you clearly have to state the TZ diff if you are using it.


I'm not quite sure I understand this, but I'm fairly sure we don't actually disagree all that much so I'll stop arguing the point now Smile
Desmont McCallock
#212 - 2012-07-11 14:02:53 UTC
In that discussion I am 'Jimi' as I'm known in the EVEMon forums as well.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#213 - 2012-07-11 14:11:44 UTC
Yeah, this is not the most interesting discussion, so I'll just leave this hear to dispel confusion, myths, or half-truths. Here is basically all time fields in UUDIF are bound by:

Quote:
The data/time in ISO 8601 format (example 2011-10-22T15:46:00+00:00)


Follow the ISO 8601 spec and you are compliant with UUDIF. If your local time is UTC, your offset is +00:00. If it's some other timezone, specify it in the tz offset. As long as you're doing that, people with ignorant consumers may get burnt, but that's only because they aren't doing what the spec demands (parsing tz offsets applied to UTC).

As Muscaat or someone else mentioned, your programming language of choice undoubtedly has some modules/packages/libraries for dealing with ISO 8601 dates. Make sure you use them and you'll be all good.

And now for some hopefully more exciting discussions :)

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#214 - 2012-07-11 14:20:58 UTC
Ilyk Halibut wrote:
And now for some hopefully more exciting discussions :)
I'll drink an ice tea on that. Big smile
Schereau Lasomere
University of Caille
Gallente Federation
#215 - 2012-07-11 19:40:41 UTC
Thanks Snarf and Desmont! that clarified things alot.

Desmont McCallock wrote:
EMDR is indeed changing [currentTime] (as a forwarder it's allowed to) and its not the same with the 'currentTime' EVEMon generates.


The weird thing is that in the data stream I'm receiving, currentTime keeps jumping back and forth by about a minute:
Quote:
currentTime,generatedAt,regionID,typeID,...
.
.
.
2012-07-11T18:40:29+00:00,2012-07-08T20:04:25+00:00,10000002,38,...
2012-07-11T18:41:38+00:00,2012-07-10T20:31:42+00:00,10000042,3436,...
.
.
.
2012-07-11T18:41:38+00:00,2012-07-10T20:31:42+00:00,10000042,3436,...
2012-07-11T18:40:30+00:00,2012-07-11T18:41:04+00:00,10000030,5401,...
.
.
.
2012-07-11T18:40:30+00:00,2012-07-11T18:41:04+00:00,10000030,5401,...
2012-07-11T18:41:39+00:00,2012-07-09T21:12:23+00:00,10000060,1944,...
.
.
.

I thought that if EMDR does change the currentTime, then the currentTime would be always increasing, rather than jumping back and forth, when data is downloaded from a single relay. Is the order data passing through two different data processors (PCs) with unsynchronized clocks in EMDR, even though I'm downloading the data from a single relay? The EMD architecture picture in the docs seems to hint at that. Trying to figure out the cause and potential effect of what I'm seeing.
Desmont McCallock
#216 - 2012-07-11 19:56:30 UTC
Although it's indeed happening, I wonder why you are so obsessed about it. Are you using the 'currentTime' for something?
Could be a side effect of ZMQ queuing but Ilyk is the man with answers to that.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#217 - 2012-07-11 21:06:43 UTC
Looks like in my hurry to set up the secondary relay, I forgot to install ntp on it. Whoops. Fixed, the drift will correct over itself.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Schereau Lasomere
University of Caille
Gallente Federation
#218 - 2012-07-11 22:29:10 UTC  |  Edited by: Schereau Lasomere
Desmont McCallock wrote:
Although it's indeed happening, I wonder why you are so obsessed about it. Are you using the 'currentTime' for something?

Guess it's mix of curiosity and worries.Smile I'm not using any of the timestamps yet. I'm still very early in the dev phase. Was trying to figure out which timestamp to use for freshness verification but the EMDR and UUDIF documentation wasn't clear on that in my oppinion.

Guess it's in my nature to find explanations to things that I can't explain to my self.


Since the UUDIF spec doesn't state what source to use for the generatedAt timestamp, I as a potential uploader developer would not know what source to use. Because of that I was worried that some uploaders might use the player PC's time converted to UTC as a source rather than the in-cahe stated time, which obviously would not be a good idea. That potential problem made me wonder how to do the freshness verification.
Muscaat
EVE Markets
#219 - 2012-07-12 15:05:57 UTC
Desmont McCallock wrote:
Ilyk Halibut wrote:
And now for some hopefully more exciting discussions :)
I'll drink an ice tea on that. Big smile

Make mine a hot tea, milk and sugar Blink
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#220 - 2012-07-13 19:27:51 UTC
I have added an 'EMDR' upload key with a hash that uniquely identifies each uploader by IP address. See the mailing list post for more details:

https://groups.google.com/forum/#!topic/eve-emdr/WNiN63KJQ_Q

Hopefully this is useful for some of you to blacklist the troublemakers we'll eventually attract (if we haven't already).

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com