EVE Technology Lab

This isn't related to the spec, but as a side note, please set User-Agent strings for the uploader! I don't think the EVE Marketeer uploader does this, and that makes nginx access logs sad.

Desmont McCallock - You are correct that adding a 'z' for UTC is allowed even preferred but having no zone offset would mean the DT was in local zone which isn't useful in this application What you are probably running into is the server is in UTC so .NET doesn't think it needs to add it. I had to do this in PHP to make it do what I needed so you might need to do something similar
gmdate('c', strtotime($row[$k] . '+00:00'));
You could append just 'z' but I felt it would make it easier for anyone that has to parser it manually to always work with numbers.

Nothing jumps out at me as invalid accorded to the format in your example other than the DT issue.

As to empty order messages I guess it would depend on how the receiver site uses that info. If they take it to mean there aren't any orders but they show some that could be useful information but it'll depend on what each site decided which would be outside of what the format cover of course

I noticed that ISO 8601 has a few different ways to denote timezone offsets. I've seen Z00:00, T00:00, and +00:00. I'm not really sure which way to go here. Can we all agree on the +00:00?


EMDR is currently filtering out empty orders, since I was concerned about that being a really easy way to wipe data items off of badly written sites, but I guess that may be the price to pay for having a badly written site. For anyone consuming the market data, would you like these empty records? I'm cool with whatever.

I have corrected this in EMDR:

2012-04-21T20:25:55+00:00

This also means that any forwarder that re-generates the JSON message has to change the generator field as well.

If this is allowed it gives an additional tool for duplicates detection.

Slightly oftopic but may be affecting all cache scrapper libraries, so it makes it kinda relative.


Edit: After doing some testing in SiSi, nothing seems to be affected. Everything works normal. I'll evaluate it again after patch goes live.

Edit2: Nothing seems to be affected. Everything works normally.

This is my attempt to explain how the market data network should be structured and work and how the message in Unified Format (from now on called UF) should be and used.

The network is currently consisted of the following entities:

• A: Data Uploader.
• B: Data Aggregator.
• C: Data Relay.

The traffic between entities is described below.

1. A can send data to B and C.
2. B can send data to C. B can gather data by subscribing to C or another B via API.
3. C broadcasts the data it’s receiving, without targeting anyone specifically.

Let’s examine the roles of each entity in the network.

Data Uploader

Let’s explain what A should be doing.
A is generating a UF message by gathering market data either from an EVE cache scrapper or the EVE API. The message falls under the specs described at: http://dev.eve-central.com/unifieduploader/start . Special attention should be given when constructing the “uploadKeys” field. This field purpose is to provide to a B type entity, info that is related to it, like a specific ‘key’ used to identify a user (bind to the entity’s DB) and give credits for uploading data. Although the use of such a function is optional for a B, the field has been added to the specs in order to provide such a possibility.
A special issue was raised when I was constructing EMUU and it involved the exposition of the ‘key’ values as a potential security risk.

There are two ways to construct a UF message.

1. Constructing the message specifically for each endpoint with the appropriate info in the “uploadKeys” field and send it only to that endpoint.
2. Constructing the message with info for all endpoints in the “uploadKeys” field and send the same message to all endpoints.

Case 1 is counterproductive but safe.
Case 2 is the most efficient but exposes user info to receiver.

As I want EMUU to be most efficient, thinking about case 2 I came to the conclusion that exposing the ‘key’ value of a user to others has no particular value to them. If someone decided to use another ones key, as own key, he will only end up giving credits to the owner of the key.

continued...

I'm not sure if the lack of comments on my last post is a good thing or not Ok so you posted your stuff at the same time I did

That's assuming that C actually received it which might not have happened.

I can see cases where someone might want the uploadKeys for farther processing so just dropping them without reason is IMHO a bad idea. They should only be dropped in cases where either something has changed that makes them invalid or where it is unknown if they are still valid.

As to detecting duplicates it seem everyone is ignoring the simplest thing. The orderID and when it was generated and received. If you are receiving something from a source that matches existing data you need to only determine if its an update or outdated. You may also what to weight how trusted the source is etc but that's really out of scope of what UF's purpose is.

The question is, can we discard incoming data by just looking at it's origin and the data itself?
If we can, we don't need to do any further processing, which is a big plus for aggregating sites.

At the ,moment i have to accept all data, and later process it. If the data i have in the DB is newer, i discard the data, if not, I update the DB.
But this requires costly calls to the database.

At the moment, only about 33% of the uploads i am seeing are unique.