EVE Technology Lab

The forwarding loop should not be an issue. All sites currently feeding EMDR don't actually use it for their own data generation yet. The correct way for them to approach this would be to point their "upload" view at EMDR, then consume the EMDR feed like the rest of us. We're probably a good way away from having to worry about that, as I'm still working on establishing a track record of reliability, and getting everything battle-tested and documented.



It couldn't hurt to at least toss up a wiki page with details on who is consuming who, for now. Though, as I have pointed out, data duplication actually becomes less of a problem if we're sane about how data is consumed.

You are correct that something like this came up before and it was agreed there was no way that couldn't be abused and there are better ways to handle it.

IMHO some duplicate data is not a bad thing since it should help in spotting anyone trying to push bogus data into the system. It also help when sites go down due to bug/maintenance etc in that when they come back up they should be able to request fills from the other sites.

The developers at CCP have repeatedly said no to any kind of market data export to the point they generally won't even response any more when people bring it up so what we have now is as best we'll get. As for some kind of transactionID they are available to the person that makes the market order via the API so if all applications that have API information would add option to forward the market order API it would be added as the orderID in the existing format. Just to make it clear this is the same orderID you get from the cache export so everyone has access to the main ID you need to track the orders.

First you have to ask yourself why you think it's important you know who is sending you the data? If it's just for some kind of credit system what does it matter if someone else want to give one of your users some extra credit? If you are trying to use it for anything else it's a mistake as you'll never know for sure who is sending something to you especially if you allow normal HTTP connections and don't limit the connections to HTTPS which probably doesn't really add any value. If you were using something in your key that is used to authenticate the user that's just asking for trouble anyway and shouldn't have been done to start with.

As far as I understand it, looking at the data of a market related cached file, it includes the orderID for each order. So even if multi-clients upload the same info (cause clients are viewing the same market region and same item), cross-referencing the orderID, when processing the data at the server side, would be more than enough to identify duplicate uploaded data.

Do I get this right or am I missing something?

Edit: Some will say that between two uploads an order could have been changed (although the ID remains the same), which may lead in rejecting the updated data. Now, we at EVEMon do an 'IsModified' check when ever we update the market order monitor. This check compares the 'price', 'issueDate' and 'volRemaining' of an order with the same ID. Couldn't this be the checking function at the server also?

Whenever you modify an order the 'price' and 'issueDate' change but order doesn't get a new ID.
Whenever an order gets a transaction then the 'volRemaning' changes. When 'volRemaining' reaches "0" order is deprecated.
At least that's how the API works and API is an extension of the DB.

Not sending it back to where you got it is of course obvious.

The destinations json section would just be a very convenient way of detecting a duplicate without having to do any database calls and with very little other processing.
Adding a key to uploadKeys would let me identify stuff I send into the relay, but it won't help me detect stuff already sent to me directly.

In your reasoning, isn't the uploadKeys section also beyond the scope of the format, since it hasn't got anything to do with the actual data?

The way I see it, routing info has a lot to do with the exchange of data, it tells you where data is going, and where it has been.

http://dev.eve-central.com/unifieduploader/start If you haven't read completely through that you should. uploadKey isn't made to really be used for what you think is needed but it can be used to detect message looping and all sites should have checks in place for them if they both send and receive data.

As pointed out by Desmont McCallock wither the duplicate data happens because you have 1000 users all doing market scans in say Jita or least people uploading to multiple sites which are cross feeding each other doesn't matter you still have to deal with dups in some way and truthfully both cases can be handled the same. I think part of the thought behind wanting some kind of tracking history like you are asking for is that you view the dups as a waste of resources vs where I and others see it as an chance to receive fills when my site might be down or confirming the data I've already receive from other source hasn't been tampered with. By the nature of how the data is provided you are always going to have to deal with dups so it's much better to design you system so it doesn't matter from the start then trying to have something added on that itself could just as easily be faked and worthless. I also don't see this being a big problem as there really isn't much need for more then 3-4 upload sites to begin with and them cross feeding each other.

On a relate thing I do think all upload sites should have some kind of uploaderID which they include in the uploadKeys so sites can try to be smart about which other sites they might forward to first. So for example if they notice an EMK key they might delay upload there until they've uploaded to Eve Central which isn't listed. I can even see having a system where by default you don't do pushes to other sites listed as receivers in the data but they can request the data even if they might already have it as a pull. Being able even to request filtered vs unfilter updates would probably be a very useful thing.

Anyway that my thoughts on making any changes.

Let me know how that goes. If this ends up being a bad problem, EMDR will just add what it needs to the JSON to prevent issues, and consumers can discard the added bits.

And my bow... lol