EVE Technology Lab

I do understand the concern and when I get around to adding uploading to Yapeal there may be a few more sources showing up as well though EveMon probably will be the bigger change in numbers My main point is that UF was created to be a simple data exchange format not a routing and duplicate data detection system as how that is handled is up to the endpoint receiving the message. UploadKeys was added as an optional part that allows data sources and receiving endpoints to be able to pass a one way basic pre-agreed authorization or ID not to solve routing issues etc.

On the possible database load issue I have to ask a question here. Are you already seeing 1000s or 10000s of queries per second now? The reason I ask is I know with a well designed DB (correct indexes, and normalized) and well designed queries that a $15US a month single cloud node can handle that kind of load. If you are seeing less than a couple hundred per second and having problems now you need to be figuring out why. I know from my own experience in working on Yapeal that some very simple changes can make a huge difference especially with anything that does a lot of writes like Yapeal or anything trying to merge lots of data like the current endpoint have to. Hopefully no one that is using MySQL is trying to use MyISAM instead of InnoDB or TokuDB. The table locks and no transactions will kill you quickly.

I'm not sure I understand the point. There really aren't any details, and I'd need to be instructed as to how I should act differently under /syndicate vs. /upload.

Hi, folks. I'm new to this discussion, so I hope I haven't missed any important details.

Desmont, I think it would help to consider actions instead of participants in your ABC model. Fundamentally, any given player can do any of the following:

• Produce a payload with new data
• Transmit a payload
• Receive a payload

A cache scraper will produce and transmit, while a relay receives and transmits, and a market analyzer only receives. Some kind of tool might do all three, but it shouldn't really matter because we should treat each action separately.

One step toward avoiding duplicates is to make sure that retransmitted payloads don't introduce differences. If there are any fields that are allowed to change in retransmission, those should be an explicit part of the standard.

Duplication comes in a few forms:

• Stored duplicates - the same payload is stored multiple times. A properly designed database should reject redundant rows, but there must be an efficient way to compare two payloads for equivalence.
• Transmitted duplicates - this is where relays and an EVEMon uploader can get us into trouble. If a few thousand payloads are uploaded per minute, and then those payloads start bouncing between relays indefinitely, we're in trouble.

If the storage problem can be solved, then a possible solution for the transmission issue is for each relay (or similar participant) to store some recent history, say the last 24 hours of payloads. Any incoming payload is checked against that cache, or if it's dated to longer than 24 hours ago, automatically rejected. Participants that store their entire history of payloads essentially have a perfect cache.

I don't think "destination tags" or other such routing information in headers is a good idea. Where the data goes doesn't mean anything to someone consuming the data. It's a bit too meta, and if the payload is retransmitted, we get the same payload sent around with different information about who's supposed to have it. Duplicates are still possible.

(Example 1: suppose A is an uploader, B and C are relays, and D is an aggregation site. A uploads to B and C, so the header has them marked down. Both B and C send their data along to D, whether it's pushed or pulled, and D is added to the header. However, it's still sent twice, and now D has a duplicate. This payload won't get trapped in a loop, since nobody will retransmit to D again, but anyone downstream of D gets the duplicates.)

(Example 2: a participant has multiple, functionally equivalent upload points. Excluding hostnames listed in the header won't protect against all duplicates.)

I think the best option is to prohibit modification of a payload. The formatting can change—white space, encoding, dates—as long as it's still valid under the original specification. Payloads that differ in formatting only should still be resolved as identical. Now, this doesn't solve the "fifty people in Jita all look at Tritanium" issue, but those are different payloads and should be treated by the network as such. It's up to aggregation systems to decide how to handle such data.

On that topic, suppose a site wants to handle redundant data (from distinct payloads) by averaging all the prices from each day. If they then upload their payloads to a relay, they are creating a new payload. The "generator" value should reflect this.

By the way, I think there's plenty of room for interesting kinds of participants. Here's a hypothetical network to get you thinking:

• A: Cache scraper: Someone AFKs in Jita all day.
• B: EVEMon/Yapeal: A trader's orders are pulled from the API.
• C: Cached relay: This relay maintains a one-day cache to avoid duplicates.
• D: Aggregate relay: This relay keeps a record of every payload it's seen to avoid duplicates.
• E: Market analyzer: Averages data from other payloads and produces its own derivative data.
• F: Archive relay: Constantly retransmits the record of its stored payloads at a certain rate so that new participants can "catch up" on historical data.

Suppose A sends to C, and B sends to D. The relays C and D both send to E and F. E sends to F, and F sends to C, which additionally sends to D. (Draw a picture if you have to.)

If there's no duplicate checking, you can see how loops form, but this system works. Suppose A produces a payload. It goes to C, where it's added to the short-term cache and sent along to D, E and F. The payload is added to the long-term caches of all three. D sends the payload to E and F, but they reject it as a duplicate. In a couple days, F retransmits the payload to C, where it gets added to the short-term cache again as a new payload. However, when C transmits it to D, E and F, it's recognized as a duplicate and rejected as usual. It's a little additional bandwidth, but that's just a consequence of the network design (which is far from optimal). The new derivative payload created by E ultimately makes its way around to the various payload stores by way of F -> C -> D.

I think both push and pull capability is important for a network like this to function well, but not all nodes need to do both, and not all nodes need a local store.

(You can still do bad things with this. Imagine two cacheless relay nodes that are hooked up to each other. They'll bounce payloads between each other all day long, and they'll copy them out to other sources, constantly spamming them. This is a basic configuration error, though, not a problem with the underlying protocol. These spam nodes would be quickly blacklisted.)

Apologies if I misunderstand anything. I had a lot of thoughts as I read the thread and I thought I'd braindump a little.

BattleClinic has been added to UF supporting endpoints.

I just thought I'd point out a spec-related issue:



I know at least nginx (and I think apache) have smallish limits on the maximum size that a GET param may be. I believe this was low enough to the point where most or all uploads will fail, if done via GET. Also, people really shouldn't be using a GET to do what should be a POST.

The other thing I had is to ask why we are using a 'data' parameter, instead of a non-formencoded POST body of JSON. Seems like that would be the simplest way to go. EMDR currently accepts uploads this way.

That's a library-specific quirk for whatever tools you're using. For example, this would post a non-form-encoded body using python-requests:

This only becomes form-encoded if we pass a key/value data structure (in the case of Python, a dict)

So, to follow the spec, I'd have to do this:

Which seems kind of silly, since all we really need is (a non-form-encoded body that is just JSON):

Hopefully this illustrates what I'm getting at better.

Need to use POST for the reasons found in this thread: http://stackoverflow.com/questions/630453/put-vs-post-in-rest
To sum up since it's not create a new resource but modifying part of an existing one PUT does not make sense.

What we are missing is a common name for the key. How about UUDIF? So as an example:
UUDIF={ "resultType" : "orders", "version" : "0.1alpha", "uploadKeys" : [{ "name" : "emk", "key" : "abc" }, { "name" : "ec" , "key" : "def" }], ...

That should make all the parsers happy.

Edit: You all were posting while I was write stuff I agree if we can work without having to do the whole key-value thing it would be nice but most libraries do seem to prefer having them and tend to make you work with the raw data directly without it which is not so nice

I just know that some platforms have limits on length of names etc and thought it would be unlikely to be confused with something else. If we start adding other parameters like data=,message= we might as well not use the format to start with. We only need a single key-value to make it easier to work with for most parsers and it would be helpful to all use the same thing but beyond that it really doesn't matter what it is.

What platforms really have to have a urlencoded form?

This is my reason for disliking the use of encoded POST bodies for our usage case, since we don't need to. A developer's inability to read documentation doesn't seem like a good justification for using form-encoded data.

Besides that, the wiki probably should be updated to not mention uploads using GET or PUT.

I agree on the post body issue. The data should just be the full post body, form-encoding it is just silly.