EVE Technology Lab

key is a free text field but it was intended primarily to be used as a simple ID number or hash. It could also be used as public key and hash for private/public key system.

Make sure you have read the JSON spec (one page) as it's important to understand it. http://www.json.org/

As pointed out "name" and "key" but not "origin" is part of uploadKeys. I'm just guessing but it seems you aren't using any kind of JSON library which make generating it correctly much easier. In the actual JSON the spacing etc used in the examples don't exist that was done purely for easier understanding of the format for us poor humans

Will you feed it to EMDR only or directly to the sites as well?

First of, EVEMon is going to support only the unified format. Therefore only the sites that support the format can participate in testing. After all, afaik only EMDR has a gateway for the UF.

Edit: In order to answer to what might had been Snarf's question, EVEMon uploader will support preset and manually entered endpoints.

Have you managed to fix the split data issue? If so, i can enable the stream again.

Edit:
Never mind, I see the are still split.

You treat buy and sell orders of the same item as separate uploads, although they belong together.
Now you have one rowset of typeID xxx in region yyy for sell orders, and a second one of the same item + region for buy orders.

I figure that is because that is the way you pull it of Eve Market Data's API, but that would not be the way it was uploaded to them, or how it should be distributed again.

It would be better if EMD could be persuaded to forward their uploads the moment they get them. There should be no need for you to pull data from any api, although I understand that you want to supply as much data to EMDR as possible.

All others should follow your example.

So according to EMDR documentation which is the correct upload gateway url, cause docs also mentions http://master.eve-emdr.com/upload/ (note the 'master)?

History uploads are a bit less common, but I think we only get them from EVE Marketdata and direct uploads. There are not always history uploads to report, so they will be somewhat far and few between for a while. I'll be adding EVE Marketeer this weekend, which may improve volume somewhat.

If you make a point of watching and still see no history uploads at all, let me know and I'll figure out what's wrong.

I'm pondering an architectural change, wanted some second opinions.

Currently, this is how things work, assuming we have two mirrored sets of daemons on two different machines:

http://www.eve-emdr.com/en/latest/overview.html#high-availability-through-shared-burden

This is working wonderfully at scale, we've got very high capacity, and processor usage barely peaks 3% on a low end Linode VM. The problem is, the gateway is a dumb relay. It does no parsing of the payload, aside from making sure a few relevant bits are there.

I'm noticing some mal-formed uploads, which get filtered out at the processor level. The problem is, the uploader has no idea that their data is bad. So what I'm considering is this:

• Get rid of the broker and processor daemons.
• Gateways take on the processing duties, and give the uploader feedback if their data sucks.
• Gateways send (PUB) the data to all Announcers (SUB).

Benefits of this approach:

• Getting constructive feedback to the users about bad data is possible this way.
• We remove some moving pieces.
• Less code.
• Less moving parts.

The downsides to this approach:

• Gateways will lose some throughput, since they'll be doing more.
• I'll need to look for gateway volunteers, instead of processor volunteers. This is worrisome only because the gateways sit behind round-robin DNS. If one of the gateways goes down and we don't automatically remove them from the rotation, a percentage of uploads will hang or never arrive. We'd need to write something to status check the gateways, and remove them from the round-robin DNS rotation if they become unavailable.

I'm leaning towards making the jump and doing this. Does anyone have anything to add?

Today's round of changes

• You may now gzip-compress Unified Uploader Interchange POST bodies, as long as you set the 'Content-Encoding: gzip' header. This saves a ton of bandwidth, so do it! For .NET users, you may run into some issues, but Desmont is working on a way to do this.
• I have eliminated processors and brokers from the stack. This simplifies things, and lets me give uploaders direct feedback about whether their data sucks or not. Previously, it was too hard to tell if you messed up or not
• I will now return HTTP 400 error codes if you pass in data that sucks, along with a hopefully-helpful error message. HTTP 500's may be seen in the rare case where your data is so malformed that the gateway freaks out and doesn't know what to do with it
• I went postal on the various date parsing/encoding issues, and finally got everything pretty consistent. We're now more permissive than we should be as far as what kind of date strings we can parse, I'm not sure whether that's good or not :)
• Lots and lots of doc updates. Instructions for other sites wishing to relay data have been updated to include pointers on how to gzip stuff. Data sources page added (let me know if I missed you). Huge make-over to the High-Level overview. It's more concise and hopefully easier to digest.

We are now receiving data from the following sources

• EVE Central SMTP feed (orders only
• EVE Addicts (all direct uploads
• EVE Marketdata (orders+history via their recently updated API

The following data sources are in the works

• EVE Marketeer (all direct uploads, Callahan says sometime this weekend
• EVEMon Uploader (we'll be in default uploader endpoint, early June

And finally, a note on erroneous uploaders. There are quite a few who are uploading stuff that we can't use due to it being improperly formatted. Please, everyone, check your re-uploaders or syndicaters and make sure that your log files don't show a ton of errors now. Here are a list of a few