EVE Technology Lab

First problem we had was coming up with a list of field that would be used to form the hash or if using UUID version 3 clients would have to know their URL which usually is hard to come up with with most providers plus since it's usually based on IP which is served to them by DHCP also worthless. What if their neighbor also plays Eve and gets the same IP but upload from a different region but the UUID would be the same etc problems. There is that plus many other problems try to come up with one that works for everyone. The methods that could solve these type of problems had problem in some parts of the world because that use to high of encryption as well and those are just the ones I've thought of.

http://en.wikipedia.org/wiki/Universally_unique_identifier
Version 4

Seems like exactly the thing that solves all the trouble. I'm not actually interested in a hash of the data of any sort. I'm just interested in a semi random number that could possibly identify that specific upload, like the autoincrementing ID column in a database. I'm not interested in whether it actually identifies the data in that upload. Just that I haven't seen it before.

For all I care this could be a random number in the 64 bit range, but a random UUID seems like a more standardized way to do it.

If some faker would like to try and guess the same totally random uuid with fake data, he can be my guest :P

Ok so I guess what we need to get everyone's input on a couple things here and decided which way to go. I've done my best to point out the problems and failing I see in trying to add any kind of ID/hash etc to the JSON message but if that's what most people want I'll go along with it because have the common format for everyone to use is what's really the most important thing.


Question One
Edit: Should we include some kind of universal ID like one of the UUID versions or a crc32, md5 or whatever in the message or leave it up to each site to decide what they want and put it in 'uploadKeys' if they decide they want it?

For clients it will make little difference as each site is likely to require them to add additional info to 'uploadKeys' anyway to give credit to user etc.

Question Two
If the answer to the above is yes what are we going to use?

• crc32 - simple to make and use and should have few problems with dups.
• md5 - Also simple to make and use and chances of dups on the small data sizes of most messages is going to be nil.
• sha1 - Takes a bit more to make and probably overkill for something this small.
• UUID version 3/5 - By itself isn't enough since it's basically the URL/hostname which for most clients is a problem because of getting their IP from DHCP and there's a high risk of more than one Eve player getting it and causing conflicts.
• UUID version 4 - Numbers totally random as long as client has a good source of truly random number generator but most have at best a relatively poor pseudo-random source instead.
• something else ?

Question three
If using one of the hash/digest methods above what parts of the message should be used or should the whole message be used?

As I've not folloved the discussion closely, my question would probably be rather stupid, but still:
What is the intended purpose of the field? As Dragonaire, I don't see any use for it, at any point.

EDIT: I didn't see you last post above answering Tonto Auri since we appearly were both doing them at the same time but I still think my post here cover the same ground I would have anyway just a little differently then if I had saw it.
PS: I also forgot before to say I'm glad to hear the good news on the new uploader I was concerned it might take a while to catch up the work that was lost.
I do understand that but why I'm saying it won't really do what you think it will. You can have someone sending different data but the same ID, different IDs with the same data and both of the first with faked data. There's also the possibility of a third party doing a man in the middle attack and doing the same things. So if the ID has no dependable connection with the actual data then you're just wasting your time having it. You end up either ignoring good data or getting mostly bad data if someone decides to.

Easiest way to make it worthless is to simple start sending the same fake data and count up through the ID range while doing so and you fill everyone's database with lots of useless data that will make it impossible to use the good anymore. That's where at least to hashes have an advantage as when the received and local one don't match you know something is wrong.

You also stated you basically want something that can act like a primary key from a database but the only way you can end up with something that does that is to have a common list where there's a single point in charge of issuing them which we just don't have.

This problem is one that's been around since before there were even computers but the solutions to it haven't really changed which is you can't do it without a single point of control.

Another way to look at it is as a synchronization problem to insure you don't have different issuers issuing the same timestamp for different groups of data.

There's other ways to look at this problem as I've shown above but they all come down to you have to trust the person giving you the data is being truthful which we all know you can't trust anything coming from a unknown party on the Internet If you can trust you have to verify it and the thing that needs verified is the data which your method ends up ignoring.

EDIT: Just one more point about the ID is when pulling data from other site there nothing saying they will forward the same set of data they got from the uploaders I would in fact myself be only forwarding verified data not just everything I've received by default if it was me. I might provide a raw feed as well but I would make it clear it's a bad idea to use it and the cooked version should be used. If sites don't do that in some of the example above it'll lead to all of them being brought down and possibly need to drop all they're tables to recover because there's no way to sort the bad from the good.

Damn... I hate EVE forum.

Ok, short version.
Don't mind the key, you're looking into wrong direction.
You're discussing data interchange protocol in context of data interchange format. You're wrong.

One thing here it won't be push it'll be pull so receiver is ask for what it wants to see.

The only place push is used is from the client-originator to the aggregator and like you said aggregator always wanting the data though it does need a way to deal with a stupid client that keeps sending it the same old data but they have to do that no matter what so still don't need anything in the message as only the aggregator can really decided if the client is being stupid by somehow comparing the data to what it already has received.

Once again they would be pulling. It wasn't clear if you where thinking push or pull here but once again they asked for it with pull so just like the aggregator above they'll have to decided if it's new. There might be a possibility here to use if-modified-since header which would be nice and save transfer dups and I think the aggregators should implement it even if not every receiver will use it.

Note that if-modified-since will also work with 2 above as well since the plan was to use the same feeds for both. The aggregator may need to start tracking when stuff is updated but I'm thinking most of them are in some way already so shouldn't take any real changes to be able to respond correctly to if-modified-since headers.

This is at the HTTP level and really out side the JSON message. Like always if the client and server both support it there's no reason it can't be used.

Just to make it clear the JSON message that is being sent will be the data using normal HTTP protocols though nothing says they have to use port 80 etc just that that's the easiest by far. This is all planned as a RESTful service so using normal POST, GET, PUT, etc.

You're touching on something I've been thinking about if in a little different way. As it is now there is both stuff like 'regionID' and 'generator' in the metadata header together. That's fine and I basically was move what had been GET parameters into the message from the original system but it could be improved IMHO as well.

If there instead was a metadata header for the 'generator' part with 'resultType', 'version', 'uploadKeys', 'currentTime', and of course 'generator' and the 'regionID' and 'typeID' where moved down into the 'result' some where then inside the result you could have multiple blocks of data for say the same 'typeID' but different regions or the other way around or even more than one of each so clients could even upload everything from the cache in one message which would save a lot of bandwidth for the aggregators not having to receive each as a separate connection and not having a lot of extra overhead being told over and over again who the data was from.

IMHO this would greatly streamline the whole process. I don't know how much of a change it would require in the aggregators to work with multiple 'data sections' but I can see it really helping them with their network bandwidth usage with more uploaders that they may end up with having a unified uploader etc that we are working to have. I'll look at adding in a new section on the wiki my ideas how the new JSON would look so we can look at them side by side and see what people think.

Yes I understand some of them now use push but I think pull would be much better as there's a lot less work for them since there's no need to track who needs what sent to them. Amost all the old services that have used push in the past on the Internet have moved to pull instead. RSS is a good example of a pull service that killed off several old push services that existed before because it simply works better. I see no problem with those that wish to continue their push stuff doing so but IMHO the new stuff should be pull.

I've put the purposed new format up for people to look at http://dev.eve-central.com/unifieduploader/start

I made results a list (otherwise it wouldn't work at all). Otherwise I'm good with the multi-payload option.

As for gzip, etc. Thats something HTTP can solve (Accept-Encoding, et al). EVE-Central already advertizes and will happily gzip/deflate all content automatically (I just globally turned on mod_deflate, it hasn't caused any CPU usage issues).

Two moments:
Receiver have no idea of what is available to send, or that is there ANY data available to send. Yes, it could use blind poll in hope there' something new, but let's we not strafe from the original subject.
We're discussing aggregator-to-aggregator syndication, an interaction between two web services, supposedly running around the clock, with nearly persistent availability. To me, it is more desirable to have data pushed towards the other party, when it's available, and only push what the party want, to save the time and traffic.
OTOH, pulling data at irregular intervals could be desirable for standalone clients, like EVEMon or jEveAssets, but that kind of pulls is limited to processed data.
As I said, you have two data sets, and each of them is related to specific demands. I agree that polling processed data through RESTful URL's would simplify one end of the pole, but on the other end, you propose to have both push and pull for the same (=raw) data, depends on the originator. Is this desirable?


You seem to have misunderstood my message a little in regard to client-server stupidity. I'll touch that later.


Possibility... Yes, I know what you speaking about, but if I can avoid endless polling, I would do so. They could subscribe for data pushes and set a schedule at which they want the data fed to them, if it's meaningful, or just let the server send data, when available.
If they want only certain categories, they could only subscribe to them.
Imagine this scenario: a killboard want median prices for all items in the game - a fairly big list of entities, but actually very little data associated to them. The problem is preparing that data. It does not happen instantly, and creating the median upon request is hardly the sane idea, even less so - recalculating it on every upload. To wear off the load, you run a cron job at certain point in time to slice your pie the way you want it.
Our respectable client (the killboard) have only rough idea of when that script is run, or when the fresh data would be ready, but he do want it fresh.
Variant I: Client poll the data "if-modified-since" every now and then. Client will need to have something like cron job running, and care for return value of the request, etc.
Variant II: Client subscribe to the data feed and will have the data pushed to it when it's ready. It'll need to implement the submission outlet of the aggregation api and then it could just sit and happily wait for new data.
Which variant you'd prefer, if you have choice?


Again, the "If-modified-since" header's purpose is to notify about changes to a data source that is changed irregularly, or requested at irregular intervals.
With raw market data, it is always changed. Rather - it's always new. The issue is that it could be duplicated (say, pilot A using multi-feed uploader, pilot B only uploading to EVE-c. EM will want the data from EVE-c, but only these from pilot B, as it's already have data from pilot A), or absent (if something broken, or you gone wrong and forcing everyone to crawn through RESTful URL's in search for new uploads).


My question was, if you know any HTTP clients, that could have issues dealing with packed JSON data. I'm only working with PHP, and I know that, at the very least, I could decode the stream manually, before feeding it to parser.