These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Why are we not combining market data, player devs?
Author
Previous Topic Next Topic
Dragonaire
Here there be Dragons
#141 - 2011-12-09 00:06:40 UTC
I can't find the the link or remember the search that lead to it but saw where someone had done research into how well several different UUID generator were at being truly random with version 4 and they found they weren't that random really as thought because most were based on the pseudo-random number generator in the system which all had clustering issues so the UUIDs also ended up being cluster over a much smaller range than there should be. It was mostly because if errors in the implementations but across all systems they saw biases that shouldn't have been there so it's a much small problem set than one would think by the number of bits used.

Ok on the push vs pull with aggregator-to-aggregator my concern is if the client isn't ready or able to receive the update then it's lost to it because there's no way to ask for it again. after all they are hopefully receive data from their own uploaders so I just figured it would make more sense for them to request it when they know they have time to work with it. It's also much easier for the another sites to poll and see if the other site is down then trying to push something to someone else and keep having to ask did you get it? you don't know if they got it and can't or decided to ignore you so after a time maybe you try it again and maybe decided to give up after a couple tries. With pull if a client can't get the data they can decided themselves if and when they really want to try again. Both systems can work but I think if it was me designing the site I'd pay attention to people like Yahoo, Google, etc that killed off they push services in favor to pull because of the easy in scaling that can be done with pull vs push and other issues they had. One thing to keep in mind is it easy to proxy reads but you can't proxy writes so push is limited to how much data one server can send but pull is limited by only how many servers and proxies you can set up. But that's someone else's problem to deal with Big smile

The rest gets to wait until after work P

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Tonto Auri
Vhero' Multipurpose Corp
#142 - 2011-12-09 00:23:02 UTC
With what I've proposed earlier (hash check for data submission), aggregator-to-aggregator push would not run if the remote party did not accept the sequence. And it will as well guard against "did you get it?" cases.
I just didn't though about a good way to create hash for a submission block, yet.
Said that, it could be lost by a number of reasons not dependent on the way it is initiated, which making your concern a bit less valid.
Turning around, I don't have any innate protest against pull services, there's a room for both, but IMO, you've picked the wrong comparison. Yahoo and google don't syndicate site indexation results (for example). We're speaking about messaging services here. SMTP coming to my mind.

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Dragonaire
Here there be Dragons
#143 - 2011-12-09 05:37:48 UTC  |  Edited by: Dragonaire
Ok I'll take it back it looks like they have start the swing back to push again Phttp://en.wikipedia.org/wiki/Push_technology But as you're finding it is a bit harder to implement. I can see it working for the site to site stuff but it doesn't work as well to end users where usually for security their firewalls block push services.

So anyone took a look at my idea how to reorder stuff to allow multiple data groups in message? Any one have any other ideas or see any problems with it?

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Kaladr
Viziam
Amarr Empire
#144 - 2011-12-09 06:05:13 UTC
As far as push is going, I'm playing with three technologies:


  1. Long polling sockets. This is currently infeasible with the current architecture, but with the new server stack (which will be slow to migrate to) it will exist
  2. HTTP web-hook callback, ala Github, PubSubHubBub, etc.
  3. XMPP PubSub (Jabber)


The web-hook will likely be implemented first, and meets the needs of inter-site syndication. Long polling is best for "desktop applications", whereas Jabber is usable for both servers and clients but is another stack.

Creator of EVE-Central.com, the longest running EVE Market Aggregator
Tonto Auri
Vhero' Multipurpose Corp
#145 - 2011-12-10 17:14:41 UTC
Seems like to took "pushing" way too literal...

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Kaladr
Viziam
Amarr Empire
#146 - 2011-12-11 06:40:52 UTC
Ok, I am moving towards supporting this format in both Contribtastic and EVE-Central core rewrite (the Scala/Akka thing).

You can watch any progress on github if that is your thing.

Creator of EVE-Central.com, the longest running EVE Market Aggregator
Dragonaire
Here there be Dragons
#147 - 2011-12-11 06:48:36 UTC
So my question is are we going to go with the new proposed version or the original from the wiki?
http://dev.eve-central.com/unifieduploader/start

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Callean Drevus
Perkone
Caldari State
#148 - 2011-12-11 09:29:44 UTC  |  Edited by: Callean Drevus
It seems someone decided we go with the new proposed version ;) in any case, I agree.

I just noticed the duplication of the columns item though, wouldn't it be better to take that one out of the resultset? Duplicating that will probably happen way more often than duplication of the header content (if you wanted to send multiple result types in one message), though it will also allow for a lot of flexibily to what kind of messages you can put in.

In that case however, we also need to add the type of data to each data package, which is now in the header.

UPDATE:

I present to you, the unified uploader: http://www.evemarketeer.com/downloads

You can now add any number of endpoints, have EVE Central lagging behind all other endpoints without any trouble and it be generally awesome and out of your way.

Currently only downloadable as a zip archive, but it'll change once I've worked out all the bugs. I invite you to try it out!

Developer/Creator of EVE Marketeer
yumike
Doomheim
#149 - 2011-12-13 13:36:02 UTC
Syn Fatelyng wrote:
I enjoy the thought of this idea, but am concerned about how quickly that places bandwidth consumption on the users. Instead of one site, they may be uploading to four (if not more). That's an increase of bandwidth by a factor of four, client side.

o/ syn havent shot at you for what seems like ever

A simple enough solution to the problem here (as it is a problem especially for people with smaller lines) is you could upload it to a central site and then all the sites could pull from it, But then theres the problem of ultimately putting up a fifth site that the other four may or may not use. with a "new" app to download & run and therein lies the problem.
Callean Drevus
Perkone
Caldari State
#150 - 2011-12-13 17:20:13 UTC
Actually, if the bandwith consumption of the uploader is the problem, you are probably not able to play EVE anyway. A few kilobytes per second is only going to clog a modem.

That said, there is no reason someone would be forced to upload to all websites. He could simply upload to the one he uses the most.

Developer/Creator of EVE Marketeer
Dragonaire
Here there be Dragons
#151 - 2011-12-14 06:52:16 UTC
Ok I've updated the wiki page descriptions to match the new format.

I do have one question though and that is on the 'generatedAt' date/time what it's used for as it's not totally clear the reason for it being there to me at least. Maybe whomever added it can explain it a little bit? Or we can at least define how it's going to be used and maybe change the name so it's use is more apparent without needed to find it in the wiki etc.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Tonto Auri
Vhero' Multipurpose Corp
#152 - 2011-12-14 22:48:42 UTC  |  Edited by: Tonto Auri
The "generatedAt" could be moved to HTTP Date or Last-Modified header. (If you/your server care to fill it.)
For it's purpose, it could be used in the scenario II, where you compile data off hand and present it to clients as cached payload. But as I said, you can do it the straightforward way by utilising HTTP headers.

Sorry for my slow replies in last few days... Nature of my work is that I'm either free for days, even weeks, and then roof crash/hamsters die and I get full hands of work for several hours at least. This time is was full days >.<
I'll be back, I still have to write protocol explanation, which forum ate last time and I haven't had a time to repost.

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Dragonaire
Here there be Dragons
#153 - 2011-12-15 17:29:45 UTC
Quote:
I'll be back, I still have to write protocol explanation, which forum ate last time and I haven't had a time to repost.
Sorry to hear it got eat I've had that happen a couple times too Sad I'll look forward to reading it when you get back to it.
Quote:

The "generatedAt" could be moved to HTTP Date or Last-Modified header. (If you/your server care to fill it.)
For it's purpose, it could be used in the scenario II, where you compile data off hand and present it to clients as cached payload. But as I said, you can do it the straightforward way by utilising HTTP headers.
That was kind of my thought. That's in part why I decided to ask the question because I don't really see where having another date/time in the message is adding any value but is just using up extra bandwidth.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Callean Drevus
Perkone
Caldari State
#154 - 2011-12-15 20:09:18 UTC
Having a HTTP Date or Last-Modified header would only work if you were to send each message individually. I rather think that if you are going to bunch up order or other data, you will want to know when each was generated.

Developer/Creator of EVE Marketeer
Kaladr
Viziam
Amarr Empire
#155 - 2011-12-15 22:17:29 UTC
Since we're mixing data into one envelope, having the generatedAt per message is very helpful (they could be minute(s) apart)

Creator of EVE-Central.com, the longest running EVE Market Aggregator
Tonto Auri
Vhero' Multipurpose Corp
#156 - 2011-12-16 08:14:18 UTC
Callean Drevus wrote:
Having a HTTP Date or Last-Modified header would only work if you were to send each message individually. I rather think that if you are going to bunch up order or other data, you will want to know when each was generated.

Kaladr wrote:
Since we're mixing data into one envelope, having the generatedAt per message is very helpful (they could be minute(s) apart)


1. You can't reliable obtain this information anywhere, unless you export these orders yourself, which, for web-based application, is highly unlikely.
2. Orders list have all the necessary information already (posted date, amount issued, amount left).

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Shirah Yuri
Tonic Empire
#157 - 2011-12-16 08:30:06 UTC  |  Edited by: Shirah Yuri
Tonto Auri wrote:
2. Orders list have all the necessary information already (posted date, amount issued, amount left).


For orders, not only it is interesting when the order was posted, but also when it was observed. This information can be used as an offset for all order dates when trying to find out current market activity. Personally, especially when multiple order groups can be transferred, I would prefer a timestamp like this "generatedAt" associated with each one, fully concurring with Kaladr.
Dragonaire
Here there be Dragons
#158 - 2011-12-16 09:03:37 UTC
Ok I now better understand where for the aggregators having data on when the rows were collected is important so I have no problem with 'generatedAt' I just wanted to see why it was needed.

Quote:
I just noticed the duplication of the columns item though, wouldn't it be better to take that one out of the resultset? Duplicating that will probably happen way more often than duplication of the header content (if you wanted to send multiple result types in one message), though it will also allow for a lot of flexibily to what kind of messages you can put in.
You are right that having 'columns' in each of the 'results' is very flexible but also a lot of overhead. The question is does anyone really need the much flexibility and if not how to fix it.

We could bump columns up directly under 'results' but then we'll need to add another container like 'rowset' Big smile to hold the array of rows. That would work but still leaves the option open to add or move the columns around but only at the message level which is probably all that would be needed.

Example:
"results" : {
"columns" : ["price","volRemaining","range","orderID","volEntered","minVolume","bid","issueDate","duration","stationID","solarSystemID"],
"rowset" : [
{
"generatedAt" : "2011-10-22T15:43:00+00:00",
"regionID" : 10000065,
"typeID" : 11134,
"rows" : [
[8999,1,32767,2363806077,1,1,false,"2011-12-03T08:10:59+00:00",90,60008692,30005038],
[11499.99,10,32767,2363915657,10,1,false,"2011-12-03T10:53:26+00:00",90,60006970,null],
[11500,48,32767,2363413004,50,1,false,"2011-12-02T22:44:01+00:00",90,60006967,30005039]
]
},
{
"regionID" : null,
"typeID" : 11135,
"generatedAt" : "2011-10-22T15:42:00+00:00",
"rows" : [
[8999,1,32767,2363806077,1,1,false,"2011-12-03T08:10:59+00:00",90,60008692,30005038],
[11499.99,10,32767,2363915657,10,1,false,"2011-12-03T10:53:26+00:00",90,60006970,null],
[11500,48,32767,2363413004,50,1,false,"2011-12-02T22:44:01+00:00",90,60006967,30005039]
]
}
]
}

There are some problems with my example but should we try something like that?

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Dragonaire
Here there be Dragons
#159 - 2011-12-16 16:19:15 UTC
After a little sleep and thought I figured out what didn't seem right with the above example. 'columns' needs to be moved up into main part of the message not just up under 'results'. Also thought 'rowsets' would be a better name now. I made the changes on the wiki instead of trying to fight with the forums to show the changes P

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Callean Drevus
Perkone
Caldari State
#160 - 2011-12-16 19:49:50 UTC
I rather think your first message was more in the right direction, as it would still allow for content of different types to be sent in the same message (with a few modifications). But having only one columns header per content type might save some/much space already.

Developer/Creator of EVE Marketeer