These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Delay of the CREST beta for developers

First post First post
Author
Previous Topic Next Topic
Dan Hour
Greater Order Of Destruction
#61 - 2012-11-21 00:34:49 UTC
Not in the first iteration of the API but are you guys considering something other than rest as well? Say something with websockets. If the API connects to the in game chat at all it would be very cool. However last time i checked Nginx did not have much support for websockets proxies. (Insert plug for xmpp with json).
CCP Zero Gravitas
C C P
C C P Alliance
#62 - 2012-11-21 12:04:58 UTC
CREST currently uses Twitter style HTTP streaming to push notifications to clients. One of the reasons we chose Nginx is that it supports very high numbers of concurrent connections, which is required if each client is keeping an HTTP request open permanently to receive notifications.

The latest version of contactjs (https://github.com/jimpurbrick/contactjs) demonstrates how to receive and filter notifications from JavaScript. You make an AJAX request, then instead of waiting for it to complete, periodically check the received data and handle any new notifications that have arrived.

I imagine the notifications could be sent over websockets relatively easily, but HTTP streaming works well on a wide variety of plaforms including the PS3 which we need to support to send notifications to Dust.

CREST wrangler
Shellac Brookdale
Cutting Edge Incorporated
#63 - 2012-11-21 16:54:42 UTC
The way I understand your code in contact.js works is that the complete contact list is re-rendered upon successfully (indicated by http status) calling the contactListUri endpoint, as well as after receiving a contact list update notification. So when exactly can we assume an operation was completed? Will notification have any details about what actually has been changed in the contact list?

Dan Hour
Greater Order Of Destruction
#64 - 2012-11-21 17:42:39 UTC
I can settle for that, but how about that chat part? Granted I've never given the in game chat system much thought about how it was implemented, but anyway I'm 99.9% positive the PS3 talks xmpp. I'd really like too do something like an irc/xmpp protocol bridge with the in game chat. It's been a while since i watched that fanfest (I think it was fanfest) video about CREST. So is chat something thats within the scope of CREST, can you say? Or what isn't within the scope of crest? Sorry if I'm asking something here that was already answered...
Liu Ellens
Sebiestor Tribe
Minmatar Republic
#65 - 2012-11-22 07:23:31 UTC  |  Edited by: Liu Ellens
CCP Zero Gravitas wrote:
CREST currently uses Twitter style HTTP streaming to push notifications to clients. One of the reasons we chose Nginx is that it supports very high numbers of concurrent connections, which is required if each client is keeping an HTTP request open permanently to receive notifications.

The latest version of contactjs (https://github.com/jimpurbrick/contactjs) demonstrates how to receive and filter notifications from JavaScript. You make an AJAX request, then instead of waiting for it to complete, periodically check the received data and handle any new notifications that have arrived.

I don't know the Twitter stream and also couldn't find (didn't look far for) the protocol definition for your stream, but how far off is it from "Server-Sent Events" as defined by W3C (http://dev.w3.org/html5/eventsource/) ? There exists a great polyfill for the client side (https://github.com/Yaffle/EventSource) as well.

The very basic example for the client side (taken from W3C site):
Quote:

var source = new EventSource('updates.cgi');
source.onmessage = function (event) {
alert(event.data);
};


(If it's not this format, I guess it's far too late in the game to ask for a change there, right? ;) )

Well, they oughta know what to do with them hogs out there for shure.
Dan Hour
Greater Order Of Destruction
#66 - 2012-11-22 08:21:59 UTC
Liu Ellens wrote:

I don't know the Twitter stream and also couldn't find (didn't look far for) the protocol definition for your stream.


here is a link to twitter doc for their streaming api: https://dev.twitter.com/docs/streaming-apis

You can kind of think of it as a tcp connection that remains open and the server will write http down the pipe.

I would imagine that they wont implement Server sent events as that is mainly a browser thing, what i mean by that is that the browser is responsible for the connection and not your code, so while it could be an elegant solution it is limited in that way. Plus it does stuff with dom events?? No one likes dom....
Liu Ellens
Sebiestor Tribe
Minmatar Republic
#67 - 2012-11-22 11:51:19 UTC
Dan Hour wrote:

here is a link to twitter doc for their streaming api: https://dev.twitter.com/docs/streaming-apis

Thank you for the link!

The focus of my request was a little bit different though, that didn't come through as I intended. Conceptually, the Twitter stream and the server-side events are pretty much similar, Twitter sends objects in JSON delimited with CRLF, server-side events are text messages (thus also applicable for JSON) delimited by an empty line.
The benefit I saw with the server-side events is that the data format is also a defined one and even supported on the client side out-of-the box, in an event-driven fashion.

I'd be equally happy with the Twitter variant (or any other defined one), there would most likely be corresponding libraries already doing the legwork.

tl;dr: I was asking for a defined stream format, ideally with existing library support on the client side. The W3C variant seemed just the most neutral ("standard") one to me. I like to build all the underlying things myself, but I don't want to reinvent every wheel all the time, especially if I know there is already one out there. ;)

Well, they oughta know what to do with them hogs out there for shure.
Dan Hour
Greater Order Of Destruction
#68 - 2012-11-22 21:02:52 UTC  |  Edited by: Dan Hour
I see what you mean now, have you done much work Server Sent Events? I personally have never touched them, even though all the work i do is "real time" Server Sent events have just never fit the bill. By that I mean all the talking between client and server is too bi-directional. I had mentioned websockets early in the thread because you can send information both ways, faster and with less overhead than making lots of rest calls. Where server sent events is just server to client and is more limited to browser (please correct me if I'm wrong about that).

The point that I'm getting at with the Server Sent Events is that they are limiting the clients to the browser (again correct me if im wrong about that), where with CREST api your "clients" maybe other applications. There would be a security concern if you are making direct CREST api calls from your browser. The user after they got their Oauth token could then make any CREST call that YOUR developer key could use. Leaving your application open for unintended abuse.

I believe the intended architecture is:
YourClient <---> Your App <---> CREST api and not Your APP ---> Your Client ---> CREST api

but now im starting to ramble and have drifted away from server sent events

TL;DR: ramble ramble use case for websockets = chat, use case server sent events? market data maybe? but that seems more like PubSub (*cough* websockets)territory.... oh boy must stop rambling.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#69 - 2012-11-22 21:17:43 UTC
Dan Hour wrote:
I see what you mean now, have you done much work Server Sent Events? I personally have never touched them, even though all the work i do is "real time" Server Sent events have just never fit the bill. By that I mean all the talking between client and server is too bi-directional. I had mentioned websockets early in the thread because you can send information both ways, faster and with less overhead than making lots of rest calls. Where server sent events is just server to client and is more limited to browser (please correct me if I'm wrong about that).

The point that I'm getting at with the Server Sent Events is that they are limiting the clients to the browser (again correct me if im wrong about that), where with CREST api your "clients" maybe other applications. There would be a security concern if you are making direct CREST api calls from your browser. The user after they got their Oauth token could then make any CREST call that YOUR developer key could use. Leaving your application open for unintended abuse.

I believe the intended architecture is:
YourClient <---> Your App <---> CREST api and not Your APP ---> Your Client ---> CREST api

but now im starting to ramble and have drifted away from server sent events

TL;DR: ramble ramble use case for websockets = chat, use case server sent events? market data maybe? but that seems more like PubSub (*cough* websockets)territory.... oh boy must stop rambling.


Kinda yes, kinda no.

Because your app might be running directly in their client. The joys of Javascript.


As for the rest, you can quite happily have an application that runs and talks to crest on the uses behalf. they just need to have an auth token given to them.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Dan Hour
Greater Order Of Destruction
#70 - 2012-11-22 21:30:19 UTC  |  Edited by: Dan Hour
Steve Ronuken wrote:
Kinda yes, kinda no.


To what exactly are you referring, in the abuse case what i mean is if you are make CREST api calls from javascript then you as a developer have lost control what calls are being made by Your application with the developer agreement, what ever it may be. And to me that would be a massive mistake. Allowing your web base client to make calls to the crest api is too open to abuse, and create more spots for security problems.

Im assuming that the CREST api agreement will before YOUR appilcation, so if you move those calls to the api to client without having to go through your app then you are leaving yourself open to having ccp shutdown you access to the api because some **** decided to make a ton of calls before the oauth token expires

Yes i agree your application should make calls or your users be haft that is not the security point i was making.
Liu Ellens
Sebiestor Tribe
Minmatar Republic
#71 - 2012-11-23 08:02:01 UTC
The selection of the transport method should not be driven by the fictional requirement to be able to use it easily in the browser, you're right, Dan - Thanks for reminding me of the security concerns, I'm with you there.

Server-side events are similar to the Twitter stream, essentially a long-poll HTTP request, where the response has no end and consists of chunks of messages. (I'm using them for upro) - I don't exactly understand though what you mean by "limiting the clients to the browser" - all three discussed variants (WebSockets, Twitter, SSE (Server-side events)) can be used by any language/framework that is capable of creating and handling a HTTP request.

From an abstract point of view, we have the difference in the way how the client sends its request to the server.
For Twitter, SSE and CREST, requests are sent in dedicated HTTP requests each (but the downstream event notification is always done through one long-poll). WebSockets would allow the requests to be sent reusing the same connection used for the downstream events.

I guess it comes down to what is supported by the used stacks, and I believe to have found the reason why WebSockets wont be available: Using nginx will deny you WebSockets as they apparently are not supported (first google hit: http://serverfault.com/questions/391274/linux-web-servers-that-support-websockets) (that is without knowing the versions involved)
Short: Only a long-poll variant is possible for CREST.

I was asking for a standard format for the CREST event downstream, but since there is no explicit need to have it easily supported in the browser, support for dissecting an input stream on our own is easy anyway in any other framework (node.js, php, java, what have you)

Well, they oughta know what to do with them hogs out there for shure.
Dan Hour
Greater Order Of Destruction
#72 - 2012-11-23 08:17:13 UTC  |  Edited by: Dan Hour
Yes cool thanks for confirming that nginx still does not support websockets, I knew that was the case awhile ago and mentioned in an earlier post. While I would love them to use websockets I'm defiantly not expecting them in when they first launch CREST. Also thank you for the clarification on Server Sent events, like i said i have never used them so I don't claim know much about them. Why I'm so heart set on WebSockets is because I'm hoping the in-game chat will someday be in the scope of CREST, and WebSockets are a two way pipe.... All of my programing experience has been in "real time" so I'm just bitter and don't want to do any kind of long polling.


and I will be more than willing to give any CREST using web app a free one day pen test if they are ok with it!

EDIT: Nginx has plans to add web socket support as well? I think... going to ask google now
EDIT2: Nginx is plan to support websockets in version 1.3.x