Before I seriously consider writing a WebSockets implementation for Wisp, I thought I'd better check no-one else is half way through one!??

I've been reading the WebSocket specs and it seems quite do-able.

The only implementation of WebSockets written in Fantom that I know of is the one in Tales Framework: Websockets.

Yeah, the Tales version is a tiny wrapper around the netty implementation - netty being the java server it uses.

I'm currently liking the idea of a pure fantom version to be plugged into Wisp / afBedSheet.

Having it built right into Wisp would be very cool. This has been on my todo list for a long time, but hasn't bubbled to the top b/c we haven't had an internal need for it yet. If you want to try your hand at it, then I would suggest as first step is to propose changes to the web APIs which will provide the published support for it.

I think the WebMod api can handle WebSockets as it is. My only concern would have been an auto-closing of streams - but looking deeper at the wisp code, it doesn't seem to happen. If need be, I can always branch web / wisp to create a working prototype.

Alas, it appears other work is looming, so I may have to put this on hold for a bit...

Hi Brain,

So after a bit of playing around, I can send and receive WebSocket text messages to / from Firefox.

I'm not sure at which layer you envision WebSockets being incorporated (wisp / web / route) but a WebMod could fit quite nicely as it essentially services (or upgrades) a particular path.

To make WebSockets work, I did need to make some changes to Wisp. The web API could potentially be changed to reflect this, but I'm probably not the right person to suggest what these changes should be. Instead I'll explain what I've done and leave the Fantom changes to the experts...

WebSockets need access to both the request InStream and the response OutStream. At the moment, Wisp denies you access to these should the req / res not meet certain criteria.

Request InStream: WispActor.initReqRes calls WebUtil.makeContentInStream which wraps the InStream if the req contains either a Content-Length or Transfer-Encoding header. If the stream was not wrapped, the request InStream is assumed to be useless and assigned to null.

if (req.webIn === req.socket.in) req.webIn = null

As a WebSocket upgrade request contains neither header, Wisp gives a null InStream. So for now I've just commented out the above line.

// if (req.webIn === req.socket.in) req.webIn = null

Response OutStream: WispRes.out supplies the response OutStream, but only after the commit method has wrapped it in a WebOutStream. Again it is only wrapped if the response contains either a Content-Type or Content-Length header. If it was not wrapped, null is returned.

cout := WebUtil.makeContentOutStream(&headers, sout)
if (cout != null) webOut = WebOutStream(cout)

Again, as a WebSocket upgrade request contains neither header, so I changed the above to:

cout := WebUtil.makeContentOutStream(&headers, sout)
webOut = (cout != null) ? WebOutStream(cout) : WebOutStream(sout)

Actually, the above code is wrapped in an if (isPersistent) block - the Socket OutStream needs to be available to WebSockets regardless of the connection being kept alive or not.

An aside: WebSockets doesn't need a WebOutStream, and to be honest, I always found it a bit presumptuous of WebRes to return one - assuming HTML will always be returned and generated from the WebOutStream methods too! Pfft!

If web gave access to the underlying socket streams then the above would be a lot simpler - but I guess also dangerous.

Thoughts?

My reaction is that we shouldn't publically expose the sockets to the web API. Rather I'd more inclined to provide some sort of hooks wisp module could use (of course it is the one that has access to raw socket to begin with)

How about if the above WispActor.initReqRes() and WispRes.commit() methods checked for the mere presence of an "Upgrade" header before deciding to null out the In and Out streams?

I'm thinking that because Upgrade requests won't necessarily follow standard HTTP protocols, it should be safe not to null them.

I guess what I was thinking is that Upgrades get handled by some plugin which would then take over the in/out stream and then ensure that wisp doesn't try to reuse the socket for pipelining.

Are you just building websockets as a normal WebMod? Do you have some sample code of how your prototype would work?

Yep, it's just a normal WebMod that services a request on a url.

If the request doesn't contain the correct HTTP headers then a 400 - Invalid Request (or similar) is returned. If the WebSocket handshake goes okay then the WebMod service method goes into a loop, holding on to req InStream and res OutStream. So they shouldn't get used for Pipelining, as wisp doesn't get a change to use them until the WebSocket connection is finished.

Sample working code below:

using wisp
using webmod
using concurrent

class WispApp {

  static Void main(Str[] args) {
    fileMod := FileMod() { it.file = `test-app/websocket.html`.toFile }
    wsMod   := WebSocketWebMod(WispApp#wsHandler)

    routes  := ["test": fileMod, "websocket": wsMod]
    root    := RouteMod { it.routes = routes }
    
    WispService { it.port=8080; it.root=root }.start
    Actor.sleep(Duration.maxVal)
  }
  
  static Void wsHandler(WebSocket webSocket) {
    webSocket.onMessage = |MsgEvent me| {
      echo(me.msg)
    }
  }
}

With the websocket.html page:

<html>
<head>
  <title>Web Socket Demo</title>
  <script type="text/javascript">
function WebSocketTest() {
  var ws = new WebSocket("ws://localhost:8080/websocket");
 
  ws.onopen = function() {
    var count = 1;
    setInterval(function() {
      if (ws.bufferedAmount == 0) {
        ws.send("Hello! - " + count);
        count = count + 1;
      }
    }, 1000);
  };
 
  ws.onmessage = function (evt) { 
    var received_msg = evt.data;
    console.warn(evt);
  };
 
  ws.onclose = function(event) { 
    alert("Connection is closed...");
    console.warn(event);
  };     
}
  </script>  
</head>

<div>
   <a href="javascript:WebSocketTest()">Run WebSocket</a>
</div>
</html>

The src is on this BitBucket repository if you need a closer look. Here's the WebSocketWebMod for example.

I was thinking that Upgrades get handled by some plugin

Yeah, they could, but then it's a lot of extra configuration which (I think) could get messy with the current design of wisp / WebMod...

The idea of a WebSocket upgrade is that it starts out as plain HTTP GET with a couple of extra headers. I think a WebMod works because you're not going to use the same URL for GET / POST and UPGRADES - that's just twisted!

Then again, how about the following:

web::Weblet.onUpgrade(Str type, InStream reqIn, OutStream resOut)

which nestles in nicely with all the onService / onGet / onPost methods.

That way you get the choice to handle it (or not) and are provided with the raw In / Out streams. (type is from the upgrade header and should be websocket).

Actually I really like the onUpgrade design. That seems clean and elegant to me. Not sure we need to do anything special with parameters, rather we can just access that off req/res.

So following that design, we would have:

1. add virtual Void onUpgrade() to Weblet
2. add bit of code in Wisp and/or Weblet.onService to handle

Do you know what the minimum changeset would be to make that work for you?

Just ensure the In / Out streams aren't null regardless of if Weblet.onUpgrade() is handled or not. (Though they can still be null on non-upgrade requests.)

The example WsWebMod is easily changed to use onUpgrade() but afBedSheet hinges entirely on the onService() method, so I'd really like everything to work from that.

As for checking for valid HTTP Upgrade requests, you can look at WebSocketCore - essentially it just needs to be a HTTP 1.1 GET request with a Host header and a Connection header that contains the word upgrade. (The rest is checking for a valid WebSocket upgrade.)

+1 for onUpgrade

Hi, so I decided to revisit WebSockets.

Below is a small patch for Wisp that allows access to the req / res streams when a HTTP Upgrade request is received.

diff -r 2f6a97c9d4fd src/wisp/fan/WispActor.fan
--- a/src/wisp/fan/WispActor.fan	Mon Aug 03 15:31:58 2015 -0400
+++ b/src/wisp/fan/WispActor.fan	Sun Aug 09 10:45:21 2015 +0100
@@ -175,7 +175,17 @@
     // Content-Length or Transfer-Encoding - which in turn means we don't
     // consider this a valid request for sending a body in the request
     // according to 4.4 (since pipeling would be undefined)
-    if (req.webIn === req.socket.in) req.webIn = null
+    if (req.webIn === req.socket.in)
+    {
+      if (isUpgradeRequest(req.headers))
+      {
+        res.isUpgrade = true
+      }
+      else
+      {
+        req.webIn = null
+      }
+    }
 
     // init response - set predefined headers
     res.headers["Server"] = wispVer
@@ -194,6 +204,11 @@
     Locale.setCur(req.locales.first)
   }
 
+  private Bool isUpgradeRequest(Str:Str reqHeaders)
+  {
+    reqHeaders.get("Connection", "").split(',').any |tok| { tok.equalsIgnoreCase("upgrade") }
+  }
+
 //////////////////////////////////////////////////////////////////////////
 // Error Handling
 //////////////////////////////////////////////////////////////////////////
diff -r 2f6a97c9d4fd src/wisp/fan/WispRes.fan
--- a/src/wisp/fan/WispRes.fan	Mon Aug 03 15:31:58 2015 -0400
+++ b/src/wisp/fan/WispRes.fan	Sun Aug 09 10:45:21 2015 +0100
@@ -191,7 +191,7 @@
       // socket stream is wrapped as either a FixedOutStream or
       // ChunkedOutStream so that close doesn't close the underlying
       // socket stream
-      if (isPersistent)
+      if (isPersistent && !isUpgrade)
       {
         cout := WebUtil.makeContentOutStream(&headers, sout)
         if (cout != null) webOut = WebOutStream(cout)
@@ -240,5 +240,6 @@
   internal TcpSocket socket
   internal WebOutStream? webOut
   internal Bool isPersistent
+  internal Bool isUpgrade
 
 }
\ No newline at end of file

The above is an example of what's needed to make WebSockets work in Wisp.

And this is a patch for web::Weblet that adds an onUpgrade() method. This part is somewhat optional:

diff -r 2f6a97c9d4fd src/web/fan/Weblet.fan
--- a/src/web/fan/Weblet.fan	Mon Aug 03 15:31:58 2015 -0400
+++ b/src/web/fan/Weblet.fan	Sun Aug 09 10:52:31 2015 +0100
@@ -56,7 +56,11 @@
   {
     switch (req.method)
     {
-      case "GET":     onGet
+      case "GET":
+        if (req.headers.get("Connection", "").split(',').any |tok| { tok.equalsIgnoreCase("upgrade") })
+          onUpgrade
+        else
+          onGet
       case "HEAD":    onHead
       case "POST":    onPost
       case "PUT":     onPut
@@ -77,6 +81,15 @@
   }
 
   **
+  ** Convenience method to respond to a Socket Upgrade request.
+  ** Default implementation returns a 501 Not implemented error.
+  **
+  virtual Void onUpgrade()
+  {
+    res.sendErr(501)
+  }
+
+  **
   ** Convenience method to respond to a HEAD request.
   ** Default implementation returns a 501 Not implemented error.
   **

Ah, I forgot to re-set a read timeout on the socket when upgrading. Wisp's default of 10 seconds isn't that useful when idling / waiting for messages!

I set it below to 5 mins, but that's never going to be enough for some people. And defaulting to an infinite timeout seems, um, wrong. So I would suggest picking it up from a wisp config variable in web.props or a new wisp.props.

New wisp patch follows:

diff -r 2f6a97c9d4fd src/wisp/fan/WispActor.fan
--- a/src/wisp/fan/WispActor.fan	Mon Aug 03 15:31:58 2015 -0400
+++ b/src/wisp/fan/WispActor.fan	Sun Aug 09 11:18:46 2015 +0100
@@ -175,7 +175,18 @@
     // Content-Length or Transfer-Encoding - which in turn means we don't
     // consider this a valid request for sending a body in the request
     // according to 4.4 (since pipeling would be undefined)
-    if (req.webIn === req.socket.in) req.webIn = null
+    if (req.webIn === req.socket.in)
+    {
+      if (isUpgradeRequest(req.headers))
+      {
+        res.isUpgrade = true
+        req.socket.options.receiveTimeout = 5min
+      }
+      else
+      {
+        req.webIn = null
+      }
+    }
 
     // init response - set predefined headers
     res.headers["Server"] = wispVer
@@ -194,6 +205,11 @@
     Locale.setCur(req.locales.first)
   }
 
+  private Bool isUpgradeRequest(Str:Str reqHeaders)
+  {
+    reqHeaders.get("Connection", "").split(',').any |tok| { tok.equalsIgnoreCase("upgrade") }
+  }
+
 //////////////////////////////////////////////////////////////////////////
 // Error Handling
 //////////////////////////////////////////////////////////////////////////
diff -r 2f6a97c9d4fd src/wisp/fan/WispRes.fan
--- a/src/wisp/fan/WispRes.fan	Mon Aug 03 15:31:58 2015 -0400
+++ b/src/wisp/fan/WispRes.fan	Sun Aug 09 11:18:46 2015 +0100
@@ -191,7 +191,7 @@
       // socket stream is wrapped as either a FixedOutStream or
       // ChunkedOutStream so that close doesn't close the underlying
       // socket stream
-      if (isPersistent)
+      if (isPersistent && !isUpgrade)
       {
         cout := WebUtil.makeContentOutStream(&headers, sout)
         if (cout != null) webOut = WebOutStream(cout)
@@ -240,5 +240,6 @@
   internal TcpSocket socket
   internal WebOutStream? webOut
   internal Bool isPersistent
+  internal Bool isUpgrade
 
 }
\ No newline at end of file

Hi Steve, why don't you send me a private email and we can hash out what exactly you are doing, and how we might plug your side work into Wisp. I'm actually not sure I really love onUpgrade after more thought, but I'd like to see more of what you are doing first

So, still no WebSockets in Wisp ? Just drafts?

It's in web not wisp - see web::WebSocket. It claims to be a prototype but I'm pretty sure it's what Brian's ArcBeam is based on.

Or, if you want an implementation that adheres to the W3C WebSocket API and RFC 6455 and hence works on both the client (Javascript) and server then there's always my own WebSocket library:

afWebSockets

As Steve said, its part of the standard API in web::WebSocket. Its not a prototype and heavily used in production, so I need to remove that note in the comments.

Brian, well to hear that, could you also add simple example to examples section how to utilize it both for client and server usage? Thanks!

@SlimerDude, thanx for answer! Maybe its an option to go with.

Does web::websocket works with wisp ? Lets assume we have server, which upgrades incoming websockets and store them to map or list, how to "drive" all websockets? How to define callbacks for reading messages from it?

I've been using Fantom's WebSocket client implementation. In order to get to work with my NodeJS server, I had to add the additional sec-websocket-version http header. I set the version to 8. I have no idea if this version is correct or not.

It's good to see ping and pong implemented. Ping and pong help with detecting whether a connection is really open or not. Therefore it would be nice if WebSocket could have an additional field that has the last time a pong was received. This could then be checked by a client to see if it's still open.

Please note, to anyone implementing a client using WebSocket, the receive method is blocking. Therefore each client will require it's own thread that will need to be interrupted to shutdown.