I've finally spent the time to get my roguelike mud displaying in the browser, using websockets and a terminal as provided by term.js (on github somewhere).

One workaround I had to do, was to send my data as binary frames. If I send them as text frames, I get a browser-side error of "WebSocket connection to 'ws://127.0.0.1:9876/stuff' failed: Could not decode a text frame as UTF-8." and the browser disconnects because of it. This is because of the embedded escape sequences. If I send them as binary frames, it arrives as a blob or an arraybuffer and converting it back to a proper utf-8 string results in the following code being required:

var carr = new Uint8Array(e.data);
          var encodedString = String.fromCharCode.apply(null, carr);
          decodedString = decodeURIComponent(escape(encodedString));

This is of course a giant hack to workaround the presumed inability of javascript to interpret an array of data as a unicode string. Has anyone else had to go through this, and come up with a better solution?

Obligatory screenshot of hack in action:


Has anyone else gotten their web socket implementation working, having dealt with the same problems, and worked out a better way to do it?

If I were you, I'd worry less about the workaround, and more about using binary websocket. To my knowledge, of the popular browsers, only recent Chrome versions support it, with caveats. Has the situation changed? If not, it kind of defeats the purpose of running in a browser.

I advise just using a higher level library to handle it. You need a lot of fallbacks to websockets currently to get good coverage no matter what.

I've used socket.io in the past.

Ah, thanks. That's good to know. This stackoverflow post seems to be relevant to the state of things.

The price is too high, but thanks anyway for the suggestion.

If you want to roll your own, I suspect the leanest solution would be to convert ansi control codes and special characters on the server, send as string, and convert them back in the browser. That will give you 75%+ browser coverage, according to most stats. (If you are committed to having a web-based client, you'll also have many opportunities to optimize the data for the client).

You can also roll your own fallback to Flash by using the minimal Flash telnet object I uploaded several years back to the repo. It will be easy once you are able to send the data as a string.

For the Portal app, I'm using another approach. Since I can't control what the servers output, I base64 encode the data in the proxy. I also turn on compression (unless the data is already compressed with MCCP), which more than offsets the inflation caused by base64. It's a lot more to do when the data hits the browser than what you're showing, and still the communication feels very responsive even on an average box. So I'm probably not going to touch it for a few years, until binary websocket becomes standard.

Erm, socket.io is open source - i.e. free. The price is too high? I'd recommend socket.io, too.

If you control both ends, you can always just base64 encode your binary strings. I've done that with database tables before, because people set the database encoding to UTF-8, which tries (and fails) to interpret escape codes as UTF sequences.

Good ideas, thanks. But binary websocket is good enough for me, for now. What you suggest are rainy day projects if things work well enough that support for older platforms comes onto the agenda.

There are other kinds of prices and costs involved in using someone else's solution.

This is what most people seem to do. I've seen figures of 30-50% additional overhead for doing it. And even people rolling their own base64 encoding/decoding solutions. What a sea of hacks web developing is and always has been.. :cry:

While skimming the HTML standard as a refresher course, I came across a solution that does away with the need to force reinterpretation of the arriving data as UTF-8. Pretty much every unicode character I use has a corresponding character reference name. The downsides are obvious. Anyway, back to productive work.

charMap = {    
    "unicode" : {
        # Map character mappings.
        WALL_TILE1: u"\u2591", # Light shade.
        WALL_TILE2: u"\u2592", # Medium shade.
        WALL_TILE:  u"\u2593", # Dark shade.
        FLOOR_TILE: u"\xB7",   # Middle dot.
        DOOR_TILE:  u"\u25FC", # Black square.
        CUBE_TILE:  u"\u2588", # Full block.
        CHAR_TILE:  u"@",

        # Line-drawing characters.
        "light-horizontal":         u"\u2500",
        "light-vertical":           u"\u2502",
        "light-down-and-right":     u"\u250C",
        "light-down-and-left":      u"\u2510",
        "light-up-and-right":       u"\u2514",
        "light-up-and-left":        u"\u2518",
        "full-block":               u"\u2588",
        "medium-shade":             u"\u2592",
        "black-square":             u"\u25FC",
        "white-square":             u"\u25FB",
    },
    "html" : {
        # Map character mappings.
        WALL_TILE1: "░",
        WALL_TILE2: "▒",
        WALL_TILE:  "▓",
        FLOOR_TILE: "·",
        DOOR_TILE:  "◼",
        CUBE_TILE:  "█",
        CHAR_TILE:  u"@",

        # Line-drawing characters.
        "light-horizontal":         "─",
        "light-vertical":           "│",
        "light-down-and-right":     "┌", 
        "light-down-and-left":      "┐",
        "light-up-and-right":       "└",
        "light-up-and-left":        "┘", 
        "full-block":               "█",
        "medium-shade":             "▒",
        "black-square":             "◼",
        "white-square":             "◻",
    }
}

For other readers interested in other options:

http://firehose.io/

Very simple way to expose data to web socket clients.

I like it because it's self hosted and decoupled from your project.

Heh, well, you can also get around the transmission overhead if you first pack your data with bzip2 (or whatever compression algorithm is easiest to use on both ends)… then you transfer the load to the CPU.

base64 would only be useful to me if at a later stage I needed to use text frames in websockets. It only encodes/decodes ASCII, and to get it to handle UTF-8, you need to take the same approach you see in my original post with escaping the bytes, then calling decodeURIComponent() to get a UTF-8 string.

But that decodeURIComponent() approach, actually gives URIErrors for me. I've had to write my own code to turn a sequence of bytes into a UTF-8 string. And whether I plug in base64 and text frames in websockets, or stick with binary frames, I'd still have to use that code.

It's interesting that for all these solutions, if you go to their web pages, they never come right out and say: "For this to be useful, you need to be using node.js and javascript server-side" or "For this to be useful, you need to be using ruby server-side". Instead, you have to discern this from the details.

Ummmm, no. Base64 encodes binary data, 8 bits at a time, into a 6-bit representation using a subset of ASCII. You can pass it a zip file if you want, and it will happily output plain text that you can safely pass along over web sockets or whatever else you can think of. What the source was is irrelevant. If you take your data and pass it through gzip/bzip2, and then pass it through base64, you get text. If you then take that text on the other end, decode the base64, uncompress it, you'll have your original data, as it was on the source end.

Now, if you're saying the remote end won't know what to do with UTF-8 strings after you recover them from the transmission… that's a different problem entirely. I was addressing the idea of allowing you to transmit UTF-8 data from the server to the client and have it show up there uncorrupted.

I'm going by what is written on this page.

The page deals with a very specific decoding issue, as Quix presciently indicated. To say that Base64 "only encodes/decodes ASCII" is simply incorrect.

Huh? Encode a UTF-8 encoded string comprised of wide characters, and it decodes as a string of 8-bit characters effectively "ASCII encoded". Therefore Base64 effectively "only encodes/decodes ASCII". I'm not sure what's simply incorrect about this.

This is what I read him explicitly saying above. That it won't give you preserved UTF-8 wide characters client-side, it would merely give you "uncorrupted data" as what is effectively a sequence of ASCII characters.