14 Mar, 2009, elanthis wrote in the 1st comment:
Votes: 0
As part of my quest to get ZMP into a few servers and clients, I have to rewrite the TELNET handling of a lot of softare out there. Rather than do it over and over and over again, I brushed up an old half-finished project and put it up on the Web. This project is libtelnet, a "block box" TELNET implementation, similar to the zlib API. You push bytes into it and it sends bytes out through callback mechanisms. That makes it (relatively) easy to splice into almost every C/C++ client or server codebase.

It deals only with the protocol parsing bits of TELNET. It doesn't automatically handle negotiation or any other features. Those remain the application's job.

The code can be found here: http://github.com/elanthis/libtelnet/tre...

Note that the code right now won't even compile, but it's a preview of where I'm going with the API and general implementation. Some half-finished docs are available too. I'm going to be splicing it into tbaMUD once I have the basics nailed down, which will include a pair of small test programs to make sure it's doing everything right. Aside from a few typos I've already spotted, the only major change I can think of left is to make sure the subrequest buffering is correct and efficient. It also shouldn't allocate a buffer at all if the subrequest is "empty" like it is for MCCP2 (no data after the request type byte), as only a small handful of MUD-related TELNET features even need a buffer. ZMP, MSSP, and ENVIRON are all that come to mind.

Any feedback on the general API (keep in mind its purpose as a block box "easy to splice in" code) or implementation would be awesome.
14 Mar, 2009, David Haley wrote in the 2nd comment:
Votes: 0
It looks generally rather reasonable, although I'd need to try embedding it into something to see how nice it is to play with it.
14 Mar, 2009, Les wrote in the 3rd comment:
Votes: 0
It's a good idea and the code as presented is a nice compact implementation. It would be really nice if it supported the q method (or variant) of telnet option negotiation (RFC1143) since that is rather delicate and something that probably shouldn't be pushed off to codebases that don't support the basic protocol at all.

Adding some extremely basic support for a couple of telnet options (ECHO and EOR) might be a good idea?
14 Mar, 2009, elanthis wrote in the 4th comment:
Votes: 0
Funny thing, even I (Mr. "Do It Right") don't use the Q method in my code. I will fixify that for libtelnet.

So far as ECHO/EOR, that really is the apps job. libtelnet makes it relatively easy. The negotiate commands trigger a callback (libtelnet_negotiate_cb) and includes helper routines for sending negotiation commands (libtelnet_send_negotiate) and misc. commands like IAC EOR (libtelnet_send_command).

For example (in semi-C pseudo-code):

muds_player_init(player) {
libtelnet_init(&player->telnet);
libtelnet_send_negotiate(&player->telnet, WILL, EOR, player);

}

muds_flush_code(player) {
libtelnet_send_data(&player->telnet, player->prompt, strlen(player->prompt), player);

if (player->flags.use_eor)
libtelnet_send_command(&player->telnet, EOR, player);

flush_sock();
}

muds_read_code(player) {
buffer[size];

length = read_socket(buffer, size, player->descriptor);
libtelnet_push_buffer(&player->telnet, buffer, length, player);
}

libtelnet_negotiate_cb(telnet, cmd, opt, user_data) {
player = (player*)user_data;

if (opt == EOR) {
if (cmd == DO)
player->flags.use_eor = 1;
else if (cmd == DONT)
player->flags.use_eor = 0;
}
}

libtelnet_input_cb(telnet, byte, user_data) {
player = (player*)user_data;

append_to_line_buffer(player, byte);
process_buffer_if_has_line();
}

libtelnet_output_cb(telnet, buffer, size, user_data) {
player = (player*)user_data;
write_to_descriptor(player, buffer, size);
}


The muds* functions are (at least looking at tbaMUD) in need of relatively little modification. Mostly just replacing code with one or two lines of libtelnet calls. The libtelnet_*_b routines are pretty easy to implement, too, as they are generally just small wrappers around existing code. In a few cases some code has to be shuffled around, e.g. moving the line processing code out of the muds_read_code() fuction into the libtelnet_input_cb() handler.
14 Mar, 2009, Scandum wrote in the 5th comment:
Votes: 0
The telnet lib/handler I wrote (MTH) is more compact, though admittedly it needs better state handling.

I'm not sure if this approach scales all that well, but it looks like a nice implementation for programmers who swing this way.
14 Mar, 2009, elanthis wrote in the 6th comment:
Votes: 0
Scandum said:
The telnet lib/handler I wrote (MTH) is more compact


Yeah, it's certainly smaller, on account of it being technically incorrect and thus fragile in the event of any TELNET command being broken across a read()/recv() boundary, as might happen with a large MSSP response. Or just random luck with even something as short as IAC GA. That is EXACTLY the kind of crappy code that I am ripping out of MUDs and replacing with libtelnet.

Your little example main() test works fine on account of you sending fake data inside of a large buffer. Actual MUD servers have to deal with actual network packet behavior, and most of them use read()/recv() buffers that are much smaller the whatever MAX_STRING_LEN is. Circle uses 512, for example. A heavy MSSP response alone could easily break that.

Either you use a state machine, or your handling of TELNET is broken. The only possible other fully-correct method would involve a lot more buffering and a lot of incremental scanning… which would be quite a bit larger, uglier, and slower than just using the same technique the original authors of the TELNET protocol used: a freaking state machine.

http://svn.freebsd.org/viewvc/base/relen... (original code over 20 years old… but 100% correct and still working perfectly today)

Quote
I'm not sure if this approach scales all that well, but it looks like a nice implementation for programmers who swing this way.


The approach "scales" very well given that it's what all real TELNET servers and clients do. The latest update I pushed also now processes as much of a buffer as it can in one go, and passes as much of the buffer as it can to the callbacks, instead of doing it a byte at a time, which is a fair win.

Cool Stuff Alert!

There is also now a very nice treat: a transparent TELNET proxy that logs all interesting traffic to the terminal… colorized between server and client traffic. It is difficult to express how cool this is for debugging client/server interactions. The very first time I tested telnet-proxy I actually found a SourceMUD bug that's probably been lurking for years! Major awesome.

I figured that if I was going to make a tool to test libtelnet, it might as well be a tool that has real practical use… and there it is! Note that telnet-proxy is a little on the hacky side (it was originally just meant to be a test, after all) so forgive some of the… uh, hacky hackish hackity-hack hacks in it. I'll clean it up some more and make another release shortly.

There is a tarball to download here: http://cloud.github.com/downloads/elanth...

Here is a log of telnet-proxy (can't see the colorization here, unfortunately). The client is the standard BSD Telnet client and the server is a git snapshot of SourceMUD. I snipped out the banner bit because it's big and a bit hideous. (Look at that! NULs in standard TELNET subrequests, and ZMP isn't even being used! Oh my!)

SERVER IAC 251 25
SERVER IAC 251 93
SERVER IAC 253 39
SERVER IAC 253 24
SERVER IAC 253 31
SERVER INPUT: <1B>
SERVER IAC 251 25
SERVER IAC 251 93
SERVER IAC 253 39
SERVER IAC 253 24
SERVER IAC 253 31
SERVER INPUT: <1B>[2J<1B>[H<0D><0A>
—-===[ Source MUD V0.26.20081030 ]===—-<0D><0A>
<0D><0A>
Source MUD Copyright (C) 2000-2005 Sean Middleditch<0D><0A>
Visit http://www.sourcemud.org for more details.<0D><0A>
<1B>[36mBuild<1B>[0m - V0.26.20081030, Mar 5 2009 21:32:25<0D><0A>
<1B>[36mUptime<1B>[0m - 0 days, 0 hours, 1 minute, 32 seconds<0D><0A>
<1B>[36mHost<1B>[0m - localhost.localdomain<0D><0A>
<1B>[36mPlayers<1B>[0m - 0<0D><0A>
<0D><0A>
Enter your account name and password to login. To create a new <0D><0A>
account, simply type <1B>[0;33mnew<1B>[0m.<0D><0A>
<0D><0A>
Enter thy name:
CLIENT IAC 254 25
CLIENT IAC 254 93
CLIENT IAC 251 39
CLIENT IAC 251 24
CLIENT IAC 251 31
CLIENT SUBREQ 31: <00>P<00><18>
SERVER SUBREQ 39: <01><00>SYSTEMTYPE
SERVER SUBREQ 24: <01>
CLIENT SUBREQ 39: <00><00>SYSTEMTYPE
CLIENT SUBREQ 24: <00>XTERM
SERVER INPUT: <1B>]2;Source MUD<07>
CLIENT INPUT: quit<0D><0A>
[/code]

Oh, do note that you will currently need to disable MCCP2 if both the server and client support it: telnet-proxy will happily forward the negotiation/subrequests for MCCP2 through and then get stuck looking at a gzip stream it can't understand. libtelnet itself is not actually compatible with MCCP2 at the moment, as handling that either requires byte-at-a-time API or giving libtelnet the ability to decode the gzip stream. That is of course on the todo list. :)
14 Mar, 2009, Scandum wrote in the 7th comment:
Votes: 0
elanthis said:
Yeah, it's certainly smaller, on account of it being technically incorrect and thus fragile in the event of any TELNET command being broken across a read()/recv() boundary, as might happen with a large MSSP response. Or just random luck with even something as short as IAC GA. That is EXACTLY the kind of crappy code that I am ripping out of MUDs and replacing with libtelnet.

A large server side MSSP response is… 3 bytes, so in practice it'll work. A more robust framework would probably be desirable in regard to future additions.

elanthis said:
Your little example main() test works fine on account of you sending fake data inside of a large buffer. Actual MUD servers have to deal with actual network packet behavior, and most of them use read()/recv() buffers that are much smaller the whatever MAX_STRING_LEN is. Circle uses 512, for example. A heavy MSSP response alone could easily break that.

It's real data, and it's just for testing. Circle muds could break it up into several responses smaller than 512 bytes, or use a bigger buffer.

elanthis said:
Either you use a state machine, or your handling of TELNET is broken. The only possible other fully-correct method would involve a lot more buffering and a lot of incremental scanning… which would be quite a bit larger, uglier, and slower than just using the same technique the original authors of the TELNET protocol used: a freaking state machine.

There is more than one way to implement a state machine, and the original telnet implementation isn't very elegant, but I guess that's personal opinion.
14 Mar, 2009, elanthis wrote in the 8th comment:
Votes: 0
Your one way to implement a state machine is so amazingly above my head that I went and thought that it didn't even maintain any form of state and that it was just a look-ahead scanner. I'm so dumb. Please get more of your code out there so the novices can learn from your wisdom and the gurus can retire knowing you've got all the technical details firmly under control. You win, dude.

On a more relevant to libtelnet note, telnet-proxy and libtelnet itself now have full MCCP2 support in git. It also looks even prettier on account of printing command and options names, including COMPRESS, COMPRESS2, MSSP, and ZMP. Few more things to do, but I'm probably out of steam for the night, it being almost 5am and all. I'm seeing a stray 0xFF byte from gnome-mud when MCCP2 is negotiated – not sure if it's a stray IAC, a bit of leakage from the compression stream (0xFF is spit out a few times at the start of a zlib stream, iirc), or a bug in libtelnet.

I am rather tickled to see the ZMP conversation taking place before my eyes, which a user normally can't see:

CLIENT IAC WILL 31 (NAWS)
CLIENT SUB 31 (NAWS): <00>@<00><14>
CLIENT IAC DO 86 (COMPRESS2)
SERVER IAC WILL 25 (EOR)
SERVER SUB 93 (ZMP): zmp.ident<00>Source MUD<00>0.26.20081030<00>Powerful C++ MUD server software<00>
SERVER SUB 93 (ZMP): zmp.check<00>net.sourcemud.<00>
SERVER SUB 93 (ZMP): zmp.check<00>color.define<00>
SERVER SUB 24 (TTYPE): <01>
SERVER SUB 86 (COMPRESS2)
CLIENT COMPRESSION ON
SERVER COMPRESSION ON
CLIENT DATA: <FF> <– why are you there? I do not know and it makes Sean a sad boy.
CLIENT SUB 93 (ZMP): zmp.ident<00>gnome-mud<00>0.11.2<00>A mud client written for the GNOME environment.<00>
CLIENT SUB 93 (ZMP): zmp.no-support<00>net.sourcemud.<00>
CLIENT SUB 93 (ZMP): zmp.no-support<00>color.define<00>
CLIENT SUB 24 (TTYPE): <00>gnome-mud
14 Mar, 2009, Les wrote in the 9th comment:
Votes: 0
The random IAC byte in Client Data on line 13 might be a bug on my end but I'm not convinced.

We send IAC SB ZMP <dump zmp data here> IAC SE which afaik is correct. It could be something completely unrelated but off hand I can't think of how a lone IAC can even be sent from us.

The 0 byte prepending gnome-mud confused me until I realized it was just the TTYPE_IS byte :wink:
14 Mar, 2009, elanthis wrote in the 10th comment:
Votes: 0
Aaah, I fixed it. Bug was introduced when I added the optimization for processing as much of the buffer at a time as I could. If the end of the buffer was reached it would try to push all unwritten data bytes… even if it wasn't in the DATA state. Oops. One "if (telnet->state == LIBTELNET_STATE_DATA)" later and we're good. :)

The behavior didn't pop up with BSD telnet but it did with both gnome-mud and (to a much lesser extend) clc, but only sometimes. This was in fact a very real example that the "packets can be split in the middle of TELNET commands" behavior is not at all just theoretical. :) gnome-mud seems to be pushing one byte at a time into gnet, which I suppose may be leading to a high frequency of packets being sent containing only a few bytes ouf of a larger TELNET command sequence. clc buffers things up a little more but it still ends up calling send() several times during the generation of more complex TELNET commands, e.g. ZMP.

I guess that means that gnome-mud may actually be a good client to test against "broken" servers. It certainly helped find a very embarrassing (and ironic) bug in libtelnet before I started deploying it in real MUDs. :)

Few more features to do and then a bit of a cleanup to how MCCP is detected and enabled (it works in a "transparent" mode now which is great for telnet-proxy but kind of funky for real server or client usage – not hard to make it better though) and then I'll convert clc and tbaMUD to libtelnet and see how things go.
14 Mar, 2009, David Haley wrote in the 11th comment:
Votes: 0
Very cool stuff, Elanthis. It's on my short list for cool stuff to play with very soon. Although for the life of me I can't see why you'd want to write telnet code to work on anything other than large 3-byte MSSP requests that only work in practice. :lol:
14 Mar, 2009, David Haley wrote in the 12th comment:
Votes: 0
Oh, and if you're building in things like mccp, it might be appropriate to call it libmudtelnet or something like that, since it's not (strictly speaking) generic telnet anymore? Eh, just being pedantic, ignore me.
14 Mar, 2009, elanthis wrote in the 13th comment:
Votes: 0
I really didn't want to build in MCCP2 support. But it ends up being impossible to support without really compromising the API and removing the buffer optimizations. If the client receives a single buffer (e.g. packet) with "uncompressed data" IAC SB COMPRESS2 IAC SE "zlib stream" then it can just process the whole buffer. As soon as it process the COMPRESS2 sub-negotiation it has to abort processing any remaining bytes in the buffer, decompress them, and then return to processing. So I pretty much had to build it in for the client.

Then I figured I might as well build it in for the server while I'm at it, even though there isn't an actual need for it there with the libtelnet API.

At this point I'm figuring I might as well add in the core ZMP protocol parsing to help the antiNULists (I might even go wild and add in parsing for ENVIRON, NEW-ENVIRON, TTYPE, NAWS, and MSSP while I'm at it). I'm also debating whether or not I'm going to implement RFC1143. The naive implementation requires a huge array of 256 option states for both sides, which would be pretty heavy on memory for many MUDs. More memory-efficient implementations are just a pain in the ass to write in C and I'm lazy. ;) Any suggestions on succinct, low-footprint Q-method implementation tricks in C would be welcome. (I'll probably end up doing the work to add it, but I'm not looking forward to it, since the other approach I can come up with will be rather tedious.)

The library will technically still just be a TELNET library. Nothing about COMPRESS2 (e.g MSSP) is actually specific to MUDs. The same is true of ZMP – it's just a simple remote-message protocol, and the core package has no commands specific to MUDs. MSSP is pretty much all about MUDs as it includes a lot of MUD-specific variables in its core spec, but I'll be nice and invite it to the party. :) Any built-in extensions will need to be explicitly enabled in the final API, so anybody using libtelnet but not wanting these features won't have to worry about them automatically turning on behind their back. The tbaMUD folks explicitly requested that, actually, as they want any advanced features to be runtime configurable with their cedit system. I could easily make it possible to compile them out entirely too, just like MCCP2 is (for in case you don't have zlib, mostly).
14 Mar, 2009, Scandum wrote in the 14th comment:
Votes: 0
elanthis said:
Your one way to implement a state machine is so amazingly above my head that I went and thought that it didn't even maintain any form of state and that it was just a look-ahead scanner. I'm so dumb. Please get more of your code out there so the novices can learn from your wisdom and the gurus can retire knowing you've got all the technical details firmly under control. You win, dude.

The chance of a 3 byte sequence getting fragmented with modern speeds is as good as zero, so yes, your tirade is pretty dumb. Needless to say that some generic state checking is a planned addition.
14 Mar, 2009, David Haley wrote in the 15th comment:
Votes: 0
Scandum, you do realize that the world is bigger than MSSP right? :rolleyes: (And network speed has nothing to do with packet fragmentation… but you knew that already.)

elanthis said:
So I pretty much had to build it in for the client.

Fair enough, I'm convinced. Same for adding the other features.
14 Mar, 2009, Les wrote in the 16th comment:
Votes: 0
elanthis said:
I guess that means that gnome-mud may actually be a good client to test against "broken" servers. It certainly helped find a very embarrassing (and ironic) bug in libtelnet before I started deploying it in real MUDs. :)


Well that's certainly a great 'its not a bug, its a feature!' comment :) I wonder if we should be doing what we're doing though. Maybe we should just do it the more standard way.
14 Mar, 2009, David Haley wrote in the 17th comment:
Votes: 0
What are we doing incorrectly that we should be doing a standard way? (not saying you're wrong, just got lost in the thread)
14 Mar, 2009, Les wrote in the 18th comment:
Votes: 0
'We' in this sense meant 'those of us working on gnome-mud'. Sorry for not making that clearer.
14 Mar, 2009, elanthis wrote in the 19th comment:
Votes: 0
David, I think he means buffering the output and generally only calling send() once each loop update (with exceptions for filled buffers and the like, of course). That would indeed make gnome-mud a bit safer for flakier MUDs. :) Naturally, fixing the MUDs is still important, because freak chance can still cause them to break.

I plan on adding a feature to telnet-proxy to force "small packets" for testing things with. Just send things in 1-byte chunks with TCP_NODELY set should do it, I think.
15 Mar, 2009, elanthis wrote in the 20th comment:
Votes: 0
Big updates to the libtelnet API, I think I have it close to what I want 1.0 to go with. Error handling needs to be improved a bit and I need to implement RFC 1143 and figure out what the API for that should look like (just pass in the event and hope the app responds? add a callback just for that which forces the app to respond yes or no?)

telnet-proxy got a little more convenient (it loops now, instead of exiting after a disconnection), and I also added a very simplistic telnet-client (needs some support for TTYPE, IP, and a few other necessities to make it usable for "real" TELNET purposes… but it does have MCCP2 support). Need to make both actually deal with non-fatal errors too, e.g. EINTR. Will then make a very simple telnet-chatd server for testing the server end of things. Still unsure about adding ZMP/MSSP/etc. parsing support, but I may do that. Also planning on adding a small utility library for line-buffering, which pretty much all MUD servers use and which almost all MUD clients use…
0.0/20