Splitting from this thread because it's a fairly self-contained topic.

I see your point about people assuming a fixed buffer size. Is there any value in having different limits for each direction of traffic? Your argument is basically that parsing it is hard esp. in C, and MUD authors are likely to do it wrong, take shortcuts, etc. I'll accept that. But the point I was making about server–>client traffic is that there's not much reason for that direction to be limited because the client authors tend to know what they're doing – and if not, it's a lot easier to annoy them into fixing it than trying to convince potentially hundreds of devs.

It's rather inelegant to have this asymmetry, but I am really not too comfortable with the idea of limiting the data that can be sent. 128 arguments might work, but then you still have to limit the size of the whole buffer, right? What number do you pick there?

From my experience, that is not at all true. They're just as poorly written as the servers most of the time. Maybe you've just been lucky in which clients you've looked at. :)

Even a well-written client is still going to have a buffer size limitation, too. Otherwise a server could send IAC SB followed by 2GB of data with no IAC SE and cause bad things to happen (which has nothing to do with ZMP – it could be _any_ subrequest). A good client will try to deal with this gracefully (that is, without exhausting the user's memory and causing his OS to grind to a halt under memory pressure), even if that just means disconnecting from the server after the first X bytes. The question is what that X should be… maybe for clients it makes more sense to allow several megabytes since generally a user is only connected to one (maybe two) MUDs at a time, while a server has to potentially fit hundreds of players into a few dozen megabytes of memory.

If a client wants to support a gigabyte of buffer space, that's fine. The spec won't say "the buffer size must be X." Only that it must be at least X. Tyche had once recommended negotiating the buffer size, but I don't see how that helps much – if the client doesn't support a big enough buffer, what's the server supposed to do about it? Just disable all commands that it thinks might exceed that buffer size, maybe? Better to just avoid the situation entirely, IMO.

I am not strictly opposed to requiring two different buffer sizes for clients and servers. It is far more likely that the server will be sending large data to the client than the reverse, although Scandum did make a good example of a file-editing package, which would require the client to send a whole file back to the server. It is a little icky to be asymmetrical, but maybe that is just something I'll have to deal with. :)

I just don't have a real answer to your question about what that limit should be. "8KB should be enough for anybody." (I'm not really the founder of a multi-billion dollar software company… but I did stay in a Holiday Inn Express last night!)

Maybe I had just had luck with clients :wink: Well, to be honest, I've been working with just MUSHclient for so long that I tend to forget that it's higher quality internally than many other clients…


Oh, sure. But instead of something like 512kb, you can say several dozen MB as you suggested.
I'm happy saying that if a ZMP command exceeds 64MB, shut it off.
That allows sending things like an archive of image files. If your archive is bigger than 64MB… do it in several passes, sorry. I can't think of single files bigger than 64MB that a MUD would be sending around…


Agreed, there's not much the server should do, better to just avoid.


A file-transfer protocol could be reasonably implemented in several commands to represent chunks. You could do something like…

ftp.start | <file name> | num_chunks | num_bytes
ftp.chunk | chunk_size | <binary data up to 16kb(??)>
ftp.chunk | chunk_size | …
ftp.end

where it is understood that you can't nest starts inside each other, chunks can't appear outside of start/end, etc.
(Oh, and you'd need to escape NULs… maybe an empty argument is used to represent NUL. So, 1234005 means the bytes 123405.)

So yes, this introduces communication overhead, but it gets around the problem of limiting a single command's size.
It's also very likely that moving files around is a very rare event, and almost certainly one limited to MUD admins.

I think it's reasonable to put the client–>server limit at:
- unlimited arguments
but
- 1-16kb data (not sure what the "sweet spot" is)

1kb seems sufficient for the vast majority of client–>server communication. It might be useful to brainstorm what kind of things the client might be sending back that are bigger than 1kb. The file transfer is one, but we can solve that by chunking it. I can't really think of anything else to be honest. Maybe something like client-side editing of multi-line character descriptions – but again that can be solved with chunks.

Aw, but I wanted HD-quality cut-scenes in my MUD! ::pouts::



Sure. But that's as attractive to some people as having to split up an inventory command is to you. :)



I would just require base64 encoding the file contents.

Don't forget that IAC would need to be escaped as IAC IAC as well. Then NUL could be escaped as IAC NUL. Maybe something like transfer speed could be negotiated as well, and some kind of option to resume a discontinued file transfer. A file hash should be generated as well so each chunk can be checked for integrity, a tiger tree hash might work well for that. That way it would be possible to download chunks from other people as well, not just the server, so the load would be balanced between the users.

Actually I thought about this and was wondering if I should make my preemptive strike more explicit :tongue:
Here's my argument:


The inventory thing is something that would happen all the time, and you have these issues of the client perhaps rendering things onto a window and so forth (so several updates can be a problem). Furthermore, inventory is likely to be fairly small compared to file chunks. I don't see the point of chunking a list with 10-30 elements, but when you're talking about kilobytes (or megabytes…) at a time, chunking is more relevant.

I mean, in practice, do we need to make the allowable size to be as large as the biggest file we can thinking of sending? What if you send more than that? You need to chunk :wink:


Oh yeah. Good idea :redface:

Is implementing peer-to-peer file sharing really something we need to worry about on a MUD? :thinking:
Anyhow if we're going to discuss the details of a file transfer protocol, could we please do it elsewhere – I'd like to keep this focused on just the issue of limiting argument/buffer size if possible. The FTP is just a useful example where people have large amounts of data to send, and it's relevant only to that extent.

I thought about that very response, too. ;) I agree, for what it's worth.

There's always mudFTP. http://www.andreasen.org/mudftp/ You know, because requiring a whole different port and protocol makes sense. ;)

I'm emailing Les (gnome-mud) to see what he thinks about this buffer limit stuff.

Les feels pretty strongly against very large buffer sizes in the client. His biggest concern is that even with broadband, a multi-megabyte subrequest can still take several seconds to come over the pipe. During that time, no other data can be sent from the server, making the MUD appear to "freeze" momentarily.

For what it's worth, I did a little math. An inventory list of even 50 items with item labels averaging 50 characters (which is honestly pretty long) would only use up 2.5K. 100 items (really big inventory) would use up only 5K. So an 8K limit seems pretty reasonable, and a 16K limit would give a lot of breathing room.

Sending several MBs in chunks would also cause the MUD to "freeze" unless the MUD server is clever about interleaving normal output with the chunk sending. I'm skeptical that MUD servers would be that clever, so I'm not sure chunking actually makes a difference in this respect.

But sure, let's go with 16K, does that sound ok?

BTW, where is gnome-mud being developed these days? The sourceforge page appears to be very out of date… (not to mention broken)

Yeah, definitely. It only becomes an issue for a file-transfer protocol, and only if trying to transfer large files. In general I agree with Les, who said that if a MUD thinks a client needs any kind of file data, it should just send a URL, as MSP does. Using libcurl or whatnot to grab the file is the way to go.

Only for inline editing of text data (area descriptions and scripts) does sending the full data really make sense, and those should be well under the megabyte range.



Sure. For both ends or just client? ;)

In a server that made use of fixed-size buffers, 64 players would use up 1MB of memory just for subrequest buffers. The biggest current MUD I know of averages about 500 players, so that would be about 7.8MB of buffer memory, IF they used fixed-size buffers. If they are smart and using variable-sized buffers (or step-sized buffers*) it would be much smaller, of course.

* I find it best to just use several steps of fixed buffer sizes, and "upgrade" from one to the next if one fills. The actual choice of buffer sizes is more an art than a science, I think, but something like 512b : 2K : 8K : 16K should work fine. Most common requests would not exceed 512b… hopefully. (If you find a large percentage of ZMP requests cause buffer upgrades, just increase the size of the smallest buffer accordingly.)

Clients will probably just use a fixed 16K buffer size.

Feedback from anyone besides me and David and Scandum would be great before finalizing any of this, of course. :)



http://live.gnome.org/GnomeMud

Little insider info he passed me this morning is that the plugin framework is being replaced entirely by Lua scripting. Which naturally will be hooked up with ZMP. Plus gnome-mud already has multi-window capabilities so making the subwindow stuff work will be dead easy. And using GTK+ widgets and Cairo rendering from the scripts should be quite easy, too. Awesome!

Yupyup. Well, it's really just a dynamic buffer that grows in specific chunk sizes.

Yes, I was thinking about clients. For client–>server, I'm not really sure what kind of data would be sent back. I'm tempted to think that even just 1k would be enough… would 4k or 8k be a good number?

I'm thinking that it won't matter that much because MUDs with that many players will have devs who (hopefully) know what they're doing enough to implement variable-sized buffers. So maybe we don't even need the asymmetry anymore if we go for 16kb.


Thanks for that. I'll check it out later tonight I think.


Oooh – very cool indeed!
Too bad it's GTK+ and not something cross-platform though. Ah well.
(Yes, I know GTK+ is kindasorta cross-platform, but it's hard to set up…)

Anything that makes it hard to set up is usually the fault of the application shipping GTK+. It should Just Work™ if the application is packaged/installed sanely.

If you do have specific issues with GTK+ on Windows, I highly recommend getting in contact with Tor Lillqvist, as he is quite responsive and friendly and eager to make GTK+ rock.

On a side note, GTK+ 2.16.0 was just released today. Woo!

This is true. If it's packaged properly you as a user shouldn't notice anything really.

That being said, Gnome-Mud isn't Gtk-Mud so we do have some gnome dependencies, namely GConf. So if you were to try to go through the effort (not recommended!) of building gnome-mud for windows today it would be fairly hellish. It's not impossible by any means and making it more easily cross platform is something I'm working on (truly :-) ). There's only so many hours in the day though and it's hardly as if gnome-mud is even close to feature complete.

More on topic, I am not against having a dynamic sub request buffer in the slightest. I just worry about problems that could happen with buggy or, less likely, malicious servers. The only reason gnome-mud's is 2k is that I really didn't know what I should put there. At first it was 256 bytes! ;-)

Out of curiosity, what would happen if a malicious server sent a line? Don't most clients run their triggers only after the line was ended, and so presumably are buffering things until that line ending?

This would be consistent with what I've seen. I can only really speak with any authority about gnome-mud's implementation but don't forget that data is sent as packets. In gnome-mud's case if we are in a state that isn't a telnet protocol state (just regular old text) at the end of the packet then we return what we have and the triggers and everything else gets their whack at it.

So there wouldn't be any unresponsiveness you would just see massive spam scrolling in the terminal. The problem with the malicious subneg string by contrast is that because the parser IS in a special telnet protocol state (handling the subneg) it will just sit there and churn and leave the user clueless.

Ah – so you don't actually process triggers on line-end, you process on packet end?
My understanding was that many clients process triggers on a per-line basis, meaning that they would be vulnerable to a malicious server sending a huge amount of normal text and perhaps causing the process to run out of memory (unless it guards against a maximum line length or something).

The subneg hanging could be handled at the UI level though; maybe you could have something that pops up a message or something if it stays in the subneg state for >3s or something.

For better or worse, yes. :) There's some contortions for the triggers' sake for lines broken across packets but that's what we do.



Sure you could do this but there isn't a solution other than disconnecting from the mud since there is no telnet protocol way to terminate a subnegotiation. So since the only options would be wait and see or disconnect I would vote for just defaulting to disconnect and informing the user their server is b0rked ;)

I think the 16K buffer thing solves the subnegotiation hang-up fairly well. :)

libtelnet just leaves it up to the app to decide what to do if a buffer is filled. When the error happens, the buffer is cleared and the state is put back in the normal (TEXT) state, and the app is notified of the error. It can either disconnect or just keep running – at worst then the player is either seeing some chunks of a freak oversize ZMP/MSSP command or he missed out on some MUD text from an untermined subnegotiation. I personally would disconnect. Or have the client retaliate by sending a huge TELNET subnegotiation command and seeing how the server likes it. :p

I qualify my comment in the other thread here. MUSHclient *does* impose a limitation of around 100 characters on telnet subnegotiations, on the grounds that if you haven't turned on or off an option in 100 characters something is seriously wrong.

However we (that is Lasher and myself) did not intend telnet negotiations to be used for the bulk exchange of data between client and server.