Structure of a networking framework based on nio

Networking & Multiplayer
#1

Hi

As the subject suggest Im looking at designing some reusable network code based on nio, and thought I would seek some input here on how to structure it.

My current plan is to have:

  • A selector thread listening for new connections

  • A simple load balancing system delegating the new connections to worker threads and adjusting the number of those threads.

  • 1-n number of worker threads each having their own selector which they register the connections on

The plan is to always have the socketchannels registered for OP_READ and then register them for OP_WRITE whenever there is data or some other event that the worker threads will then query the ChannelSockets “manager object” for.

On of the big problems with this structure is of course if the data processing and handling of the incomming data takes a lot of time, but I plan on making decoupling heavy / time consuming tasks the responsibility of the application code.

So, is this structure totally messed up? :slight_smile: Suggestions and comments on this and other possibel structures are very welcome.

#2

[quote]Hi
My current plan is to have:

  • A selector thread listening for new connections

  • A simple load balancing system delegating the new connections to worker threads and adjusting the number of those threads.

  • 1-n number of worker threads each having their own selector which they register the connections on
    [/quote]
    Depending upon how you interpret this sentence, this could be undesirable. In general you want the worker threads to share a Selector - Selectors are for multiplexing lots of stuff together, you might as well make use of that! People often ask “is there a performance drop in using many selectors?” and I’ve never seen strong evidence one way or the other, but there may be; IME with Sun classes, any “unusual” usage tends to be broken or low performance until it’s been around a long time. I’d suggest you benchmark and find out if you are intending to use many selectors.

Personally, I enforced the decoupling of every task in my basic framework. For details, c.f. “Thousands of Clients per Server” in GameProgrammingGems4 (sorry, I can’t really quote it in a public forum or I’m bound to get in lots of trouble :)).

Many of the links from

http://grexengine.com/sections/people/adam/gpg4/index.html

also contain details relevant to what you are attempting to do; in terms of getting high performance and preventing worst-case scenarios and even making the code easier to maintain.

#3

My first question though would be

Why bother?

The things you are suggesting (thread pooling, load-balancing) are pretty hardcore, and are you sure it’s worth your effort?

Obviously, if you are working for a company that is trying to make a generic networking layer for games, or if you intend to release many games in a row all with thousands of simultaneously connected clients, then it’s a good idea.

Otherwise, I’d suggest you just go for a simple NIO framework. Just have either one selector for EVERYTHING, and just one thread, or three selectors (one for each of accept,read, write) and three threads (one for each selector). This will be about as fast as you’re going to get anyway, so performance isn’t something to be worrying about.

One of the nice things about NIO is that is already is a networking framework - which is part of why it’s so much better than the old IO stuff in Java. It forces you into good application design. The links I referred to above are all about making hardcore servers - some of them IIRC are designed for hundreds of thousands of clients on one physical server; for this kind of thing, you need to add some extra controls to NIO. For almost anything else, it’s a waste of your time (…because your time could be better spent improving your 3D engine, etc).

#4

Ill try to explain my needs and goals a bit better. I intend to use the code/framework to replace my old-io network code in a bunch of existing projects (including some chat server applications that runs with loads of 1000-1500 users). Its defintly not high load applications but still enough that I have to think at bit about the performance.

The main reason that I have considered this approch is that I find the structure with a read and write selector somewhat messy and that it introduces some trouble with doing a gracefull shutdown of a client.

A single selector and a bunch of worker threads on a wait/notify is perhaps a better and simpler solution but it just involves a lot of queuing events to and from the selector thread.

Im defintly a noob on this area, so Im really just trying to find some best practice patterns for a semi-scaleable framework.

#5

[quote]Ill try to explain my needs and goals a bit better.
[/quote]
OK, that makes things a bit clearer, thanks :).

Out of interest, what do you find “messy” about it? Also, what problems have you had with shutdown? I’ve had a lot of pain on linux (IIRC ?only? on linux; thanks sun! :P) with not being correctly informed of certain channel-death/disconnect/hangup events (c.f. the thread on this board somewhere entitled something like “SocketChannels die; I’m not invited to the funeral”).

Again, forgive me for any assumptions I’m about to make :). Just to be clear:

[]You shouldn’t need to use wait/notify with NIO unless you really want to; I’m guessing you’re thinking of using this to implement a multi-threaded queue (given your later comments), but you can do it without if you don’t like them (which I don’t :))
[]Are you sure you need worker threads? Some bitter advice here: I’ve tried pooling multiple threads on single selectors before, and ran into what appeared to be platform-specific (linux) bugs in sun’s VM; YMMV, but this appears to be one of the parts sun hasn’t really tested :(. (nb: there are several such; e.g. I’ve got an outstanding JDC bug on the fact that gathering writes don’t work properly with more than about 10 buffers :))
[]Are you sure you want worker threads? Since dumping the pooling, I’ve usually found performance more than adequate for hundreds of clients with just a single thread per selector. Since I usually do message-parsing in that thread, and in my own projects I haven’t ever anywhere done any optimization of parsing etc, this suggests the performance is good. I’d need to check what the grexengine does here, but I know some servers have been stress tested without any worker-thread balancing and were fine (nb: IIRC there was significant room for theoretical improvement on the performance, but it met the “sufficient” threshold that we’d planned in advance).
[] Is there anything wrong with queuing events? What did you have in mind for this (i.e. you seem to dislike it, but I can’t think what’s wrong with that, so I’m wondering what you’re thinking of that is different to what I’m thinking of :))

#6

Hi again. Sorry for the lack of replies, but ive been so busy with work that I havnt had time to play with java =)

[quote]Is there anything wrong with queuing events? What did you have in mind for this (i.e. you seem to dislike it, but I can’t think what’s wrong with that, so I’m wondering what you’re thinking of that is different to what I’m thinking of
[/quote]
The reason Ive been a bit worried about queuing events/objects is that I was affraid it would introduce to much object creation in the basic operation of the netcode. After Ive had the chance to create a nio server it seems it was an unneeded concern, since nio does do a lot of the work as you said.

Another thing Im a bit concerned about is what the performance cost is for constantly changing the events you are registered for (and perform a wakeup to make the selector update it) plus the good description in SelectionKey.interestOps(int:ops)

[quote]This method may be invoked at any time. Whether or not it blocks, and for how long, is implementation-dependent.
[/quote]
Say we have a setup where one object contains the selector and has its own thread who perform select operations. The selected keys are then handed over to a pool from where 1-n workers pick them up to process.

Correct me if im wrong here, but when the selectionkey is put in the pool I would have to cancel the registered OP_READ and then reregister it after processing has been completed, to prevent that multiple threads get a reference to the selectionkey (by cancling i mean changing interestOps not canceling selectionKey).

Its not a problem to implement Im just a little annoyed with working with may/may not work documented features =)

[quote]Out of interest, what do you find “messy” about it? Also, what problems have you had with shutdown?
[/quote]
My problem with having seperate read/write selectors is mostly that I then spread out the code that can “trigger” the need to disconnect a client, so it takes extra code to handle those different scenarios. I cant really see advantages of splitting it up (which is probabbly because im new to this area =) so Im just going for the most simple design.

[quote]You shouldn’t need to use wait/notify with NIO unless you really want to; I’m guessing you’re thinking of using this to implement a multi-threaded queue (given your later comments), but you can do it without if you don’t like them (which I don’t )
[/quote]
Thats what Im going for in my rewrite yes, with multiple worker threads picking the up the SelectionKey’s from a queue. Im not sure how you’ll do without wait/notify to handle that transfer though ???

[quote]Are you sure you want worker threads? Since dumping the pooling, I’ve usually found performance more than adequate for hundreds of clients with just a single thread per selector
[/quote]
Nope Im not sure if its needed, but as I said there can be a fair load of users at the chatsite where this code if first intended to be put in use (where some user commands result in multiple-second sql queries so decoupling from any bottleneck in the system is needed, and it seems a better and better idea to do it in the networkcode =) My current plan is to try a rewrite with the structure describe in this reply, and then try out how it works.

#7

Nothing to say here, except that you probably need to re-learn the role of objects in java performance. Read through the performance categories on this forum…

Personally, I think you’re coupling at completely the wrong point. It sounds like (nb: but I don’t know enough details of your project…) you haven’t really thought about where the bottlenecks are, since the only reason you’ve given for wanting to pool threads is a very late stage operation (SQL queries). That only suggests you wanted to pool and cache your DB access, and that you need to find someone who knows how to write fast SQL. It sounds like you’re attempting to optimize completely the wrong code.

…so you need to explain WHY you think this is anything to do with the network code? The mantra for optimization is always “code first, optimize later”, unless you already know where and why the bottlenecks will be. You haven’t cited any existing problems you’re aware of or have come across with the network layer code, so…?

Gross generalization :wink: : Any user-triggered SQL query that takes more than 0.1 seconds (assuming MySQL) is usually a sign that you need to optimize your SQL source, or that you shouldn’t be using SQL in the first place. Admin queries are different, since they are more often used to generate reports ranging across all the data; user queries nearly always only need to work on small subsets (or on summary information). Obviously there’s many exceptions to this :).

#8

Just a quick answer as Im on my way out the door. The multiple second query is already optimized (not by me =) but is simply dealing with a lot of data and table relations. The majority of the sql calls has very reasonable execution time, but what I tried to say was simply that since the application code will perform actions involving io then there would need to be a decopling in the application layer if the netcode just ran with one thread. My feeling right now is that its easier to simply place that decoupling at the selectionkey processing.

Anyway I didnt start this thread to discuss specific performance issues, I was just interested in hearing if there was some well known, good and elegant ways to structure nio code and work around the different bugs which I find is a bit of a problem keeping track of. Ie how other people have handeled the different mechanisms needed in a framework.