Login and lobby server network architecture

Networking & Multiplayer
#1

A friend and I are creating a multiplayer lobby. The deployment will consist of a single login server sitting out in front, handling all authentications, and then arbitrarily many lobby servers to manage different lobbies. The idea is that any given user can go to any lobby, like the North America West lobby, the Europe lobby, that kind of thing. The lobby servers are not located on the same LAN as the login server.

All queries against the login server are synchronous, so we just use a simple request/response protocol, with no persistent connections. Session state is managed in a way similar to how HTTP does it.

We are about to start on the lobby servers and are not sure how best to handle connections and authentication. The communication here is partly synchonous and partly asynchonous. For example, a user may request to host a game, so that would be a synchronous request. Or the lobby may indicate to the user that somebody just sent him a chat message, or that a new game is being hosted, or whatver. Asynchonous there.

We are thinking that for any given user, there should be a persistent connection to his current lobby for handling asycn communications. Synchronous requests would be sent to the lobby server in the same manner they’re sent to the login server. We would want all such requests to be authenticated.

  1. Does the above seem reasonable?

  2. Should different types of async communications be handled on different ports? (E.g. should incoming chats be handled on a different port than new game notifications) What are the common ports (and are which are commonly TCP-based vs. UDP-based)

  3. For authenticating lobby server requests, we were thinking that on entering a lobby, the lobby would request the auth info from the login server one time and cache it so as to avoid making the lobby server query the login server with every request. Is this reasonable?

Willie

#2

Don’t bother with authentication until you understand exactly why you need it; given the questions you’re asking, it sounds as though you may only have a fairly vague idea that you “want security” (apologies if I’m guessing awry…I’m trying to read between the lines here: if you knew what you were doing with auth, you would have very probably provided a different set of questions).

[] Your login/lobby server design is terrible in terms of scaling to large numbers of players. Given you expect to have multiple lobby servers, that suggests sufficiently large numbers of players that you are doing entirely the wrong thing (hint: Anarchy Online’s biggest single mistake was using exactly the same design you’ve suggested. It doesn’t work. I’ve even seen it fail with just THREE lobby servers!). A usefully-detailed explanation of better ways is not trivial, and I’m afraid I don’t know any good resources off the top of my head, although I suggest you do some googling on “lobby server architectures”. There should be some good meaty post-mortems on XBox live games that should give you some ideas…
[] Don’t do bits “similar to HTTP”, do it “using HTTP”. Unless you have a particularly good reason not to. An HTTP/1.0 server is extremely easy to write…and you get the benefit of an almost infinite array of test tools!
[] Don’t worry so much about your choice of ports and who’s listening where for what; it’s a common thing to worry about because you know it’s a pain to change in the future, but it doesn’t tend to be a bottleneck, so you almost never care whether you did it “the best way”. Tthere’s plenty of different ways you could do it, and they don’t really matter that much how you choose (unless you want to be really diligent, in which case it depends more upon how the rest of your server and game is designed, and what is easiest to implement using them)
[] Common ports? Who cares… Until your game becomes a mega-hit, it won’t matter what you choose. Hackers will scan every port anyway, and firewalls will block your traffic based upon contents, not port number. All that really matters is that you never want to listen on ports below 1024, because that requires root access on unix + linux, which is a PITA to setup without creating major security problems (linux in particular being an inherently unsafe OS [although WinXP etc aren’t much better]).
[] It is entirely pointless to ask questions about “how should we do our authentication” until you have supplied the following:
[] [list]
[] a list of things you need to guarantee (eg the name of the player, or their IP address, or…)
[] an explanation of why for each
[] a list of things you think attackers will do to try and compromise that data
[] an explanation of why they’d want to for each

[/list]

#3

Hi blahblahblah,

Thanks for the thoughtful response. Let me answer some of the questions you raise and see if you might point me in the right direction.

As far as security goes, part of what I’m trying to accomplish is a single sign-on environment with respect to the various lobby servers. Players should be able to move from lobby to lobby without having to reauthenticate.

It is true that the login server would be a bottleneck in the system, but only the initial login would really use the login server. After that, control would be handed off to one of the lobby servers. I believe that is a fairly standard configuration, at least in IT settings, because it actually supports scalability (you can always add new lobby servers) even if it introduces certain risks; viz., single point of failure in the login server, and login server scalability. The single point of failure risk is mitigated in part by making the login server as simple as possible, and the scalability risk is mitigated in part by giving the login server very limited responsibilities (just auth the user and then handoff to a lobby server).

As to why I want authentication at all, I want to be able to audit users and I want to be able to place access controls on what users are able to do. For example, it would be nice for the server to support remote admin, such as booting users, deleting users, whatever, and obviously we wouldn’t want such features to be readily accessible to normal users. And we’re not excited about the “security by obscurity” approach of simply assuming that nobody will figure out how to access such features because they’re not widely known. They may indeed become more widely known as more developers become involved with the project.

Concerning using HTTP–I agree with you on that, for exactly the reasons you give. (For example, would be nice to use Tomcat as the login server, write a servlet, and handle login server scalability with mechanisms already supported by Tomcat should the need ever arise.) I intend to bring that up with the other developer on the project at some point.

On the question of ports, I think my question was not as clear as it could have been. I don’t care about the actual port numbers, whether I’m using standard ports, or anything like that. What I care about is the extent to which a given port should be overloaded with lots of different kinds of traffic. For example, should chat traffic have a dedicated port, or should it share the port with other kinds of traffic (e.g. downloading the Terms of Use). What are the general principles in deciding how to assign traffic to ports? E.g., maybe it is true that the number of ports should be minimized because it makes firewall configuration simpler.

Thanks again for your response!

Willie