Network Game Engine

Networking & Multiplayer
#1

[EDIT: latest source and demo is here http://www.java-gaming.org/index.php/topic,18019.0.html, in the topic ‘Games Showcase’ under title ‘Multiplayer top-down view shooter’.]

Hi Everyone,

I’ve been working on this server-client network game framework for ages and I’ve decided that it’s at a stage where I’d like you to take a look and let me know what you think.

Of course I want to make a MMOG like everyone else :), and eventually I want this framework to be that good, but in the meantime I think that it could do LAN and maybe small internet games pretty well. At the very least it should demonstrate some different ways of dealing with latency (lag) between computers.

I’ve made 2 webstart demos with GUI options 8) that you can adjust to test it out. Once you set up a server game, you should be able to set up a client game and join in whether the client game is on the same computer or on another machine connected on the internet or LAN: http://www.freewebs.com/commanderkeith/sydneyengine/SydneyEngineDemo.jnlp
I’ve added tool-tips to the labels so they should provide some info on what you’re adjusting. Remeber to click the update button after adjusting stuff. The keys to move are left, right, up & down as well as page up and down to zoom.

I haven’t made a proper game with it yet, but I did convert Kev Glass’s great Phys2D API :smiley: to use the framework and made a little 2D top-down view demo where you’re a ship in space and you can crash into other players if they join your game. There are 2 things about the Phys2D API that mean that my framework doesn’t work well with it, but both are solvable[1]. Here’s the webstart: http://www.freewebs.com/commanderkeith/sydneyengine/physics/SydneyEngineDemoPhysics.jnlp

http://www.freewebs.com/commanderkeith/sydneyengine/screenShot.jpg

There are 3 parts to the framework, which for now I’ve called the sydney engine:

  1. The network package which has interfaces and classes that are just meant to send byte arrays. Right now it uses super-stripped down NIO TCP communication.

  2. The ‘SuperSerializable’ streams which just serialize (turn objects into bytes) and deserialize the game objects that are sent between clients and server. They’re as fast as regular java.io.ObjectIn/OutputStreams but they update existing objects instead of replacing them, which doesn’t cause garbage-collector slow-downs.

  3. The engine package, which has interfaces and classes that manage all of the threads, messages, and the game world updates. The engine has a couple of threads:

  • A thread that listens for incoming connections (ConnectionListener) and spawns other threads (ConnectionWelcomer) to join the client to the server game.
  • A thread that does the game loop (ServerController or ClientController), timing the game world updates and rendering.
  • A thread that queues messages to be sent to the server from the client or vice versa.
  • For the ServerController, a thread that waits for incoming messages from clients so that they can be relayed to other clients straight away.

The framework’s aim is to efficiently join and remove players into the game, send player events to all other players and also to synchronise game worlds between all machines by sending the server’s game world to all clients. This sounds easy, but it was hard since you need to manage threads without too much synchronization block code, and you must guarantee that messages do not go missing, double-up or get out of order.

But the other job is in the client code which needs to hide the latency between machines by appearing smooth. There are 2 methods that I’ve been working on to mitigate network latency/lag, and I’ll just cut and paste the explanation from the lousy javadoc I’ve been working on :P. Note that UserEvents are just key presses and other client events.

// For UserEvent.eventTimingStrategy == ORIGINAL_TIME_SYNC,
// userEvent.getTimeStamp() returns the original gameWorld time that the UserEvent occurred on the original machine. This strategy will mean that everything happens at the same time on each computer (as soon as the computer knows about it), at the expense of smoothness. The clients will appear to ‘jump’ or ‘teleport’ around. Smoothness can be improved by setting a large delayBeforeEventAppliedSeconds for each player, however this decreases responsiveness for those players with large delayBeforeEventAppliedSeconds. The physics demo uses this approach.

// For eventTimingStrategy == ORIGINAL_TIME_UNSYNC,
// userEvent.getTimeStamp() returns the original gameWorld time that the UserEvent occured on the original machine, plus a certain fixed delay which can be different for each player. This strategy ensures that there is the same time between UserEvents (from the same player) on all VM’s whether client or server. But there will be ‘alternate realities’ since the client and server see things happening at different times. For example, in a racing game if the client and server both accelerate at the same time (and at the same rate), then each will think he is in winning when in actual fact they are tying. The other simple demo uses this technique, but I’ve made it so that it can switch between eventTimingStrategies.

Apart from these, I have heard of averaging techniques where, for example in the racing scenario above, a lagged player’s car will accelerate on other computers more than is normal so that the lagged player will catch up to where it would be if there was no lag. I haven’t implemented this in any demo yet.

I’m going to put the source up real soon to get some feedback for improvements, I just need to clean it up a bit. The current (sparse) javadoc is here: http://www.freewebs.com/commanderkeith/sydneyengine/javadoc.zip

Many thanks to the people on this forum for their help and encouragement with things along the way :). Thanks to Kev Glass and Endolf for their network library code and Kev’s Phys2D API, Adam Martin for the ClassLocator code, and Jeff Kesselman and Thijs for the pointers to use SNTP for finding clock differences. 8)

Keith

[1] You’ll notice that the physics game crashes after more than 15 minutes since too many objects are made in the physics calculations which max’s out the unique int hashCode that every object is given. Also, if the lag is set too high then client smoothness deteriorates since the physics world does not perfectly reverse positive time updates with negative time updates.

An important limitation: can’t have more than 15 clients join the game world. That is, after there have been 15 join requests, the game could go berserk since there are only 2^4 ‘VM numbers’ numbers that are set aside.

#2

Why are you making all those new threads for the clients. The whole point of NIO was to do all your IO in one thread.

No synchronization at all.

#3

Sorry, I should have made clear that most of those threads are used only by the server, not the client.

The client actually just has 3 threads at the moment - the AWT event thread (EDT), the game loop thread and another for queuing byte array sends (MessageSender), which is optional. Without it, Nexus.send(byte[]) will send the bytes straight away and the code will wait for that to be finished. Because the Nexus.send(byte[]) method can be called from the EDT or the game loop thread, it’s synchronized to be thread-safe. With the (MessageSender) thread, it’s still synchronized when the bytes are added to the MessageSender thread’s queue (to guarantee order), but the actual byte sending is not within the synchronized block which prevents the EDT from getting jammed up when the game loop thread sends a big byte array.

Also, I actually sub-classed the MessageSender thread to be a MessageSenderLagSimulator which delays sending the message for a certain lag so that lag can be simulated when there actually isn’t any. You can adjust the lag in the GUI.

Thanks for your interest Riven, I’d really like you to look at the code when I clean it up.

Keith

EDIT: For the server, I use the MessageReciever thread instead of the game loop thread to recieve messages so that they can be relayed instantly instead of waiting for the next frame to be shown in the server’s game loop thread before messages from one client (or the server) are relayed to other clients. But on the client machines, messages are recieved in the game loop thread (ClientController).

#4

Interesting…the networking portion seems pretty tiny.

I know at one time you were looking at JGN, right? What was it that made you go off and create your own? Just curious if there was something lacking in JGN or something.

#5

JGN does more than what I needed (I only needed to send byte arrays over TCP), but I since I’d like to implement NIO UDP communication eventually I’d like to use JGN at some point 8). Thanks for your interest Matt. By the way, do the demos work?

#6

Here’s the source for the engine and the simple demo:
http://www.freewebs.com/commanderkeith/sydneyengine/sydneyEngineSource.zip

Also, the (slightly) improved javadoc:
http://www.freewebs.com/commanderkeith/sydneyengine/javadoc.zip

Any comments appreciated, especially criticism on the design/architecture and how it could be better.

Meanwhile, I’m going to try to make a game out of it! 8)

Keith

PS: To anyone for whom WebStart does not work, you can download the simple demo jar file here (190kb): http://www.freewebs.com/commanderkeith/sydneyengine/SydneyEngineMini.jar

#7

Some highlevel pointers would be greatly appreciated. Javadoc only really works well when you have a genral feel for the API, not to figure out how everything interacts.

#8

Yes you’re right, I’ll try to paint a big picture. I’ll write up something and post it up tomorrow, work’s taking all my time right now :’(

I’ll be interested to see what you think Riven :slight_smile: You’re pretty cluey with these kinds of things I know! 8)

Keith

#9

Some smiles to get you through this horrendous block of text:
:slight_smile: :wink: :smiley: ;D :o 8) ??? ::slight_smile: :stuck_out_tongue: :slight_smile: :wink: :smiley:

Here are some important objects in the framework:
Controller
GameWorld
Player
Nexus

Controller is a Runnable and its job is to run the game loop which mainly involves

  • showing the game world on the screen and
  • updating the game world with the time elapsed since the last update.
    The ServerController and ClientController are the 2 main types of Controller and in any game one computer must create a ServerController and all other computers can join in by setting up ClientControllers. The other Controller is a DedicatedServerController and it has no players of its own so it’s very fast since it doesn’t need to render any game world.

Each Controller has a GameWorld which it updates using GameWorld.updateNanos(long timeElapsedSinceLastUpdate).

Players are stored in the GameWorld and if the player is controlled by the current computer, that computer’s Controller will have a reference to its Player (accessed using PlayingController.getPlayer()). So each computer has one Controller which has one GameWorld which has several Players, and if the computer operates one of those players, the Controller will have a reference to that player.

Each Player has a Nexus and the main job of a Nexus is to hold all of the messy network-code so that the Player objects can hold only game logic code. This way a Player object can have the same code even if it is on the server or the client. Nexus’s have basic methods such as send(int messageType, Object message) and receive() which are used to communicate player events (such as key presses, wanting to exit, etc).

There are 4 types of Nexus’s and I’ll explain why with an example. In a 3-player game there is S1, C2 and C3 who are players on different computers. The computer used by S1 hosts the ServerController, and the computers of C1 and C2 joined in the game by creating ClientControllers.

On the computer of player S1 (who will have a ServerController), player S1’s Nexus will be a ServerNexusPlayer and when it does send(int, Object), the message is merely put on a queue until it is removed when receive() is called. The only complicating factor is that if the message will affect the GameWorld, then it should be sent to all other computers so that they can see what player S1 has done.

This is taken care of by the ServerController’s Reciever which is a Runnable that just constantly calls all player’s receive method to find if any messages have arrived. Say player S1 pressed the forward key, then the Reciever gives the message to its RelayerHandler using the method RelayerHandler.handleMessage. The developer can sub-class the RelayerHandler to do what he wants but the default RelayerHandlerImpl generally just relays all messages to all other players. So how is the forward key, pressed by player S1, actually sent to the S1 player on the computer that controls C2 (and C3)?

Well unlike player S1’s nexus, the nexus of player C2 is a NexusServerShell. Instead of its receive method returning whatever was passed to the send method like NexusServerPlayer, NexusServerShell’s receive() actually returns what was sent from the Nexus of the C2 player on C2’s computer. Player C2’s nexus on her computer (which runs the ClientController) is actually a NexusClientPlayer by the way :slight_smile: .

On C2’s computer, when player C2’s ClientNexusPlayer receives the relayed message, it gives the message to the player S1’s nexus which is a ClientNexusShell so it can start moving forward (or whatever!).

All of this plumbing is taken care of inside the Nexus’s and the ServerController’s RelayerHandler so this messy network code is hidden from the game’s logic. The developer need not mess about with Nexus code in general since the AbstractNexuses provide all of the functionality I mentioned. The processMessages method of each of the 4 nexuses must be implemented however since that method does a bit of game-specific logic – it decides which messages are given to the Player and which are dealt with otherwise. The two demo packages provide a guide.

Sorry for the long explanation, I’ve spent 2 afternoons proof-reading it and I really did try to be succinct. Does it help much at all?

If anyone is interested, next I will try to explain how the SuperSerializable streams work. They’re quite separate to the framework and all that they do really is just turn the game world into and out of byte arrays.

What do people think of having a general network engine for real-time games? Is it helpful or is every network game unique and a framework is too limiting?

Keith

PS: and here’s a picture that might indicate why I’m not an artist:

http://www.freewebs.com/commanderkeith/sydneyengine/diagram.jpg

#10

If a design is too complex to explain cleary, it’s cleary too complex.

Ahem :slight_smile:

Without a doubt you have put great effort in it, but you are raising the bar a little bit high.

The problem is there is a Nexus, NexusClientPlayer, NexusServerPlayer, NexusClientShell, NexusServerShell, ServerController, ClientController, DedicatedServerController, PlayingController. You never explained what a {xyz}Shell is supposed to do.

What’s the point of making the ServerController poking itself with messages? Why don’t you let S1 spawn a DedicatedServerController and it’s own ClientController? Then remove ServerController and rename DedicatedServerController to ServerController.

Anyway, I always build my APIs in (abstraction) layers. If the deepest layer has game-related stuff in it, it’s a sign of moving in the wrong direction. You can however design the deepest layer with certain features in mind, so that the next layer can take advantage of that. For example the first layer can only send byte[]-messages, the next handles some protocol, after that you can build POJO-handling (like SuperSerializable?) and after that it would be the task of the game-programmer. The deepest layer never knows about the higher levels*, IMHO that’s where the network-API ends. Once you embed game-logic/design in the API nobody will use it, because their engine handles that differently. Let’s say I already have a game and I want to add multiplayer-support, should I adapt my engine to use GameWorld or Player objects from a network-API? :o
It’s great when you have an API with a lot of features, but you won’t find much people that are willing to code a game around a network-API.

About the naming, from what I understand, a nexus has to do with: focuspoint / center of gravity (abstract, no physical) / interlinked group with things in common. That doesn’t really sound like a server/client, to me it feels like P2P.

The attached image is confusing. Why are there 3 dots for both C2 and C3, can they communicate directly? Can they view their state? Or do they just know they exist?

I’ll dive in the sourcecode a bit later, maybe that will explain it some more. Your explaination was extremely hard to follow (or is it just me) so I can’t really give any reasonable feedback on the innerworkings.

* just like TCP/IP packets, where the Ethernet-packet simply has a header and unknown data. The IP-packet is inside the Ethernet-packet, and the TCP-packet is inside the IP-packet.

#11

Thanks for your feedback Riven, I’m really grateful.

Nice quote, yeah you’re right. On the one hand the problems with latency and network games are complicated :-\ but on the other hand a framework is meant to take care of it. I think my framework does hide the detail, I just tried to explain (a small part!) of how it works behind the scenes and obviously I didn’t do a good job :stuck_out_tongue: .

Well you could do that, but then you’d be pointlessly serializing messages and sending byte arrays around on the one computer. But yes, I could do that and this would eliminate a lot of the confusion about the different types of Nexus’s. ServerNexusPlayers would not be necessary - the other 3 types of nexus’s would still be there though.

Sorry, the names of Nexuses are like this: Nexus[SeverOrClient][PlayerOrShell]. The middle part tells you if the nexus is on the server or client. The last part tells you if the Controller is on the computer that has the human who actually operates the player. So the NexusServerShells and NexusClientShells represent players that are not ‘played’ by the human on this computer, they are just empty shells which listen for relayed events from the actual human player which controls them on another computer.

Yeah my apologies, that picture isn’t very good. Each computer and its Controller has a GameWorld which has a Player object for every actual player. They can see each other etc, but since the human on the computer operates only one Player, only that player and its nexus will generate events, the other players/nexus’s (the shells) will listen for the events which happen on the computer that has the human which controls them. Note that all players on all computers have different nexuses, but the same player will have a different nexus depending on which computer it is on.
About communication, since you only want one network connection between each client and the server, that connection is between the NexusClientPlayer on the client and the NexusServerShell on the server. So to send an event or some info to all clients from the server about a certain player (for example a relayed event), that message will go from every player’s NexusServerShell (on the server) to every NexusClientPlayer (on the client). When it receives the message (which is meant to be given to another nexus), the NexusClientPlayer will enque it on the correct NexusClientShell using the method NexusClientShell.insertRelayedMessageFromNexusClientPlayer.

Well I suppose all I can say is that it’s a framework and not an API :). I’ve gone to great lengths not to embed game logic in the code and where I have, that class is named SomethingImpl or AbstractSomething and it implements an interface which can be extended by the developer’s own class. All fields have ‘protected’ access so that the class extension of the SomethingImpl should work too. As to forcing the developer to a certain design, I don’t think that this is escapable as far as you must separate the game world and logic from all the other code like the view, etc. Yes you have to implement GamePlayer and GameWorld but what’s the alternative?

Wow thanks. Well actually your questions are really good and I think I’m explaining myself better when I can address the specific questions that you have. Thanks heaps for such a good run-down of what doesn’t seem right with the framework.

#12

Maybe I should list the problems that you get in a network game and how the framework tries to over-come them:

Letting all computers know when an event happens on one computer. All events should be sent through the player’s Nexus (nexus.send()) and if the player is a client, the nexus will automatically send the event to the server. The server has a thread that continuously waits for messages to be recieved from anyone (including its own player), and when it does recieve one, it relays it by sending it to all other computers straight away. Since this is done in its own thread, events get to all players in the shortest posssible time so there’s no waiting for the next frame to be displayed etc if this were to be done in the game loop thread.

Keeping the client game worlds in sync with the server’s. The SuperSerializable streams let you turn the server’s whole game world into a byte array and this can be read by the clients so that their game world is updated to be exactly like the server’s. The problem with just sending the whole game world down the pipe is that it replaces everything in the client’s game world, so any events that happened on the client and were updated into the game world could be lost if they haven’t been recieved on the server yet. So the client’s nexus does some book-keeping to keep track of which events have gone missing and re-inserts them into an ‘unprocessed events’ list if necessary.

Having to make separate code for the Player of it is on the server, since that player does not communicate over the network using byte arrays. This is taken care of because all of the communication code is in the Nexuses.

Knowing the clock difference between the client’s system clock and the server’s so that time stamps have some meaning. Also knowing the latency between computers. This is done by the LatencyCalculator and stored in every Player’s Nexus in a LatencyInfo object.

Being able to test how your game goes if there is latency. The SenderLagSimulator can simulate lag by delaying byte array sends for however long.

Finding a simple NIO network API that works :slight_smile: The network package (which I changed slightly from the one by Kev and Endolf) is bare-bones and reliable (but it’s only TCP for now).

#13

Yeah, a note about that ‘core’…

for a heavy-duty server it’s really really bad (really). Every message takes at least two (!) NIO reads… that’s horrific. You should really handle that better, because those reads are very expensive. Read into reasonably sized (64k?) dedicated buffers. Then grab your data from it, with some slightly intelligent code. Steer clear from ByteScattering and ByteGathering operations, because that’s about 10x slower than reading into a single buffer.

Further you’re not at all making use of java.nio.Selectors. Polling the sockets over and over again is really preventing any scalabilty.

I don’t understand your blocked-writing on non-blocking channels.

Any blocking on a channel at a server, will either result in 1 thread per connection, or all connections are lagging at the speed of the slowest connection.

If you’re looking for a core-network API (that doesn’t force you into a certain design) I might have some nice framework for you that I use in all my projects.

#14

Now we’re talking!!! :smiley: Well I’m not an expert with NIO, all I know is that most frameworks don’t actually work and often when you try to send big byte arrays they get muddled since they don’t read the whole thing, that’s why that read method of the network API blocks…

About Selectors, I just didn’t see how they were necessary but if you’ve got something working with them, great!

I’d love to try your network package, let me know if it’s available :).

Also, in the current framework there’s only one Sender thread, but if the server was big enough you could easily have a couple of them… but first I need to make a game that is so popular that it’s necessary!

#15

Well I might be interested in it 2, so could u post the code/link here?

Tnx.

#16

I think the word you’re looking for is “proxy” ?

Sticking to standard naming conventions would make it much easier to understand.

#17

nononono - the problem is that it’s too much data! :). Any interesting game, you’ll want to be carefully choosing what data to send. A common approach is to do a conceptual “diff” of “last game state sent to client X” and “current game state” and only send the diff. Various ways to do that of increasingly complexity and performance, but a basic version should be pretty easy for you to do and would handle much larger game-states much more gracefully.

I wouldn’t mention this, except that diffing and java serialization don’t necessarily mix well, so you might want to start getting used to custom serialization (i.e. not using the java api’s for it) sooner rather than later…

NB: the approach I described also neatly solves the problem you cited of server-data clobbering client data - the data the client sends will get picked up by the server after it sends the update, and so it will notice that has changed next time it does a send.

#18

I did some refactoring on my network-api to make it work better in there general case, and it’s fully functional.

So here’s my networking framework url=http://www.songprojector.com/static/riven_nio.jar[/url] url=http://www.songprojector.com/static/nio_riven_doc/[/url]

Basic NIO classes
[x] Network
[x] Server
[x] Client
[x] ClientHandler


public interface ClientHandler
{
   public void onClientConnected(Client client);
   public void onClientReadyToRead(Client client);
   public void onClientReadyToWrite(Client client);   
   public void onClientReceived(Client client, int bytes);   
   public void onClientSent(Client client, int bytes);
   public void onClientDisconnected(Client client, Exception cause);
}

Streaming NIO classes
[x] NonblockingBidirectionalStream (package access)
[x] NonblockingInputStream
[x] NonblockingOutputStream
[x] StreamedClientHandler
[x] StreamUnderflowException


public class StreamedClientHandler implements ClientHandler
{
   public StreamedClientHandler(ClientHandler handler)   { ... }
   public NonblockingInputStream getInputStream(Client client)   { ... }
   public NonblockingOutputStream getOutputStream(Client client)   { ... }
}

Usage (basic)


Network network = new Network(..);
Server server = network.createServer(1234);
Client client = network.createClient("ip_address", 1234);
client.write(ByteBuffer);
client.read(ByteBuffer);

Check out craterstudio.nio.test.TestBasicLayer

Usage (streaming)


Network network = new Network(new StreamingClientHander(...));
Server server = network.createServer(1234);
Client client = network.createClient("ip_address", 1234);

NonBlockingInputStream in = streamingHandler.getInputStream(client);
NonBlockingOutputStream out = streamingHandler.getOutputStream(client);
out.write(ByteBuffer);
in.read(ByteBuffer);

Check out craterstudio.nio.test.TestStreamLayer

Only these methods need some more optimisation:

  • NonblockingBidirectionalStream.read(ByteBuffer)
  • NonblockingBidirectionalStream.readFully(ByteBuffer)
    which I’ll do sooner or later (it’s actually pretty easy)
#19

I tried the JAR link to no avail.

#20

Pfff… fixed :-[