How to minimise internet lag/latency?

Networking & Multiplayer
#1

Hi everyone,

Thanks to Riven I’ve been able to test an internet-server-hosted version of my network game (which I’m itching to show you), but there is a problem with latency/lag and I’d really like some advice.

I’m not asking how to deal with lag, just how to minimise it over a TCP internet connection. For example, I find that sending lots of small messages(<100 bytes) tends to jam up things and latency spikes. I send a message for each mouse-move and this creates lots of messages per second. The latency between the computer I’m testing on and the internet server is normally 175ms (Sydney, Australia to Amsterdam, The Netherlands) but it spikes up to 700ms easily by waving the mouse around, and eventually seconds if two instances of the game are joined and both wave their mouse around for a while. Waving the mouse around can cause > 40 mouse move events per second, each of which I send separately.

So here are my questions :P:

  1. Is sending lots of small messages a bad idea? Should I delay them a little and send bigger byte arrays less often?
  2. I also send big byte arrays up to 10kb over the tcp connection every 5 seconds from the server to the clients (10kb only occurs when there’s lots of activity, normally it’s only 3kb), could they be jamming things up?
  3. When a lot of stuff is being sent from computer A to computer B over a TCP internet connection, does this delay stuff being sent from computer B to computer A?
  4. Is there any way to tell when the TCP-connection is being stressed so I can lay off it and not send stuff for a while? It seems to get progressively more bogged down when more stuff is written to the stream when it’s already lagging.
  5. Got any other handy tips?! ;D

Thanks! :slight_smile:
Keith

#2

For lots of small messages, you’ll want to turn off Nagle’s algorithm ( Socket.setTcpNoDelay() ) as it will batch up your messages to reduce overheads at the expense of increasing latency.

Socket.setTrafficClass() looks interesting too, but I wouldn’t be surprised if it had minimal effect.

It’ll probably be a pain in the bum, but have a think about moving to udp and handling the missing packets yourself. Especially for time-sensitive stuff like mouse move events, by the time a missing packet is detected, requested and retransmitted (3 round trips, assuming that it’s detected immediately and doesn’t get lost again), is the missing data of much use? You might be better off just ignoring it and lerping between the last known and latest mouse positions.

#3

I just checked and I’m already calling setTcpNoDelay(true).

About Socket.setTrafficClass(int tc), what int argument do I pass to it? I don’t understand what the java-docs say: http://java.sun.com/javase/6/docs/api/java/net/Socket.html#setTrafficClass(int)

Good idea with UDP, I suppose that’s the real solution for transmitting timely stuff that’s not essential (like mouse moves). So do most twitch games still use UDP? I was under the impression that UDP was on the way out? If having udp (as well as tcp) is the way to go then I’ll have to learn a proper network API.

Thanks for the ideas bleb :slight_smile:

#4

setTrafficClass( 0x10 ) would appear to request low latency service, but it requires the cooperation of every switch and router between Sydney and Amsterdam to have any effect. Don’t get your hopes up. It’s kind of weird that there doesn’t appear to be constant definitions for the flag values though.

UDP isn’t going anywhere, it’s pretty much the atomic element of networking. It seems to be a universal rule that (bandwidth|memory|computation) requirements expand to fill available capacity, so there’s always a need for the bare-bones minimum.

#5

A quick google on “game-ports” or something like that reveals that most games today use UDP (often in combination with TCP). I’d also go for UDP in your case, like bleb said :slight_smile:

#6

UDP or TCP, if you’re sending messages for every mouse move when it happens you’re going to get into problems. Maybe time to rethink the algorithm a bit - depends on what you’re sending of course, but if it’s like the direction in which a player is aiming based on the mouse position maybe only send every half second or something?

Kev

#7

Regardless of whether you use UDP or TCP (and IMHO you should stick to TCP…), you have to combine your packets.
Write all your packets into some buffer, and ‘flush’ that buffer every N ms.

If you want to ake it a bit more highlevel, you should have a structure that removes packets (or highlevel structures) that are outdated, before they are sent.

#8

Thanks bleb for reading it and giving a simple answer, I’ll try it out.

Riven, I’m interested to know why you think UDP isn’t worth it? If using it is likely to fix the latency problem, I’ll learn a network frame work and then it shouldn’t be too hard to use it in my game (I’m going to look at JGN and Apache Mina tonight). That’s a good idea about the buffering. Hopefully one of the network frameworks takes care of it…

Thanks for the comments Noya, Kev and Riven. So if I use UDP to transport most of the mouse-move events (and only send say 1 per second over tcp), is that likely to fix the problem? In other words, is the problem because those events are being sent, lost and re-sent over tcp, or is the lag due to just sending too much stuff over the network full stop?

#9

Most likely a combination. You want to send as little as possible to give the feeling of syncrhonisation but how you send it matters too. The best algorithms seem to make use of the features of the protocol - TCP gives you ordering, guaranteed delivery and congestion control (unless you turn off nagles), knowing that the messages will always be in order or that it will always get there helps with client algorithm, UDP lets you fire and forget - this helps design algorithms that send more data but can cope with lost steps, sometimes writing a layer over UDP to guarantee delivery but don’t worry about order drives the algorithm.

By the sound of it you’ve got network traffic directly proportional to player input - that sounds like a bad move to me, it just won’t ever scale. You need to throttle the amount of traffic a client to send in some way, normally the how or with what is specific to the game.

For real time games theres a good article about X-Wing vs Tie Fighter hanging around on gamasutra, though I can’t find the link any more - maybe someone else has it book marked. I think Endolf looked into it not long ago for his Dark Void game.

For rts/delayed action game I’m finding a variation of lock step where the step control is handled by the server and is continuously moving forward works really well - increadibly low traffic for huge amounts of units.

Kev

#10

Thanks for the long reply kev.

Good point, I suppose 40 mouse moves a second is a little over-kill. I’ll have to slim it down. Hmm, maybe I’ll do that before going to town on UDP - it’s looking like a giant ‘pain in the bum’ as bleb put it.

It’s surprising how bad MINA is at providing examples - in a simple server-client example they manage to tie-in about 4 separate jars as well as log4j and spring (I don’t even know what this is ???). It’s not for the feint-hearted!

Interesting… if you ever get time I’d like to know more about this rts design - how does it handle lag?

Thanks for the info guys, it’s really helpful.

#11

I blogged about it last time I was working on the game concept (like most of them, I’ve since given up again, however the networking was up and running with multiplayer across the internet and hundreds of units).

http://www.cokeandcode.com/node/915

Ignore the first half of the post, it game specific, the networking stuff comes on later.

Kev

PS. Can see some screenshots on the game running here: http://slick.javaunlimited.net/viewtopic.php?t=325&postdays=0&postorder=asc&start=0

#12

Thanks for pointing me to that.

First of all: sweet game, it looks terrific.

About the networking approach, I think it’s pretty clever - sending only the incremental changes is a good idea in a big rts, and having a 100% deterministic world is great since you never need to send the whole world to all clients.

The only downside is that the clients can’t do any prediction. So a mouse click doesn’t do anything until it’s sent to the server and the server sends it to all clients and/or sends a ‘tick’ to the clients, allowing them to move forward another step. But as you said, that round-trip lag isn’t really an issue in an rts, and it’s minimised if you provide visual cues.

#13

Hi,

In case anyone is interested, I tried using Socket.setTcpNoDelay(false) on the server and client and it resulted in latency of about 300- 600ms compared to when it is set to true on both server and client and then the latency is 160-190! (This is the latency from Sydney to Amsterdam).

So Nagle’s algorithm makes a difference!

I’m now using the Apache MINA (http://mina.apache.org/index.html) network code and it works pretty well. I’m still using TCP but I’ve limited the mouse moves to 20 per second and that seems to be OK. I think that MINA does efficient network code since it obviously uses NIO properly and it also does thread-pooling for sending and receiving stuff on the client and server.

If anyone wants any help using MINA, let me know. It’s actually very hard to figure out :o - it’s way over-engineered and the examples are hopeless. I had to jump through hoops to figure out how to send a byte array but now I’ve got code that sends byte arrays and does compression using jzlib. 8)

PS: I also tried Sunset’s (Matt Hicks) Java Game Networking code (http://forum.captiveimagination.com/index.php/board,4.0.html) and it was much easier to get going and also supports UDP out of the box :D. It works just as well as MINA, but when stress-testing it would throw com.captiveimagination.jgn.queue.QueueFullExceptions. It’s probably my fault, so I’ll try and work through the issues with him.

#14

…and yet people keep bashing us for not using it :)…

#15

Hi !
it’s been a while since I was interested in networking in java games. Now that I’m aiming to an overall implementation of the net packages of Java in my own game, I have read the last posts and I can say someth’ about TCP vs. UDP.

  • First thing is, Transmission Control Protocol (aka TCP) has been intended to integrate all the necessary and the capabilities for a synch’ed communication between 2 remotely connected computers. That’s about how TCP/IP is defined.
  • Second thing, unlike TCP, the User Datagram Protocol (aka UDP) is targeting communications requiring an user defined transmission DataGram to be created and sent over a netowrk connection. This is done by identifying the type of the Datagram and when it can be sent. That is how we could define an asynchronous communication. Because the server always send a message to inform that someth’ has to be sent to the client, where the latter client is always PENDING for data unlike TCP clients are always REQUESTING data. 8)
    Hence games are always PENDING for the remotely connected computer to send new game STATES or EVENTS, that’s why I’d really target all the net stuff over some UDP connection, where UDP connex’s are obviously faster than the TCP ones.
    Tell us if you find more accurate definitions and I’m may apologize for the eventual mistake ! :slight_smile:
#16

Please don’t. There’s an entire thread on this forum about UDP vs TCP. Read that. If you get to the end of it, and STILL feel you have something to add, please add it to that thread, or start a new one. (hint: there’s a lot of reading ahead of you…)