[UDP] Packets getting clogged up in server

Networking & Multiplayer
#1

Hi, I’ve been making a server and client program. It is mostly going well, apart from the fact when clients join and send lots of packets to the server, the server starts to get behind in responding to the clients. The strange thing is that i have a while loop executed each frame in the server that doesn’t break until there is a timeout(no more messages to receive). And I only reply to the most recent message from each client and discard the rest. The server works fine with say 2 clients sending at 30 packets per second, but if i keep adding clients (say 4 or 5), this problem starts occurring.

The only reason I think that this could happen is if the clients are sending so many packets that the server gets stuck in the while loop forever, but this isn’t the case. The server replies, it just can’t reply as fast as the clients are sending the packets, and over time it is replying to packets that clients sent 20 or 30 seconds ago or even longer if I just leave it there. I don’t know why it is replying to old packets though - my while loop should receive every message and remove all the old messages. It shouldn’t be replying to messages it received 20 seconds ago when there are packets that have just arrived.

Does anyone know why this could be?
Thanks,
roland

#2

Sounds like a bug to me. The While loop is not really doing what you think its doing. Or its taking a long time to do something.

Perhaps you could post code?

#3

Here’s some hopefully relevant code. I took a lot out.


//Server thread, created when server starts
public void run() 
{
	while(m_bRunning)
	{
		Process(); //includes sending packets if possible
		try{Thread.sleep(SLEEP_TIME);} catch (InterruptedException e){} //sleep time is 5ms just to not use all the cpu
		Thread.yield();
	}
	Release();
}


private void Process()
{
	if (!m_bRunning)
		return;
	
	ReceivePackets();
	
	//UDPConnectedClient contains stuff like client address and port, packets received, packet waiting to be sent, etc.
	
	for (UDPConnectedClient client: m_clients.values())
	{
		byte[] packet = client.GetReceivedPacket(); //returns the last received packet (pops off a stack of received packets that are cleared each ReceivePackets())
		
		if (packet != null)
		{
			ProcessPacket(packet,client);
			if (System.currentTimeMillis() - client.GetLastSendTime() > PACKET_INTERVAL) //can take this out for instant reply to received packet
				SendPacket(client);
		}
		else
		if (System.currentTimeMillis() - client.GetLastSendTime() > PACKET_INTERVAL) //packet interval = 1000/30 
		{
			SendPacket(client);
		}
	}
	
	//other code here for checking client disconnects etc.
	...
}

protected void ReceivePackets()
{
	if (!m_bRunning)
		return;
	
	for (UDPConnectedClient c: m_clients.values())
	{
		c.ClearReceivedPackets();
	}
	
	while(true)
	{
		try
		{
			byte[] data = new byte[PACKET_SIZE]; //1500 bytes
			
			DatagramPacket recievedPacket = new DatagramPacket(data, data.length);
			m_serverSocket.receive(recievedPacket);

			// get the client details
			InetAddress clientAddress = recievedPacket.getAddress();
			int clientPort = recievedPacket.getPort();
			UDPClientInfo info = new UDPClientInfo(clientAddress, clientPort);
			
			UDPConnectedClient client = m_clients.get(info); //find the client that matches the ip/port from a hashmap
			if (client != null)
			{
				client.AddReceivedPacket(data); //pushes the packet onto the top of the client's received packet stack.
				continue;
			}
			//Else It's a join message and process it seperately
			ProcessOtherPacket(info,data);
		}
		catch(SocketTimeoutException e) //times out after 1ms if there is no message
		{
			//THAT's OK, there just isn't any message.
			return; 
			//AddDebugMessage("Timeout: "+e.getMessage());
		}
		catch (IOException e)
		{
			AddDebugMessage(e.getMessage()); //never had this occur yet
			return;
		}
		
	}
}

private void ProcessPacket(byte[] data, UDPConnectedClient client)
{
		//read the packet, add an acknowledgement of the packets sequence number to the client's next packet to be sent,
		//discard the packet if it is corrupted or the sequence number is lower than the last received packet
		//calculate ping from the ack inside the packet
		//send chat messages back to clients
		//etc...
}
#4

Well there is some odd code there… I will start with what looks like the biggest issues first.

The ReadPackets method will never return if there are packets getting sent all the time. This is already going to create unintended behavour pattens once there is any load at all. The proper way to do this to have that method run in a different thread then use thread safe structures between the send/receive parts or the code.

The lines


if (System.currentTimeMillis() - client.GetLastSendTime() > PACKET_INTERVAL) //can take this out for instant reply to received packet
           SendPacket(client);

make no sense to me. why is that conditional? if you need to send something as a reply… you need to send it, why is it conditional?

A style comment, methods don’t start with capitals as a java convention. Only class names do or static fields which are typically all caps. Its a good idea to stick to conventions.

My last comment is why are you using UDP?

#5

Thanks for the reply delt0r :slight_smile:

Ok, I will do that. Thanks
EDIT: what’s a good structure for making sure that you aren’t sending a packet in one thread at the same time as reading a packet from another thread (using the same socket)?
EDIT 2: does it matter if i use DatagramSocket.send() and DatagramSocket.receive() at the same time?

Because I want to limit the amount of packets sent to 30 per second… I will reply, and because my model allows for dropped/out of order packets it doesn’t matter if i send the packet 15 ms late. If you know a better way of limiting the packet rate but still replying instantly please let me know. This is a real time game btw.

Yeah, I’ve been programming in C++ for atleast 3 years, I’m trying to switch over… :clue:
It also helps differentiating between my methods and other people’s xD although that is a really bad excuse I know

I know you think that TCP is just as or nearly as fast as UDP. I don’t agree with you there - there is a lot of information about it, I’ve asked people about it, seen what GOOD networked games use, and concluded that UDP is the better option. anyway, 95% of my data is unreliable, so I see no point in using TCP just for the last 5%.

#6

I have been working in networking for a long time. I have done the seti online thing for other projects and i have quite a bit of deployed network code of my from my commercial life monitoring some very large teleco networks. I assure you i base my assertions on fact and experience. Seriously. UPD and TCP use IP packets on the same network. Why do you think the ones that are TCP somehow travel slower?

Also most games out there do not have good networking. Read their forums if you don’t believe me. In fact after looking at how some behave with even a small amount of jitter and lag, many big games have very poorly implemented network layers.

UPD is only better if you don’t need flow control and don’t care about order and don’t care about receiving the packets. Quake3 fits this very nicely. New games are starting to use TCP more and more now. ie WoW.

Also based on the code you have very little experience with networking and reading between the lines i think perhaps you have some pretty fundamental gaps in how it all works. Even if you still want to use UDP, i would recommend trying TCP as an intro into networking.

Otherwise you need to get up to speed with basic networking from a Coders perspective. For example why would you not be able to send at the same time you receive? What does one have to do with the other? So yea, you can do both. Typically in different threads… You need to read up on how to sync threads if you need to or use thread safe data structures etc.

Also when you limit your packets… this really is not how you should do it… in fact i think you just miss sending anything in this case. Also why would you only send packets if you receive them? If there is something to send… send it. Flow control is a hard problem of course, one that TCP got wrong originally and has been refined over the years. But again having different threads makes this much easier. However if you have more to send … then what? the que fills up till you run out of ram? Again… not as easy as it looks…

Sure you can do it, but its a lot of work, and quite advanced work at that.

#7

Yes, I am very new to UDP (Not UPD :P), I have to learn somehow. Why would I be asking for help in the first place if my code was ultra perfect? I can’t just make TCP apps all the time and expect that it will teach me how to use UDP. I agree that TCP is in most cases the best option, but when every ms counts UDP is better. I know that TCP must Acknowledge every packet, which does slow it down. The quake 3 model is faster than TCP because it doesnt care about dropped packets, it just sends the unacked data in every packet. I am using the quake 3 model, and no, I don’t want to use TCP, not even if “WoW” uses it. I didn’t say lots of games had good networking. I said the games I knew that had good networking used UDP.
Googling “tcp vs udp multiplayer real time game”, TCP seems ok for MMORPG but for FPS games not really. I’m not afraid of doing extra work if the outcome is better.

#8

TCP does not need to wait for ACK before packets are avalible and the sender sends packets ahead of ACKs with the assumption it will receive them soon. So on a network with low packet loss, you get the data with exactly the ms delay as you do with UDP, sometimes even faster depending on the isp DSLAM settings. I have tested this quite a bit. Its quite easy to verify. These IP packets don’t travel slower. UDP is faster is a mostly a myth perpetuated by the gaming industry, its the same ip packets over a the same network. And if every ms counts, you don’t use a network where players are going to be having a ping of 20-200ms over the internet. UDP can’t beat the ping. And network jitter over the internet can get quite bad ie you gain and loose 10s of ms all the time. You can’t need something to count on every ms.

I suggested TCP not because your errors are UDP specific but because they are quite general to network api concepts in general. I have my doubts you’ve done this with TCP before.

A FPS where the total player count is low is one area where UDP is a good choice. But they use a much simpler method to do things. This is basically the q3 protocol and googlling can give you some details. First the server sends the whole state of the game out at a fixed rate. ie this is just part of the game loop on the server. Then the clients just take the most recent ignoring anything it has seen before. Finally the game clients just send packets when needed of player actions, (move, point shoot). These are just send as needed. However you can rate limit if you want, this would be done with a thread that just empties the “action que” once every n ms by combining them into one packet. I would just use Thread.sleep for that. Its good enough. But this does add artificial delay.

Note that this protocol on its own does not give you ping times… Since there is never a response that is caused by receiving a packet. I do believe in quake this was done with special “heartbeat” packets.

#9

So I could send full packets at fixed intervals of 30 ms or whatever, but whenever i receive a packet i could instantly send an acknowledgement for that packet?

#10

Why do you need to send an acknowledgment? If that packet didn’t get through, you send another one in 30ms anyway right?

#11

yes

#12

If you don’t mind me asking delt0r, are you suggesting that Roland use TCP due to lack of experience, or because you think it’s a geniunely viable option for real-time mutiplayer gaming? I’ve been looking into this myself recently and there is a lot of conflicting advice, though the majority say that UDP is more suitable. That said, many comments from people with real networking experience (though not necessarily within games) state that TCP isn’t nearly as flawed as is made out.

I recently integrated a TCP solution into my game and with a bit of tweaking it’s fine apart from the odd ‘hang’, which I assume is either a packet that has been delayed/dropped or TCP ‘backing off’ because of network load. I’ve been lead to believe that these hangs don’t occur with UDP because it doesn’t need to guarantee packet delivery - but as far as I know that just applies to sending packets, not receiving them. If TCP packets aren’t getting through, could I expect the same delays with UDP? I basically need to be able to receive at least 1 packet per second or the game really suffers.

#13

If you need the packets, you need the packets. If you have to time out for the UDP packet, you’re reimplementing TCP badly. If the network is so congested as to start interfering with TCP, UDP is going do do even worse.

#14

[quote=“sproingie,post:13,topic:37842”]
Seems so obvious now that you’ve said it. I’ll see if I can improve my implementation, at the very least I can probably reduce the amount of packets I’m sending.

#15

I never mind anyone asking :smiley:

I was saying several different things here. First i was suggesting TCP because it a little bit easier to learn the api IMO. Simply because it behaves just like a stream/pipe/file.

Next i am saying that the vast majority of games network code is written by people who are not network people but game coders. So they do what other game coders did. Pretty common problem, but almost always the wrong way to make the decision.

The dogma on the internet out there with TCP==slow is the same as the dogma that java is slow. Sure there are slow java apps, there are also really slow and mem hogging C apps out there… But C is never blamed… Same with UDP… if a game has bad networking, the decision to use UDP is never blamed… perhaps it should be.

For the vast majority of people starting with networking, TCP is the right choice. Switching to UDP if it makes sense is pretty easy. Making UDP behave as TCP is not. It took a planet full of expert network engineers to get TCP as good as it currently is. There was even total network collapse while they where figuring it out (yes in the old days the internet crashed!). It is still a very active area of research. Both linux and windows have updated their flow control algorithms in the last few years.

I assure you, if you have not heard of the 2 Generals problem, don’t know what flow control is or why you need it, don’t know about MTU and IP packet fragmentation, and/or need packets somewhat reliably delivered. Use TCP! Don’t think that because the crap protocol you came up with works well on a lan it will work on the Internets… the tubes can get clogged ya know.

The exception is outlined above…If you can keep the packet at the standard MTU of about 1500 bytes IIRC. [edit] That is including headers, so payload is smaller.

I could go into more detail. But perhaps that is something for its own thread.

#16

Thanks for the input, I’m pretty new to networking in general so it sounds like TCP is my best bet for now. I guess in a way I was hoping UDP was some sort of magic bullet with regards to latency, but the more I read the deeper the rabbit hole gets. It’s looking more and more as though the issues I’m experiencing are due to my implementation anyway, although it’s a real time game it’s not particularly fast-paced and I’m pretty sure TCP is the right choice for me.

Thanks again.