UDP Vs TCP/IP

Networking & Multiplayer
#81

[quote]Well FWIW I know that analog PPP does header compression. It was important to us back at TEN.

Since PPPoE == PPP over Ethernet I’m assuming that the packet protocol is identical.
[/quote]
If you want to check, I’ve provided all the necessary links above! I couldn’t find mention of it. If you are sure it’s there, feel free to read the RFC and enlighten us…

I’m not sure if you’ve read the rest of this topic, but I think we’ve pretty much clarified that PPP does compression by now. The point I was making above is that the reasons behind PPPoE were not some desire to improve technology (as you said yourself, there’s no real advantage to the consumer), and hence even the name is really just to reassure ISP’s who are trying to decide how to implement their BB strategy.

There is, of course, a long and glorious history in the IT industry of deliberately misleading naming schemes :).

Why add to the specification something none of it’s implementors will ever want? It’s a tech for ISP’s, not for customers (except that, since it’s cheap and easy to implement & maintain, theoretically customers will potentially benefit from lower cost BB access).

#82

[quote]Why add to the specification something none of it’s implementors will ever want? It’s a tech for ISP’s, not for customers (except that, since it’s cheap and easy to implement & maintain, theoretically customers will potentially benefit from lower cost BB access).
[/quote]
You are assuming they started from scratch.

I’m assuming they hacked existing PPP code.

In either case it is indeed an assumption. shrug

In any case we are getting stuck on something which, as you
rightly alluded, is technically interesting but performacne wise insignificant. If you have broladband you probably arent sweating an extra 22 bytes per packet.

Where the PPP handling of TCP really counts is right at the point where you need it most, across your thinnest pipes which by
definition are your analog users.

#83

A real-life example of the imporance of counting the number of bytes per packet (from a Gamasutra article that just went up).

Points to note:

  1. Xbox inflates the header sizes for packets
  2. Elsewhere in the article it’s revealed that Xbox’s TCP implementation is fatally flawed: it drops a lot of TCP connections. This allegedly breaks TCP’s “guaranteed delivery”, although I suspect that they were in fact using the API’s / protocol slightly wrongly (note the first para :slight_smile: ). They claim the flaw is due to Xbox’s memory-management, but I wouldn’t be surprised if MS fixes this; it doesn’t sound so permanent a problem as they make out.

"This is where I display my lack of experience in network programming, as I’m sure it will be common knowledge to anyone familiar with such things. At the start of the project I understood in an intellectual sense that packet headers could be expensive, but it wasn’t until I started working out some actual numbers that I realized the scope of the issue.

A standard UDP header is 28 bytes, and a TCP header is 40 bytes. On Xbox they are larger, due to the overhead of the packet encryption and NAT traversal, requiring 44 bytes for a UDP header or 56 for TCP.

This may not seem like much at first, but consider the case of 16 players sitting in a lobby talking to each other. MotoGP uses a 40 millisecond granularity on the voice compression, which outputs an 18-byte chunk of speech data. 18 bytes times 25 packets a second times 15 listeners gives a total data rate of 53 kbps, but if you add in the UDP header sizes, this goes up to 182 kbps. That is an overhead of 243%, and makes the difference between meeting or missing our target of working over a 64 kbps connection.

Obviously, you don’t actually need to send voice data every 40 milliseconds. The tradeoff is that the longer you buffer it up, the more latency you introduce, but also the less space you waste on packet headers. In MotoGP we send voice packets 4 times a second, which gives a 73 kbps data rate, 38% packet overhead. That’s still slightly over our 64 kbps target, but not by so much that I can’t get away with ignoring the math and pretending it is OK!

With so much of the total bandwidth wasted on packet headers, it is crucially important to minimize this in every way you can. Never send two packets where one will do. If you have a bit of data that you’d ideally like to send this frame, consider whether you might be able to hold it back for a while and then merge it in with some other information that is due to be sent in the near future.

Don’t mix TCP and UDP. If you send some data by one and some by the other, you won’t be able to merge packets, so you’ll end up paying twice the header overhead. Even if you have to increase the payload size in order to do this, you can reduce the overall bandwidth usage by combining everything into a single protocol. Since TCP is unsuitable for game data and voice packets, in practice this means you should use UDP (or the Xbox VDP equivalent) for everything.

Once you have all your packets running over the same protocol, make sure they all use the same port, too. That minimises resource usage, and due to an optimisation in the Xbox network stack implementation, if you put everything on port 1000 you can save four bytes per packet, too."

Link to full article:

#84

Well, I see a couple of potential flaws in his thinking.

The biggest one is that XBox live is broadband-only so unless hes being incredibly wasteful I don’t really get why hes sweating bandwidth at all. I don’t know where he’s getting his 64Kbps target.

Next, he doesnt mention whether his VOIP is done with nagle on or off. If nagle is left on then hes going to get amalgamated packets which will significantly drop his TCP overhead. Given that audio is usually buffered anyway and can handle quite a bit of latency this would same the logical way to go.

And his “don’t mix TCP and UDP” is too simplistic and off the cuff. There are legitimate reasons to use a combined strategy. There are some very sophisticated latency reduction techniques that depend on it (eg TEN’s B.U.L.L.E.T.).

The truest thing he probably says is that he’s not a very experienced network programmer, it sounds like he’s approaching it all at a very simplistic and naive level

#85

I’ve been re-reading this thread, now that i have a good set of reference books and such (TCP/IP Illustrated, Vol1-3)…

There seem to be some errors way back.

For instance… blahblahblah wrote that TCP packets can be much bigger than UDP packets and therefore TCP was just as fast or faster. But looking at the specs I see that this is not really the case. large TCP packets would be split. In fact if ethernet is used somewhere along the way the actual transmission packet size is often limited to 1500 bytes a typical Maximum Transmission Unit - the larger UDP or TCP packet must be reconstructed from these, but it does mean multiple ethernet headers etc. TCP headers (not counting fancy compression with PPP) are larger, they have to be to allow it to implement the ordered delivery and such that you get with TCP.

In terms of raw speed something that wasn’t addressed well are the throughput issues that the TCP stack has due to the need to process acks and re-transmit. This puts a theoretical upper bound on the transmission rate as a function of round trip time (ping) regardless of the speed of the channel.

Mind you I agree with all of the stuff about not trying to implement your own TCP-like reliability with UDP. For sure that’s a waste. But it does make sense to look at alternative protocols and strategies where appropriate. For instance it is obvious that if you can take advantage of multicast there could be tremendous savings.

There are also considerations like not bothering with guaranteed delivery and acknowledgments if you are just going to send the updated state in a few milliseconds anyway. (E.g. there is an implicit re-transmit because the client must be kept up to date… you know that if the client missed a packet the next one is already on the way. Of course this only works if the packets don’t depend of the client having an accurate picture of the previous state. But that’s all the sort of stuff you need to consider when you are building a protocol for a game.
Even if this information should be guaranteed, if you know the same state info will be sent many times then you should consider the probability that all of the packets will be lost and is it worth going for 100% reliability if the connection is that bad anyway. Maybe the problem having the client temporarily believe that things are not in the state they really are will sort itself out in a couple seconds anyway and that really won’t effect game play as much as it would if they were lagged because the protocol was forced to be reliable. (E.g. while waiting to make sure the last packet gets through or simply be acknowledged a new more up to date packet is being blocked from delivery.)

Considering the ratios of header size to payload etc. It might be more economical to send the full game state instead of deltas… Since sending a packet of just a few bytes and then the overhead of TCP/UDP (+ ethernet) headers doesn’t give you optimal use of the transmission channel.

It pretty clear that using UDP properly is more difficult that using TCP. I think we need some articles about it in the wiki. There are so many things to consider.

I have pretty much no experience with networking protocols - that’s why I bought the books… This is just my initial take on things after reading a few introductory chapters and looking at the fairly significant overhead in terms of achievable max packet sizes and headers at each layer of transmission.

#86

true

Not True for serial PPP. Serial PPP does header compression by storing the header info on the fat end of the pipe (the network end.) So if you are supporting 56K modem users TCP headers are actually SMALLER then UDP headers across the critical bottleneck, the last few miles to the user.

Not True. The sender doesn’t wait for ACKs. Its a NAK based protocol (at least in all modern implementations.) The sender just keeps pumping packets but keeps a “sliding window” of history of past packets. When it gets a NAK on an old packet, it retransmits from that history.

Reliable multi-cast protocols can be very useful within the datacenter. On the internet at large Multicast is still a dream.

Yep if your message doesn’t matter, send it parcel post :wink:

Yep. And trying to recreate TCP’s attributes is silly. A lot of VERY VERY smart networking folks have worked on TCP, and the network knows it and can do things to help.

#87

IMHO this is still the most important point. Guaranteed delivery allows NOT to send ‘unimportant’, discardable data. Or only to send IMPORTANT data.

And NOT sending anything is the by all means FASTEST protocol you can think of!

Another point - that I really dislike - is that brute-force UDP transmission wastes a lot of SHARED resources. It is in no way a computationally minimal approach. (for the same reason I dislike simple desktop games that run with 200fps in a tight loop although nothing moves)

Reducing network load to the bare minimum in my eyes is a fine art. I love that. Cannot do it with unguaranteed delivery.

#88

Hi
Is it your opinion that UDP is obsolete if TCP protocols are implemented effeciently?

Endolf.

#89

No. TCP or UDP still is application specific. And UDP is the better choice to setup own, task specific protocols (like Quake or NFS e.g.).

Maybe TCP is not the best protocol for games - but in my eyes it comes pretty close. Designing an own, guaranteed protocol can be extremely difficult (as Jeff mentioned) and the common approaches often taken in the games area turn out to be too naive and end up being very unefficient and yet not as reliable as TCP is. Doing it is just difficult.

And it IS a matter of taste. Quake is successful with UDP - bc. Carmack does not feel bad with flooding the network with redundant data. I do.

So far TCP did not cause any problems for me. My flightsim sends 1-3 messages per second. This is possible bc. I can be sure they will arrive. And I love to have a true algorithmic approach to avoid redundant data transmission. Latencies are there - as they are with UDP. Maybe higher maybe not. I have to deal with that anyway.

#90

Writing a TCP-like layer over UDP is actually not as hard as it sounds, because the people who write the TCP specs also document the algorythms for everyone to see.

I have recently written my own MMOG network layer (to support 1000+ clients on a couple of ports) using the TCP SACK system outlined in ‘RFC 3517’ (gogle for it, as it also contains links to the previous relevant RFC’s), and was quite pleased with the results and the time it saved using these docs.

  • Dom
#91

Hi
This is actaully the whole point, why bother?, if your gonna write TCP, why not jsut use it? If on the other hand you really want datagrams that are unreliable, then use UDP, but there is no point to rewritting TCP on UDP unless it’s for pure research.

Endolf

#92

I’ll have a look again asap (and edit this post - but I’ve no access at the moment and am on slow mobile dialup), but FWIW IIRC I was making the point that in practice TCP packets could be sent with larger payloads, precisely because TCP would efficiently manage fragmentation for you - whereas UDP packets would just die if you sent them too large.

IIRC I was talking about the way in which you use the protocol (take advantage of TCP features) rather than mere theoretical abilities of either - given that TCP can be implemented pretty well in UDP, at some point you of course have to look at the practicalities of each.

#93

[quote]I have recently written my own MMOG network layer (to support 1000+ clients on a couple of ports) using the TCP SACK system outlined in ‘RFC 3517’
[/quote]
;D If you implement TCP, you’ll get TCP …

What are the advantages of your impl over TCP? Maybe you are willing to share?

TCP in fact is not based on UDP but on IP - so I assume this can turn out as an advantage over every other approach based on UDP.

#94

[quote]No. TCP or UDP still is application specific. And UDP is the better choice to setup own, task specific protocols (like Quake or NFS e.g.).
[/quote]
i.e. UDP is always going to be useful as the “empty pizza base” on which to write whatever protocol you want? Libraries exist in all languages for dealing with UDP packets, and it’s usually easier to code to those than to talk directly to your network hardware etc.

E.g. in java it’s not possible to hand-craft packets except by using UDP.

PS TCP is not inefficient, it just has extra features that many games developers don’t use and don’t want. If it had two flags “disableInOrderDelivery” and “disableGuaranteedDelivery” it would probably be used for all game protocols :wink: - most people enjoy the other features of TCP that UDP lacks.

#95

I did it mainly because Im planning for 1000+ connections. For this:

  1. Header size too large
  2. Socket per connection is too expensive
  3. Congestion measurements can be differentiated between ‘client link’ congestion, and ‘server side’ congestion. This allows us to throttle either the whole server, or just specific clients which is impossible to do with 1000 seperate sockets. Each one just performs congestion avoidance on its own link.
  4. Network layer level encryption and security is better implemented at the lowest level.

For a simple 8-16 player peer-to-peer session then TCP is perfectly satisfactory. But it does not fit every case. Although i would agree that the range of cases it does work fine for is generally a lot larger than people give it credit for.

  • Dom
#96

[quote]PS TCP is not inefficient, it just has extra features that many games developers don’t use and don’t want. If it had two flags “disableInOrderDelivery” and “disableGuaranteedDelivery” it would probably be used for all game protocols :wink: - most people enjoy the other features of TCP that UDP lacks.
[/quote]
By jove!, i think he’s cracked it!. This probably sums up most of the past 90 odd posts :slight_smile:

Endolf.

#97

[quote]I did it mainly because Im planning for 1000+ connections.
[/quote]
1000 connections - damn good argument against TCP sockets.

Putting TCP-things over UDP - how much code is that? Just to give me an idea?

BTW, how do stateless app servers handle 1000 or more clients? Do they use TCP and create connections for each server access?

#98

Its around 2000 lines of code (all in C++ at the mo), half of that is mainly buffer management to remove any dynamic allocations for the game server (decryption of a valid packet is placed in a reference counted buffer that is tagged to a linked list of packets for each client, when done its added to a linked list of free buffers). The actual TCP you could probably do in around 2-300 lines.

Not sure what you mean by a stateless server, if its like a web server then they create & destroy the connections each time I think.

  • Dom
#99

[quote]The actual TCP you could probably do in around 2-300 lines.
[/quote]
2 lines really is very few code ;D

An appserver dealing with stateless beans. Do stateful beans keep connections open?

#100

[quote]An appserver dealing with stateless beans. Do stateful beans keep connections open?
[/quote]
Are you talking client-server J2EE here?

Stateful or stateless, a J2EE server will (usually) still be using HTTP to communicate. So the connection is made and broken for each interaction, apart from using Keep-Alives for referenced resources.