I happend on an old thread and noticed that BB had asked me some time ago to describe BULLET and how we used a combination of TCP and UDP at TEN to get what at the time was measurably the best Quake2 play on the internet.
Did I answer that question well enough or should I write something up now?
I would be very interested in an explanation as I’m trying to write the best game networking API for Java and am currently in the last stages of adding joint TCP/UDP support for the client/server architecture.
-Matt Hicks
Okay. This si from memory but…
B.U.L.L.E.T. or Bills Ultimatew Latency Lage Elmination Technology worked as follows.
Background Information:
In order to understand BULLET you have to understand a tiny bit about TCP/IP. The first thing to realzie is that, under normal conditions, TCP and UDP perform almost identically. The difference in behavior, and thus performance, is when a packet is lost en-route. UDP just ignores it and keeps flowing the packets-- the data is lost. TCP/IP stops the flow of pacekts and sends a packet back asking for a re-transmirt of the lsot data. Because TCP gaurantees in-order delivery it cannot deliver the later packets until it gets that lost one so it queues them up. The result is a latency spike and then a whole bunch of packets being delievered one right after the other.
What BULLET did was to cover that latency spike with a second channel of data. In addition to sending the packets via TCP, they were collected and periodically sent as a “window” of packets in a UDP channel. We found it was fairly rare for BOTH the TCP and the relevent UDP packet to be lost at the same time. So if the UDP packet arrives with packets that the TCP channel hasnt delivered yet, we just continue the flow by sending those to the client. When the blocked up TCP does arrrive, we throw away the already seen packets.
Because order is not gauranteed in UDP its possible we mkight recieve the “windows” out of order. We handle this by locally storing any “future” windows until we’ve passed through their packets.
Obviosuly we cant use the TCP layer’s packet numbering for this ebcause its not exposed to us, so all these packets have a few bytes of packet nukmber pre-pended before they are sent.
We found that this radically reduced the ocassional laytency spikes you see in TCP and which caused us so much grief in gameplay.
It was preferrable in our judgement to a pure UDP reliability scheme for two reasons:
(1) For various reasons, the network-layer-level TCP has advantages over an attempt to do the same thing at the user level. For instance, internet routers can reckognize TCP packets and thus give them priority over UDP packets when the router backs up.
(2) Its not reinventing the wheel. TCP was already there so why take the extra cost and added bugs-risk of building reliable in order delivery ourselves out of UDP?
Very nice. Is this an open-source project I can cannabalize for parts…cough I mean look at exactly how you did this? :-p
Aren’t you essentially sending all the same information twice? Once through UDP and once through TCP? Isn’t that a large waste of bandwidth?
I actually ended up at the same conclusion, albeit further down the road. I realized that UDP had significant advantages over TCP for latency and speed but there is also the need for reliability in many cases as well. I ended up writing that “user-level” reliable UDP and it works actually quite well but it’s significantly slower than TCP for obvious reasons when a message is undelivered. Now, I’ve gone ahead with JGN and added support for both TCP and UDP and a CertifiedMessage sent through UDP is guaranteed deliverable because of the implementation how it handles it, but that same exact message can be sent over TCP and all the “CertifiedMessage” aspects are completely ignored and just relies on the features of TCP.
I would be very interested in a critique of my API as I’m sure you have a lot more insight to the concept.
BTW, I noticed a strange problem that started popping up when I would receive a message via a Datagram and would get the InetAddress of origination from it and then would try to use that to send a message back via TCP (on a different port - the correct TCP port for the remote server) and it would say “Protocol not supported”. Have you ever run into this? I ended up having to create a work-around that simply:
byte[] addy = address.getAddress();
address = InetAddress.getByAddress(addy);
Then it would work just fine. It may also explicitly have something to do with both client and server being on the same machine.
-Matt Hicks
For my networking system was designed to save bandwidth at all means, I cannot like that idea.
I dislike the need of an additional int to be send just for packet identification (although sending everything twice is worse, of course 
).
But there are other things I don’t feel comfortable with:
Socket#setTcpNoDelay())What were the environments BULLET worked on?
Herkules,
I would say you’re an extremist though. :-p Bandwidth must sometimes be minimally sacrificed to a greater purpose.
For networked games cannot live by bandwidth alone. :-p
-Matt Hicks
By no means. I just started this networking thing when 14.4k modems have been common. Every byte is sacred.
But even today, upstream bandwidth on DSL lines still is poor. And thats what counts for a ‘casual server’.
numplayers = k * (bandwidth/messagesize) … this holds today and forever
And I’m still a believer that the battles are lost and won on higher levels, e.g. gameplay first, smart update/distribuition schemes and proper factoring next. Not on the lowest network protocols.
And I don’t like probability in computation.
And I have seen networks (LAN!) dropping 90% UDP.
And I’ve seen projects being delayed for month bc. somebody said ‘UDP is faster’ and after that failed to deal with all these buffers, threads, locks, timeouts, states, re-sends, overlaps, … necessary to make the thing work reliably.
I’d suppose the InetAddress object retrieved from a datagram is of a special subclass of InetAddress which contains information about the protocol. Or InetAddress carries this information with it. InetAddress is quite a complex object…
I’m with Herk here. I’m quite surprised that adding a UDP in conjunction with the TCP improved things:
If you throw out a whole bunch of UDP packets onto the line the TCP algorithm is going to do what its designed to do - detect load (according to the TCP RFC the algorithm is designed to work across mimimal connections) - and then back off - adding the UDP would probably cause the TCP to backoff and increase the window size more often - which in turn would cause the detection of lost packets to be detected less often - and hence the lag spikes would be increased.
The UDP data you recieve might be useful but you can’t then (especially from java) go diving down into the TCP stack to tell it that you’ve actually got the bits of data its been waiting for and hence the TCP is still lag spiking over and over - does your UDP stream ever get synced again? Of course this would be case except for…
As you said, TCP and UDP perfom pretty similarlly on a lossless connection. So, if you start getting gaps in your TCP stream then its because you’re losing packets (or some nasty router has stripped them because they’re not “priority”). If you’re losing packets you’ll lose UDP too… its just you won’t loss the requests for resends sent the other way. Seems like a very slim improvement (although one I’ve found quite useful so far)
However, given this is Jeff we’re talking about I’m pretty sure this is anecdotal performance figures and hence this definitely did give improvements. So, could you give us some more details in the environment where this improved things. Were there dial up lines involved alot? (TCP header compression?) Was this quite a while ago (before routers bothered looking at UDP packets enroute)?
Bandwidth/Network performance is the same as Grapihcs Performance - the more performance you’ve got, the more other things you can put in (of course theres a trade off here for time in development). However, if we were talking about networked physics for instance - I’d want all the room I could get
Kev
Speaking of networked physics… :
I’ll be putting up a new release of my networked physics API as soon as JGN 1.05 is released. JGN 1.05 will offer complete support for TCP and UDP, time synchronization, and P2P. The new release of the PhysicsNetworking API sends all physics messages over UDP and any player information (join, chat, disconnect, status change, etc) goes through TCP. It works quite well. I’ve got Roll-A-Rama (my PhysicsNetworking demo game) using all the new features and it’s working quite well on my LAN. 
Herkules, the basic content of a Message in JGN consists of a short (message type) and two longs (message id and time sent - this will be adjusted when time sync to be the time sent on the remote machine that is receiving it). Even peaking the number of messages I’m able to send over the network because of software performance (over 2,000 sec) that won’t even put a dent in standard DSL bandwidth. :o Now, definitely I agree that bandwidth minimization is beneficial, but I gain several advantages by sending that information across that could be sacrificed, but I think the advantages outweigh the disadvangtages.
I agree that InetAddress is a complex object but you wouldn’t think it should have any ties to the protocol. That just seems wrong. :o
I’m still hoping one of you networking elites will give me some good constructive feedback: http://javagamenetworking.dev.java.net
-Matt Hicks
?
2xlong+1xshort = 18byte
*2000/s = 76000byte/s
Thats a multiple of a common DSL upstream and at least 5x overcommits the usable capacity. For now just with overhead.
Though I haven’t confirmed your “+20bytes for anything UDP” that is not entirely accurate for my system as it allows multiple messages to be sent in a single packet which even if what you say is true would significantly reduce the amount of data being passed through the pipe.
-Matt Hicks
http://www.networksorcery.com/enp/protocol/udp.htm
Where does this 2000 come from?
I benchmarked HQ message sending a bit. Sending + decoding empty messages leads to ~20000/s using the loopback TCP line and ~300000/s using HQs ‘local’ setup (which just copies bytebuffers) on my poor notebook. I’m pretty sure profiling the network case would show that logging eats up the time.
Nope sorry, it was a TEN propriatary thing, which is now wholly owned by EA along with the rest of the TEN/POGO. In fact AFAIk this is the first time its been really descibed publicly. Its so old now,though, and I’ve seen enough thinsg out there that are simialr enough, that I figure Im not doing any great harm by discussing it.
yes indeed. The tradeoff is bandwidth for reduced latency spiking.
Im honored to be asked, but til GDC is past I have really limited bandwidth 
.
We actually just ran into something very similar in Project Darkstar. In our case the behavior was unique to windows. I believe one of my guys is going to file bug on it shortly…
Its directly related to the worst spike you are willing to tolerate.
Yes on the first half but not likely on the scond. It depends on the load at the router, which is concetrating a great many users’ pacekts. Just doublign your data isnt likely to impact that much.
true. We didnt find that sort of behavior on the networks TEN worked on which wer egenrally of pretty high grade. Most of our users were dialup which meant we had some control of the netwrk from connection point in.
The TEN network. I dont remember the statistics but probably mreo then 50% of our users used the dial-ins we provided, which were supplied by Concetric. TEN itself was tied pretty well into the backbone of the net. We had both east and west serves AIR because at the time the MAE EAST/WEST bridge was a bottleneck. We had multiple connectiosn to back bone points in order to get aroudn some of the other worst bottlenecks.
[quote=""]
I have an easy solution, get JGN an invitation to GDC and I’d be happy to explain it in person. Speaking of which, I have been working on a networking demo utilizing Swing in jME, jME-Physics, PhysicsNetworking, JGN, Shadows, and other cool things hoping that jME would be invited to GDC and that jME would be willing to show it off. Is there any word on this?
No worries, I’ll bug you about it again after GDC. :
I guess your premise in this API is the scenario that bandwidth is cheaper than lag. For most broadband connected machines (which makes up the majority of the gaming community) I would suggest that’s probably pretty accurate. I guess the real potential problem lies in situations where a TCP message gets lagged and the UDP message never reaches its destination. In that case you would still see a lag spike but hopefully minimized occurence.
-Matt Hicks
On GDC, I’m sorry, but we are seriously limiting the demos we are showing this year. The focus this year is very much on Project Darkstar.
I might be able to scam you an expo pass, but thats the most I could do this year.
No problem, I’ll just have to wait until next year…I’ll have a lot more to show off then anyway.
I bet a could write an implementation of BULLET (based on the info you’ve shared) in JGN relatively easily since it would just be a combined message queue with a UDP and TCP receiver that keep tabs on the messages so duplicates get dropped… hmmm. :o
I’ll have to play around with that…it’s so hard to test networking concepts because you need a crappy internet connection to test with. :-p
-Matt Hicks
What’s project Darkstar?
-Sam
…and colesbury was never seen or heard from again. :’(