Shardless... related

Networking & Multiplayer
#1

Hi… I’m quite new to this MMO server stuff and looking for some adivise.
I always belived that it would be technically hard to serve more than 4 digits ppl in a single shard.
I heard its max 5k per shard.

But Second Life and Eve allows everyone on at the same time in the same place. (1 shard only). At first , i was like HOW??? and studied their tech docs and stuff but couldn’t find any clue
how they did it. Since SGS’s one of the main feature is to provide shardless architecture… (right?) please enlighten me to the path of shardless game server architecture.

thanks in advance… and sorry for my poor english…

#2

You can imagine a system with one big server, and all the players are on that one server, which keeps track of where everyone is and what they’re doing, right?

At the other extreme, you can imagine a system where every player has their own server, and each CPU has to communicate with all the other CPUs in order that they can keep track of what everyone’s doing and where they are – right? (This is an extreme P2P architecture.)

So now imagine various schemes that are in between the two.

In particular, I believe that the SGS back-end is designed to be load-balancing, and to allow additional servers/CPUs to be added to the cluster at any time, and load is just shifted off to them.

If you’re new to these concepts, google “load balanced servers” and similar (or try wiki.)

#3

Okay, first EVE. I went to a talk by those guys at GDC…

Eve does not really run all the players in the same simulation. The EVE simulation is deterministic and they run many many simultaneous instances that do minimal inter-communication.

In the center they run a central database that they have pushed to the maximal edge by throwing Big Hardware at it (millitary spec TI RAM hard disks for instance.) Their only answer to future scaling was “well Moore’s law has saved us so far…”

That being said, they still run into big scaling issues even today when combat happens because it doesn’t fit this model and they admitted in the talk that it doesn’t scale well at all.

Second life uses a computer per “continent” and limits how many people can be on the continent at once.
This is incredibly expensive but they get away with it by making user “land speculators” pay for the computers.

So the short answer is, both your examples work only by putting severe limits on the game play and the kind of game that is implemented. Neither are anything like Darkstar. Darkstar works by actually solving the problems of a distributed interactive database and distributed processing of the users.

#4

Jeff,

Maybe if you had a little time you could throw in some thoughts you might have regarding distributing the simulation. Personally I am still having a little trouble getting my head around the idea that the simulation can be driven by something other than what is essentially a “tick” (Yes I know the evil performance destroying tick!).

This weekend I started working on version 2 of my chat application, which is the 3D version. So far what I have done is essentially convert the thread driven graphical simulation into a headless simulation that can be driven by an external event, like a tick. The headless version keeps track of how much time has passed between updates and should update the simulation accordingly when it is time for something to happen.

The most obvious thing to do at this point (at least for me) is to drive the simulation off a PeriodicTask that executes at a specific frequency. For example lets say something like 500ms.

So with this scenario I would have something like:

tick -> -> -> done.

With this approach it seems like (for example) that any time an object requests its position in the world the operation is relatively cheap because at any given time this value is calculated and known. By adding something like event queueing, I can allow the simulation to regulate the rate at which it consumes events generated by clients to make sure that all the processing can be handled inside of one tick. By doing that, I can guarantee that the events for a given object are processed in order, but it seems that it would become more difficult to assure that globally events are processed in the order the arrive. I wonder if that is important…Hmmmmmm… :slight_smile:

I guess an alternative is a more “quantum” approach where the simulation is only updated when something looks at it.

Perhaps something like:

Observer.lookAtSomething() -> -> -> done.

The downside of this seems to be that basically any operation could be expensive because any given operation would require that the entire state of the simulation be re-calculated. To combat that, I could somehow add a limiter which would only perform a re-calc if the previous calculation is more than X age, but in practice that seems like it would end up being no different than just updating the whole thing every X seconds.

#5

Sure, happy to brainstorm with you.

Having said that it would be helpful to know some more details.

As i understand it, this is basically a MUD, right? What information has to be present server side and what info client side?
How significant is spatial position on the server? How much are you worried about cheating?

This is the first mental step I take, determining on a high level the requirements for the client and server with the understanding that client cycles are effectively free but unsecured whereas server cycles are more expensive but have security of state.

#6

Jeff,

The current code is essentially a MUD, but I am basically dumping the MUD-esque portion to convert it to 3d. I was originally just going to convert it from room based to roomless, but I felt the various control options I came up with were rather, um, crappy :slight_smile:

I would say that in my case, the server is absolutely the final word regarding state, including spatial information. Currently, clients can essentially request a state, which is tracked on the server. The options are MovementState or CommandState. In the case of MovementState, clients can basically request to move in a direction or stop. The client would only really be performing simulation by proxy. They could request a movement state (move forward) and go ahead and start rolling, but periodic updates from the server would reposition them.

I would be worried about cheating, so the same story applies there. Clients can request whatever they want (Give box to Jeff) but the server is the arbiter of the transaction (Is Jeff close enough to take the box, etc)

#7

Okay,

Sounds like a “click to move” type system which is definitely easier on the server but arguably a bit less “immersive” then driving controls.

Next question: What kind of obstacles do you expect to have? In other words, what blocks movement?

Just the static geometry of the environment? Moveable objects? Other characters?

Relating to that, is character position going to be tile based or free-positioned?

Lastly, do you expect to try to do any physics modeling or is the only physicality of the world in its movement and
action blocking?

JK

#8

I didn’t intend to convey it was a click to move system. It is just your standard First/3rd person/MMO style movement. The environment would consist of both static (terrain, things fixed to terrain) and dyamic (other players) objects. Positioning would be free, not tile based. Physics would be neat, but that is probably a bit too ambitious for my first round.

I am using jMonkey right now for the 3d portion, so my (completely theoretical untested) plan is to use the scenegraph from jME to render all the geometry information on the Darkstar side.

RE: Movement. I kind of made the assumption that the only way to drive movement in this situation is with a movement state machine, where the client requests movement to begin movement with a specific vector, and the server tracks the state. The client would then have to predict movement and accept position updates from the server.

#9

Don’t.

No.

Don’t

JME will have every node out of the data store at all times, that means you will end up with 1 server processing everything, which defeats the purpose of SGS. Having said that, my collision detection currently does the same thing, but I’m planning on spliting it so that one game area is multuple smaller sections.

Once you do that, you have to handle mutiple scene graphs, or do it all yourself. Multiple scene graphs would cause issues on boundry cases where the object origin is in one small area, but the whole object is not contained in that area, think quad tree division of an area.

HTH

Endolf

#10

What Jeremy said.

A scene graph is a data structure optimized for render, not really for server handling.

Some other issues you will run into. If your doing FPS style “driving” then your going to over-load the server if you try to post every little update. Its also unnecessary. For render you need precise position but for most server calculations your going to be working with simplified data structures and can afford some “slop”. It sounds like you have already been thinking about this. The problem is that, unless your game is newtonian, changes of vector can happen at any time and are high-frequency.

As a general goal for a non strictly-newtonian game, think about sending full data on channels btw clients that can see each other, but send data on a lower resolution to the server. There are a few ways to do this. The simplest is just to send 1 out of every N move packets to the server. What we did in the ill fated BattleTrolls was to send movement packets when one of 2 conditions occurred:

(1) State change of movement (stop, start, turn)
(2) Every N units of rectilinear motion

If this is starting to sound a bit newtonian in language, well, were dealing with motion so its related.

We also angle limited the turn to only send the a turn every N degrees or when you stopped/started turning.

This gave us enough information to build motion line segments on the server to check for cheating v. our collision geometry.

#11

Hehe, yeah, I realize this is generally a bad idea but I’m mostly just trying to get something (even something that is very non-scalable) going as a starter. Personally I was thinking along the lines of splitting up the simulation into smaller scenes and dealing with the boundary issues, which of course is a nasty bit.

#12

So… now that I have an idea of the game. Talking distributed processing.

As much as possible we are trying to do this under the hood for you. The users as they come in are spread around the hosts. As they access ManagedObjects those objects come to them from the Object Store. The production back end tracks users’ object access and channel subscriptions and will move users around to build “affinity groups” in order to minimize our back-end traffic and keep everything responsive.

All this is handled by the system for you. The biggest thing you need to worry about is object contention. Where different tasks contend for write access to the same Managed Objects, those tasks have to serialize. In addition to losing the potential for parallel execution, there is a performance penalty paid for resolving the deadlock.

So to get best parallel performance you want to try to design tasks that wont compete for object write access. Ive got a good example of this in Bunny Hunters which Ill be releasing as source along with the talk on it after JavaOne. Basically, the game processing happens on a per-board tick. The boards thus don’t interfere with each other and can be spread about the back-end. The tricky part was dsiging the Hunter class which is my SessionListener. I had to write it in such a way that updating the incoming player actions didnt conflitc with the game board processing “turns”. I went through 3 iterations til I found one that both performed well o the client and didn’t create contention on the server.

I could try to explain it here but it will be easier to follow once you have the slides to look at…

#13

That is interesting. I had not considered the idea that using physics might actually simplify the simulation. I was operating under the assumption that doing physics on the server side would gobble up all the CPU.

#14

I think that sounds roughly like the idea presented above to break the scene down into smaller pieces (or boards) and handle them separately. Do you make any attempt to synchronize the tick between boards, or does each board proceed with it’s own timer and assume that the other boards will do their own thing? I would guess the tricky part is handing control from one board to another without creating interesting bugs.

#15

Well, this isnt physics really. Just using some common terminology.

But simple Newtonian physics can reduce your work load. The issue is that it only works well for objects that “thrust” like rockets or cars. Creatures don’t really obey newtonian laws in the sense that they are constantly exerting forces to change their direction of motion. If we are moving, we are almost always ‘accelerating’ in the physics sense.

But if you look at the motion of the character if you just hold down the W key, for instance, it looks the same as an object given an initial force and moving in a vacuum-- your moving along a straight line at a constant speed. You just have a very sudden reverse thrust when you pick up your finger that stops motion.

The important thing about that though is that a straight line at a constant speed between a start and a stop point define a line segment. And a line segment is very easy to sanity check against the walk-mesh or whatever other structures you are using to determine movement blocking.

#16

In BHO I don’t have to worry about it because each board is a sepreate 4 person game (think Bomberman.)

However you can certainly see a typical MUD room as a “Zone” (thats really where they came from. MUD room + geometry == Zone.) Each Zone can tick itself separately as long as you don’t have Zone interactions. In point of fact, as an optimization a Zone doesn’t have to tick at all if nothing active is in it.

Typically your MMOs that use Zones break them up with zone-transfers. This is also when they load the zone specific data on the client.

#17

Now to take the Zone idea a step farther…

All you may need is to break things up by Zones, but its possible depending on zone size and density you mayneed sub-zones.

This starts to get tricky,

Observation: Until combat occurs exact inter-character timing is not important.

Ergo: You can let your characters free run around the map. You can have a monster tick to do monster AI if you wish that is zone based, monster type based, or any other organizational unit you’ld like.

When combat occurs you can create a “combat clock” that is only stepping through those characters involved in that combat. This allows many different combats in the same zone to be going on simultaneously and across multiple threads. The trickiness is how you merge combats. If a member of one combat attacks or is attacked by a member of another combat, you’ll need to merge the set of combatants from both into one combat clock.

Note that combat clock combatant sets can be purely based on who attacks who, can be based on party, or based on an area around the combat depending on your specific game design.

#18

I agree. Most of the Diku and derived muds used a Zone to define a collection of rooms, which could usually be brought up and down independently of each other. You are either in Zone A or Zone B and you can’t interact across the border. I don’t think I have ever played an MMO that I would consider truly zoneless.

There are surely a lot of advantages to zoning, and I think a lot of the reasons might be related to content, and managing the amount of textures and models one can have going at one time.

#19

It seems like the sub-zone idea is critical for really allowing processing to be distributed evenly, and maybe the zone itself is important for things like keeping the terrain from becoming super-huge. Without sub-zones you either have situations like a “zone rush” like people would do back in EQ where they would bring down a city zone by invading, or you have to place artificial caps on the number of players because of the fixed amount of processing available. I guess the nasty part is when players find that one spot where they can move back and forth between sub-zones and exploit various issues on the boundaries.

OTOH, it is my understanding that going completely zoneless makes it harder to provide a lot of variety in things like available terrains and models. I suppose to some extent you can handle this by loading and unloading things as needed. Second Life seems to spend like 50% of the time loading and unloading textures and such while you are walking around, and even they appear to have hard zones at points, because occasionally when you port far enough you experience “Loading”.

Do you know how SL handles physics? I am curious how much of that happens server side.

#20

'fraid i don’t know a great deal about second life beyond how they organized their servers.

Some other folks around here might.

But is there actually any physics in second life? Ive only played with it a little but from what I could see there were effectively two modes of motion-- ground rooted walking and free camera type flying. neither of which requires any real physics simulation.