Jeff's 10 Answers to Gordon

General Discussions
#1

Sicne BBB asked for it, I looked it up. here’s my 10 answers to Gordon’s (IMO in some cases naive) 'reasons":

JEFF’S 10 Answers to Gordons 10 Reasons not to do a Massively Multiplayer Game:

#10. Too many are being built. Walton compared the current crop of in-development games to the “RTS frenzy” of a few years ago. It’s a fine genre, but there are just too many in development.

Answer. This is like saying there are too many single player games. The category is huge. There are too many almost IDENTICAL online RPGS being built. That I agree with, but thats a tiny tiny slice of the potential market.

We are seeing so much “me too-ism” mostly because of the difficutly of building good scalable games today. There is a (actually pretty lousy) existing model of a scalable RPG in EQ and everyones basically tryign to copy that one model for fear of failure if they try something new.

As in ANY place in the game industry if you stick to the safe ground you will be in with lots of competiotrs. If you go new places and do new things there are a wealth of new opportunities. As for their difficulty, well thats what we are solving with the Sim Server.

This actually becomes a reason TO do Massively Multiplayer games-- unlike the platform game market there are all kinds of well known genres that havent
even been touched yet. Break out into a new area and you will have 0 competition.

#9. The craft requires mastery of too many disciplines. These include managing a huge team of dozens of people, customer service, community relations, network operations, billing, marketing, and communication and service coherency. Most MMOGs fail in at least two of these crucial areas, Walton supposed.

The answer to this is simply not to try to do it all. A technology like the Sim Server allows non networking or parallel processing savvy engineers to be fully competant massively scalable server programmers. It “knows about” databases and multi-processing and such so the game programmer deosnt have to.

Simialrly rather then trying to do all your own customer service, there are well known (by the rest of the computer industry) third party solutions you can engage.

HOWEVER the Sim Server helps here too. It provides a common back end administration interface to many games at once. It also shares the laod across those games. The result is that one operations center, and thus a single operations center team, can handle ALL the back end administration functions for a whole raft of games.

This makes the epicenter/hosting model a good model for both developers and hosting centers and offloads that responsabiltyifrom the developer. All the back office services, including customer support, can by handled one outsourcer for any number of games-- splitting costs between all clients and taking the load off the developer.

#8. It requires a huge time with multiple, diverse skill sets. These include client, server, database, and Web programming skills and generating gobs of content. Walton said a game that is three times bigger is at least 10 times harder to develop.

Not with the Sim Server. It handles all your persistance. It handles your scalable server design.

You, as the developer, write what appears to be event driven monothreaded code. All your sim objects automatically persist and your code gets automatically scaled out across the entire back end.

The result is deadlock proof, race proof, massively scaled code that is as easy to write as a mono-threaded app.

[continued]

#2

#7. Getting the credit card from the customer is hard. Not all customers have credit cards, and consumers are generally suspicious of online transactions. New customers don’t always fathom the value proposition of an MMOG until they try it.

I don’t see this slowing Sony down. Getting the CC is easy if the user trusts you. People trust Sony so sony ahs no prolbem with it.

This again is where large hosting providers can help. The customer trusts them and they hanbdle the billign for you. In addition, they use existing Sun enterprise billign systems which are already highly secure adding to that sense of trust as well as providing other ways to bill.

The last point is just silly. How many “cancel in 30 days” offers has Gordon recieved. Why does he think they structure things that way? All packaged MM games come with at least a “free” bundled 30 days for exactly that reason.

#6. Online games are completely counterintuitive to packaged-goods game company management. MMOGs are essentially launching all the time with staggered launches and new content being added, rather than centering around a single one-shot launch as packaged software is.

I’m sorry but this is insulting to the industry. Its like saying there is no way a book publisher can publish a magazine. Sure the model is different. Packaged games, like books,are all about acquisition-- the original sale. Online games are exactly like magazines, while acquisition is important. retention is the key to real money.

This is exactly why they are so good for the game industry. In the old model every game has to be marketed, sold and then the process is doen all over. And that marketing is very expensive. Acquisition is the hrad part of any sales mode. With MM online games
once yo uhave acquired subscribers you keep them ,and keep makign money, as long as you put a reasonable amount into keeping the service they paid for interesting to them.

But saying publishers can’t grok this is calling them low grade morons who can only mechanically do what they have done before. This isn’t exactly phd level economics here and there is an existing successful industry to emulate.

#5. Everything developers know from making single-player games is wrong in MMOGs. Well known formulas of discovery and secrets don’t apply to online communities, and cheats that in a single-player game affect only the player who chooses to use one can ruin the experience of hundreds and even thousands of paying subscribers in a persistent world game. Walton also pointed out the importance of documentation and maintenance issues to MMOGs that often fall by the wayside in single-player-game development.

The last one was insulting to people who publish games, this one is insulting to those who design and develop them.

“Well known formulas” (formulae, properly) are a road straight to chapter 11 in ANY entertainment medium. People crave new experience and the only reliable “formula” there has ever been in entertainment is “be different.”

Good game deisgners and developers are pushing the envelope daily in their work-- thats their job. Sure there are new challenges in the MM space but thats what makes it interesting and fun! And the fact that so much design hasn’t been done yet means theres a ton of low hanging fruit before we get to the “oh god what do we do THIS time” phase.

[continued]

#3

#4. The Internet sucks as a commercial delivery platform. Not only that, when players have a bad Internet experience, whatever the reason, they blame the game providers.

Frankly, this sounds like a bad artist blaming his medium.

All new game paltforms bring with them new challenges and limitations. Designing around and for them is the essence of real game design.

I haven’t noticed Sony suffering for the problems of the medium. or Bioware. Are there new challenegs?Sure. Thats what we are paid for-- to solve them.

#3. Customer service is hard. Walton cited customer service as the single biggest cost variable in online game development, and the ramifications of the customer service strategy and project planning are far-reaching. Walton pointed out that whereas in most traditional businesses customer service is a cost center whose expense is to be minimized (like the call center you phone to complain about your cable bill), in MMOGs it is essentially the entire business. And that 24x7x365 business is extremely people-intensive, which by definition is costly and messy.

Okay a few comments. First one is this:
" in MMOGs it (the call cnter) is essentially the entire business."

To be blunt: Only if you have a crapy service. The vast majority of the time the customer should be happily using your service. If they arent then thwres something very wrong with it.

If telcos operated like MMOLRPGs today do then yes, their call center woudl be flooded with complaints/questions/issues. but they don’t. They deliver the service reliable, as promised and as expected.

Quality of Service on todays MM games is terrible, there is no doubt. Thats someplace the Sun Sim Server halps. It provides the tools necessary to allow game developer to reach the “5 nines” that telcos tlak about-- where the service is up and functioning 99.999% of the time. We do that for the telcos, we’re going to do it for online game services.

Once you are AT the telco stage the question is how do you minimize the cost of even that? Well again the telcos (who Sun is very familair with and works intensely with today) have some standard answers. The first is automation. As much as possible is handled by computer systems. On the second line, they go to ecomonies of scale in large call centers. In many cases they actually outsource these call centers to companies that specialize in 24x7x365 service. All you need to do this is a scale of usership thats appropriate.

AND again thats where the Sim Server comes in, by making the game hosting epicenter possible. With a single administration team and call center servicing all games installed in the call center you can share expenses with all the other game publishers uisng that provider. The hosting provider handles user accounts, customer service and the rest and you pay a small amount per account as “your share.”

#2. There are lots of legal issues. These issues range from terms-of-service contracts to end user license agreements, frivolous lawsuits, the commonplace use of “volunteers” to help administer the game, IP protection, and the question of legal ownership of virtual “property.” All these laws and regulations are in constant flux, which put legal issues so high up on Walton’s list. His advice? Get good lawyers and be sure to budget to protect your IP.

Where has he been? Certainly not in the US.

Anyone who runs an IP based business without an IP and contract lawyer is 100% certifiably insane. Any packaged game developer who signs publishers’ contracts without a lawyer’s assistance will be out of business within a year.

My parents have had a 2 person IP company (print text and photography) for 40 years. I grew up knowing the name of their lawyer as well as I knew the names of any of their friends.

This is a non-issue because anyone in a real business already has to deal with this, MM or not. Where it comes to the specifics of TOS and billinf contracts, again a service provider who handles that for you will have standard stuff of their own from their pet shark.

[continued]

#4

and last but not least…

And… Gordon Walton’s #1 reason You Don’t Want to Make a Massively Multiplayer Game…
#1. They cost too much money to build and launch! This of course is the ultimate gotcha that turns the best laid plans of mice and game developers to very costly muck. Development costs continue to rise, and, in Walton’s words, “the faster you go, the slower you get there.”

Today, he’s right. To go online for massively multiplayer you need to build a machine room designed to handle your maximum expected load. Thats very expensive.

We put a million dollars into our machine room at TEN before we even put in any computers. (Switches, racks, climate control, telco equiptment, fire supression, statid supression, power back up, etc).

Even if you cheat on that stuff (which will come back to haunt you) you still have to build out a machine back end itself capable of handling your maximal load. Failure to do so can result in the “success disaster” wher you get too many users to handle, your system goes down in flames, and you gain a permenant reputation as a crappy game.

Whats worse, that means a lot of equiptemnt sitting ideal adding to your ongoing operating costs. Try to do fail-over by simple replication and youve just doubled that cost. Fill it up past expected maximal capacity and you are in shard-ville and have another set of mostly unused hardware again.

The Sim Sever addresses this on two levels. To begin with, you can invest in a minimal set of hardware. Because it scales symetrically, handling additional load is as simple as calling up your Sun rep and slapping in some more blades as your user base expands.

More so, when deployed in a hosted environment you can start by using a fraction of a blade. You pay the hoster a micro-payment per user account. As your user base grows you automatically user more resources and pay him for those new accounts. Your costs scale as your incoem scales and you can make money from the very first day. In fact, you can make as much money from 5 games that only ever reach 500 accounts as you would from one game that has 2500 accounts.

As I may have mentioned we already have a very very major outsourced computing resources supplier (one of the biggest in the world but I can’t name names) who wants to do just this-- encourage a market of hundreds of niche games rather then juat a few big ones. The ones of those that take off and become huge, thats gravy for everyone involved.

So in short, which of Gordon’s 10 does the Sim Server adress? My answer is every one thats real!

#5

Thanks for the very detailed responses, Jeff. I’m afraid I haven’t got time to do your comments the justice they deserve :(.

Remember that Gordon is no clueless fool; he ran two of the Ultima Online projects before running The Sims Online.

The main thoughts that came to mind whilst reading:

[] You come across as a techie with little interest in accountancy, finance, etc. You discount (or, in at least one case there, completely ignore) wider commercial and financial issues. You also seem to have little appreciation of the key drivers for the business of anyone developing and or running an MMOG - you only see the technical problems.
[] One of the few of Gordon’s reasons that I agree with strongly is the point about customer service. That you write this off as being a sign that his game was “a crapy [sic] service” underlines the above point.
[*] It sounds like you’re system is competing very closely with Butterfly, in terms of the sales benefits. I will be particularly interested to see if your technology can compete head-on with them, given their huge first-mover advantage.

#6

Hmm. your questions readvery very general and vauge to me.

As much of my post deals with a new and better financial model for doing MMO development and deployments that this system makes possible, I’d have to disagree with you on the financial point.

On your second, I think your vision is clouded or incomplete. MM QOS IS crappy today. Its that simple. How would you react if your phone failed as often as your online game? Think about it. Customers don’t call CS if they don’t have a problem.

I’m not worried abotu Butterfly. Butterfly is a grid. grid is wrong for this app space as it only works well for problems that can be divided up into cells that communicate only to their neighbors.

This works pretty well for the “assign CPu to space” model but that mdoel is exactly whats wrong with today’s backends as it causes large scale CPu wastage ina reas thata re nto in use and in anadquate cocnertatio nof reources where they are needed. They also can’t fail=over except by wasting mroe resources to run duplicate nodes. Finally, AFAIK they cannot sahre a node among different games.

In short Grid has none of the advantages over the current models that our system does.

#7

[quote] How would you react if your phone failed as often as your online game? Think about it. Customers don’t call CS if they don’t have a problem.
[/quote]
While telecos have very advanced software/equipment for real-time/heavy usage, their area of interest is a lot better defined than MMORPG. As long as I want to create a game where two or more people can talk together while display IDs of each other (in short, text-only chat), then I can aim for being bug-free. Everything over that is bound to fail - and no amount of server technology is going to help here, given the problems with client machines alone (various configs, viruses, hardware failures, clueless users, etc). Then add political/social problems (griefers, lost passwords, etc) and call center is going to be big part of cost even assuming that everything on server side is perfect.

Just to pull your leg: I’m not suprised that people are not calling their phone providers when their phone is dead… Same probably is true for sending emails to admin about broken mail server… :wink:

#8

True but guess what? Phone systems fail too. In fact
we had a phone rack catch on fire in our machine room at TEN. Quite spectacular.

Five nines isn’t about equitpment, five nines is about systems. Noone knew our phone rack ate istelf at TEN because we had a second phone rack in place. Similarly game servers WILL go down, but since the system allows for instant reconnection to a new server stack noone need ever know abotu the failure.

So let me say this. Remember I worked for TEN. We were an ongoign successful on-line service. We handled thsi just fine as a start-up with (for a start-up) a relatively low cash-burn. As i alluded to before we oursourced 98% of our customer service to a copmapny that handled it very well and very efficiently. We had an expert team of I think 5 people as backup available 9 to 5. With that setup we serviced a user population upwards of 30,000 users and never had problems.

Now we also had a very reliable service. And we had volunteer folks in the online community who would help others with day to day issues like “why can’t I launch TopGun??” or what have you.

So I just don’t believe this is all that big a problem IF you don’t have a system that by its nature generates constant suer problems. The same is true of yoru games BTW. IMHO it is your job as game designed to design griefless games. If you create lots of ways for players to harass each other then yes you will deal witha lot of harassment complaints. This is something most online games though have already figured out.

So I know your kidding and yes its a funny thought… but not really true anymore what with cell phones and such.

#9

Ok, I’m not a gamedev(yet), just a newless clubie, but I thought I’d share some of my ideas on the MMORPG craze.

He brings up the point that there are too many, and you reiterate by saying that it’s like having too many single player games. Unfortuneately, that’s not quite the case. Keep in mind the point you made about MMO games trying to attract and keep subscribers, so you get the most profit out of the system. With a single player game, you’re looking at 20-80 hours of gameplay, averaging around 40. A good player (or a person using cheats), can beat such games in let’s say…75% of the estimated time, maybe less. Anyway, the point is, you play a single player game, you beat it, and if it’s good, you might beat it again, but people can shelve a game they’ve finished and move on to another.

This is not so with MMOs. You want players to invest their time in the game with some regulartity. That means you need a stream of new content, new options, and new fun to keep players interested. What I see wrong with my MMOs is the way they do this–repetitive tasks. I don’t know, for some people, watching your avatar slay monsters for you might be real exciting. I get a kick out of it for about an hour. But to spend a year or more doing that?

Now, if this engine/game you’re talking about (sorry, haven’t read much on Sim Server yet) is out to fix this problem, then you might have a winner. But right now, I have to agree that the market just does not need another game where you sit and talk to people while your character does some dull task for you. Let’s say that the rule is, if you think “social interaction” is a feature because you put a chat system in, then you need to make sure you have enough fun, immersing gameplay that players don’t have to resort to roleplaying over the chat system alone.

#10

Well, most people assume that all MMOGs target the same audience, all need to hit the “magical” 100,000 player subscriptions, and everyone feels the same way they do about how the games can be played. So let’s examine each of these in turn.

  1. I doubt, very much, that the same people who played Motor City Online are the same people who play DAoC are the same people who play There.Com. The market for MMOGs is far and away beyond the dungeon crawl, fantasy based genra of games. In this sense, you can, and should expect, there to be hundereds of MMOGs in the future.

  2. Now, how do all of them survive? Surely not all of the mwill hit a 100,000 player mark to remain viable. And why are they not viable under that mark? Server infrastructure to support the games are too expensive and creating content for 75,000 people is a lot of work. Ok. Lets say these assumptions are true. What if you had to create content that would only serve 10,000 players? Less work from a brute force perspecitve (people), less customer service, etc. But infrastructure is still huge because you need to build for CAPACITY, not for what you will actually use. This is where the compute utility model comes into play. A service where the game is hosted and all game related services are doen on a usage basis, not an infrastructure basis. :slight_smile:

  3. So, not everyone likes to do the same tasks over and over (running on the treadmill as Raph calls it), but some do. MMOG is still new and everyone is copying everyone elses examples. Gameplay wil evolve over time and become better as new servcies become available. But remember, even in the VAST amount of games released to the market, most fall into 5 catagories and share 75% of the same mechanics :slight_smile:

-Chris

#11

Ah, you got me. I totally disregarded the MMOs that aren’t fantasy genre, and I really meant not to do that. Game like Planetside and Puzzle Pirates offer an interesting appeal in their own rights for fans of their respective genres.

One point that Planetside brings to mind is that thing about the “magic number” of players. An MMO should definately offer plenty for the “soloist” player to do. Many upstart games (that I"ve seen) seem to think that give players a big playground will draw people in for the amount of interaction and socializing that there will be. I can’t say I’m above that design, since it does sound fun. The downside is that when you don’t get enough players to make it fun, then you lose players. It’s a slippery-slope problem.

Anyway, I’m not trying to put down MMOs or downplay their importance. They are here, they are popular, and there will be more of them for sure. I’mjust saying, and I think everyone will agree with me, that they need variety in their gameplay and creativity to attract new players. All these EQ “Me-Too” games are cluttering up what will eventually be a fun and unique new genre.

#12

[quote]All these EQ “Me-Too” games are cluttering up what will eventually be a fun and unique new genre.
[/quote]
Not to you in particular, but I’m having difficulty understanding why so many people feel the need to say that?

How is this any different from the last 15 years of mainstream games development? Why is it even worth drawing breath to say it?

#13

Out of interest, which people are you referring to?

For instance, most MMOG developers don’t think this (I’ve asked them). Obviously, it would be a major (expensive) task to ask every mainstream developer too, but from those I’ve seen, spoken to, or listened to (e.g. at conferences), a fairly large proportion of them don’t believe it either (although there are good reasons for pretending that they do, especially at trade shows ;)).

No need to doubt, the figures (and even some of the statistical analysis) are freely available. (and the answer is not trivial; it depends largely upon what period after the launch dates you are examining, but for some particular pairs - e.g. there vs daoc - I believe you’re spot-on (IIRC))

As I’ve said before (and so have many of the people you’d like to sell to, if only you’d listen to them), the first is irrelevant, and the second is only a part of a larger problem (although it is a major part).

For instance, in general terms, it’s often cheaper per-player from a hardware perspective to support fewer players: pricing for 30-way Alpha clusters with dedicated Gbps bandwidth is considerably less commoditized than 3 intel boxes and a bog-standard b/w contract…

#14

I’d say people always have problems when new genres/gimmicks appear or come back into popularity and everyone starts copying them.

Mind you, not innovating or improving in any dramatic way. Just pretty much blatantly copying them. The fact that after 15 years or so it’s still common practice just annoys people I guess. It’s the same in any industry. Look at movies right now, comic books are hot and so every graphic novel in existance is getting a movie. Eventually, people will get tired of them or one will bomb or something new will come along and it will fall to the side.

But my annoying ranting has drawn this thread way off topic hasn’t it? I appologize.

#15

Here are the slides from the HIGHLY respected industry veteran Jessica Mulligan from her presentation at the Austin Game Developers Conference:

http://www.gameconference.com/conference/jessicamulligan.ppt

I don’t think she is pretending :wink:

[quote]No need to doubt, the figures (and even some of the statistical analysis) are freely available. (and the answer is not trivial; it depends largely upon what period after the launch dates you are examining, but for some particular pairs - e.g. there vs daoc - I believe you’re spot-on (IIRC))
[/quote]
Agreed. The issue is not wether or not people enjoy playing online, but rather, the genre the like to play. Now, if there was a roadsters online out there when MCO was out, you might have some crossover. Not necessarily, but possibly.

[quote]As I’ve said before (and so have many of the people you’d like to sell to, if only you’d listen to them), the first is irrelevant, and the second is only a part of a larger problem (although it is a major part).

For instance, in general terms, it’s often cheaper per-player from a hardware perspective to support fewer players: pricing for 30-way Alpha clusters with dedicated Gbps bandwidth is considerably less commoditized than 3 intel boxes and a bog-standard b/w contract…
[/quote]
Who said anything about selling 30 way ANYTHING? The solution we demonstrated at GDC allows them to add capacity, on demand, one $1200(list price) blade server at a time. And the solution scales to accomodate more than one game at the same time allowing the developer or ISP to utilize all available resources at their disposal. We HAVE been talking to developers, and the response from them has been positive because they see that we have been trying to solve the problems they told us they had (along with some nifty kit to boot ;))

Oh yeah, and the cheap-o 3 intel boxes model? Surefire way to shoot youself in the foot from a stability and CS POV.

-ChrisM

#16

[quote]Oh yeah, and the cheap-o 3 intel boxes model? Surefire way to shoot youself in the foot from a stability and CS POV.
[/quote]
:slight_smile: I must admit, I’ve never seen an Intel box that actually works. Seriously. Every one I’ve use had a glitch or dysfunctional hardware interface of some sort. Even the expensive ones :).

#17

I don’t have access to powerpoint until I get my workstation back, so please could you post the relevant text?

You talked about “Server infrastructure to support the games are too expensive” and games being “not viable under that mark” (75k players); I thought you were saying that this was because the hardware cost was too high unless you could amortize it over > 75k players. Did I misunderstand you?

I was just illustrating how hardware costs do NOT force games to have lots of players to cover their costs, because smaller games may easily have disproportionately lower hardware costs (I’ve seen the figures for quite a few games that run on 1-4 intel servers, and it’s usually extremely low, even factoring in hardware failure/support).

#18

Directly from Jessica’s slides:

Average Annual Subscribers: 100,000
Monthly Rate: $12.95
Monthly Gross: $1,295,000
Annual Gross: $15,540,000

She used these starting points as the “typical” target for a successful MMOG in the space.

[quote]You talked about “Server infrastructure to support the games are too expensive” and games being “not viable under that mark” (75k players); I thought you were saying that this was because the hardware cost was too high unless you could amortize it over > 75k players. Did I misunderstand you?

I was just illustrating how hardware costs do NOT force games to have lots of players to cover their costs, because smaller games may easily have disproportionately lower hardware costs (I’ve seen the figures for quite a few games that run on 1-4 intel servers, and it’s usually extremely low, even factoring in hardware failure/support).
[/quote]
This depends on what you want to target, of course. However, starting with an infrastructure that does not scale is a nightmare. EQ was designed to grow to handle 75,000 players over the first 1 1/2 years of its life. They got 150,000 players inthe first 6 months. Their environment did not scale and they were left with a tremendous problem. On the other side of the equation, you have licenses that expect to have huge playerships so you plan and build for scale (TSO is one good example). So the company throws a ton of money at the problem to build for capacity and what happens when they don’t hit the magic number? Left with a huge hulking set of resources that they still have to pay for.

A solution that starts as small as you like but able to quickly, and on the fly, scale to meet demand is what we are building. Sun has been in that business for over 20 years :slight_smile:

-ChrisM

#19

Is it stated whether that is a historical analysis or a strategic one? The former is obviously true; however I’ve had emails from plenty of MMOG devs (including e.g. SOE staff) confirming that they don’t believe it strategically (“any more”, in several cases; many of them used to).

I agree completely. But you seemed to be saying that server infrastructure was a major cost that prevented small scale games from being viable, which is a different concept entirely; sorry for misunderstanding.

#20

No problem :slight_smile: Now that was nice ;D

-ChrisM