I’d like to know, if when building a program that will probably eventually need the horse power of several computers, what you have to do to work it on a server farm basis. Is there software out there so that you setup a server farm, and to software running on that farm it sees the entire farm as a single computer, or do I, as a programmer, have to setup my program to be distributed over that farm for example, using RMI calls so I can split up the application into several parts (like moving some expensive utility functions like looking up a specific object in an arraylist that could get very large and is frequently accessed) etc. Is there a good reference from a programmers perspective in how to take advantage of a server farm, or is it fairly automatic and handled by the operating system or some other software?
It’s going to need to be software supported and alot depends what type of game you’re trying to write. Project Darkstar seems to support this concept pretty well, except that the Object Store must be contained on a single server. However, it’s understandable why this limitation was necessary. If you’re planning on doing a massively multiplayer game then Darkstar is probably going to be the quickest solution to your problem.
-Matt Hicks
Darkstar doesn’t have the object store contained on one server. All the servers in the cluster can access the store. The store itself is distributed across the cluster too if I understand Jeffs comments correctly.
Project darkstar does sound exactly the sort of thing you need. The only negative thing I’ve seen and agreed with so far is that it doesn’t allow you to do complex queries on the data, but I believe this is being looked into.
Endolf
If you’re able to work within the structure of DarkStar, I would STRONGLY suggest you try to go that route. Coming from a background in my “real life” of supporting and now architecting large scale J2EE environments, I cannot stress enough how powerful it is to have an application server platform that is specifically designed to handle the raw structure for you and allow you to focus your efforts on the game itself. Scalability, failover, load balancing, etc are all important factors that can very easily be done wrong. And any time you, as a developer, spend in writting, testing and debugging those capabilities is time you are NOT spending writting your game.
Of course, it does really depend on the type of game you are doing and some possible rethinking on how you implement. But it should be well worth your time to look into it.
From my reading it was my understanding that only one Object Store could exist per application, and though could be shared by multiple GLEs it was a single point of entry.
Can anyone tell me for sure whether that is an incorrect assumption?
-Matt Hicks
That is correct. The SDK purposefully limited to run only on a single stack, but the real production system (or “big backend” as we call it in the project) has a fully destributed Object Syore and any machine has full access to it.
And yes, we’re looking at the queries and data mining issues 8)
This is true that tehre is one ObjectStore per application. It is acessible however from N machines who are cooperating in executing that application.
Okay, that explains better, thank you. The way that the diagram in the server tutorial for SGS is laid out makes that a little less clear. From appearances it would seem that the ObjectStore could be accessed from multiple servers, but must only exist on one server. However, the mention of “cluster” in that should have been a tip to me otherwise. 
I will be writing a sample application very soon to get some real hands-on SGS, so perhaps my ignorance will lessen then. 
-Matt Hicks
Yeah that diagram is a logical diagram, not a machine architecture lay out.
The actual machine room in the “big back end” is a good deal more complex.
Pictures!, we want pictures!, lots of flashy lights and shiney toys.
Mmmmm, toys.
Endolf
P.S. No, I won’t grow up :).
I wasn’t aware of DarkStar when I started the project, and at this point I don’t know that it’d be benificial to try to adapt to it. I will have to spend some time researching it. I think I can get a long ways before I have to worry to much about anything other than the datastore in terms of performance, but I hate to get to the point of needing to know and not know. I’ll have to dedicate a little time to making maybe a really small scale game with it to see if it has potential, and see if I can reasonably move my current game over as I fully expect to need more than a single machine handling clients at some point, though I expect the data storage to be the first issue and I believe that generally most db’s (at least commercial ones) can be setup to work on a farm pretty easily.
[edit] I am working on a MMORPG, if I was working on something like diablo 2 where the games are small load balancing wouldn’t be an issue, something like that is pretty easy to seperate as each game is small enough to easily run a lot of copies and the data between games is not important… [/edit]
[edit2]After reading some of the docs I’m pretty much sold, it looks like other than the time I spend learning the API I won’t lose much of my current project… now… for how long will it be free…[/edit2]
The SDK will remain free, I can almost gaurantee. (I give any gaurantees and the lawyers will be up my — before you can say boo.)
Hosting for commercial purposes will almost certainly NOT be free. We do however want to make it highly accessible and affordable for people without huge budgets (garage shops and other independant type developers.)
Non Commerical use, particaulrly where the users is using their own equiptment such as unversity labs, is something we are very interested in makign possible.
But the final busines model is still in management so until thats done I can’t make definite precise statements on any of this, sorry.