Managing ManagedObjects

Networking & Multiplayer
#1

I’ve just started to play around with Darkstar and have been really impressed so far. I have some questions as to how to handle ManagedObjects during a standard development lifecycle.

So let’s say I have my first AppListener object. I make it’s initialize method and create a channel object. This works fine for my application but a week later I realize I want my AppListener to create two channels instead of one on initialize. I update my code and all looks well, except that initialize will never been called again during the lifetime of my existing AppListener. It seems in order to “upgrade” my server, I have to delete the database and start from scratch.

So my question is, in a live development environment with lot’s of parallel developmenet updating ManagedObjects it is necessary to have an upgrade path for your persistance data. In real world projects I’ve worked on where the persistance layer was a standard DB (such as postgres), this was often done with scripts that would upgrade the DB schema to the latest version so that the latest code could read & write persisted objects into it.

I’m sure you’ve thought about this issue, I’m just wondering what the suggested method of handling this is?

Thanks!

#2

Yup. we’re doing a lot of thinking about both sides of that issue.

(How to recreate individual objects and how to avoid having to recreate all the objects when one changes in a serialization incompatible way.)

I’d love to see a wish list of behaviors and any other suggestions you have…

Thanks

JK

#3

To me the server should automattically drop anything in which the serialversionId doesn’t match without erroring. Thus when new versions of the app are deployed previously existing objects would be discarded.

Additionally, in a production server one would imagine there owuld be a startup/shutdown. Maybe objects should live only as long as the server remains running. When it is properly shutdown (aka other than a crash) maybe a destroy() method or something like that on the AppListener should be called allowing it to cleanup how it wants to clean up.

Finally, in a production environment there should be someway of deploying an application. Maybe as part of that deployment process here could be some indicator as to whether or not the objects in the datastore should be flushed or not?

Just some ideas… Although a huge cluster (in which you might upgrade one at a time while the others remained running) could pose an issue with these ideas… Maybe the persistant store should store its objects based on their version and the version a particular node is requesting?? Just an idea…

#4

I definately don’t like the idea of dropping anything in which the servialVersionID doesn’t match. I think this is much worse than an exception, it like deleting possibly needed data without so much as a warning.

What is needed is a safe, simple and easy way to translate from older versions to newer versions. Maybe a way to catch this exception and handle it?

This type of translation could handle converting old object types into new object types and deleting old types that are no longer needed.

In previous applications I’ve written where they had to maintain backwards compatibility with prior serialized data (this was in C++), I would write translations between versions. I might have a translator from version 0 to 1, from 1 to 2, and from 2 to 3. So If I loaded version 2 I could translate into version 3. Even if I loaded a version 0, I could go through the chain of translators until I reached the needed version. The problem is these translations are not automatic. They often require a decision from the developer on how to generate default values or convert types to new types.

That being said, I don’t have any idea how to implement such a thing.

#5

These are all really interesting and useful ideas, its got my wheels spinning :slight_smile:

Don’t stop…

#6

Well Serialization kinda allows you to do that. You just have to implement the writeObject readObject methods. Of course you would need to have defaults for certain fields and your read/write code could get fairly complicated depending on the complexity of your object and possibly its children objects.

But both Externalizable and Serializable have a means to do this on a class by class basis. But it seems to me that maybe a better way to do this would be to use more of a JDO system rather than a pure serialization model.

I mean are we using serialization to store persistant data (aka data that should be stored in a database in a traditional way?) Or are we using it to store short lived objects for failover purposes and clustered access? If the former maybe their in lies the problem and the solution is to use a JDO model as opposed to a serialization model…

Just a thought…

#7

Well I think it breaks Darkstar’s transactional safety if you aren’t using ManagedObjects for your persitance. ManagedObject are definately not short lived objects, since they exists for the lifetime of your application.

I wonder if it would be possible to write some sort of helper class that would serialize/deserialize a standard object for you, but would also detect version changes and aid in translating between old and new versions. While I really do love the simplicity of not having to define the serialization code, I think in a true production environment something like this would be needed anyway, with Java serialization being a bit heavy most of the time anyway.

#8

Actually we are getting fine performance out of Serialization. Its a long way from being our data bottleneck.

Java Serialization only gets heavy when its abused… serializing out large graphs at once. The ManagedObject/ManagedReference system makes that fairly unlikely to happen and if it does, you probably need to be breaking your data up into more discrete ManagedObjects anyway.

As for life-cycle, ManagedObjects are short lived in memory, but long lived in the ObejctStore.

The reason we dont use JDO, or some of the other OR mapping techniques, is three fold:

(1) We don’t wish to burden the developer with either pre-processing their code or defining and maintaining an explicit OR schema.

(2) We need more control over the actual process of data storage then many systems would allow us.

(3) None of the implementations we know of have the right speed and scaling characteristics. (Blindingly fast access, massive scaling over an equal number of reads and writes. That last part is the big gotcha.)

I’d like to turn this discussion away from the “how we do it” space its drifting into and back into the “what it does” it started at because thats a whole lot more useful to me.

Thanks

JK

#9

Just spitballing it here, but if the server finds itself in a situation where there is a class version error, instead of barfing an exception it could look for a registered “translator” object. If there was no translator then it could go ahead and barf. This way even if you renamed the class there would be a way to migrate it forward. I imagine writing the new readObject() methods would be kind of a pain, but doable.

#10

So going back to “what it does”, I can think of a variety of things I’d like to see in one form or another (or at least a reason why I don’t need them :slight_smile: )

Re-factoring - Often during a project, we might re-factor our code simply causing our managed objects to change in a way that doesn’t really affect the underlying data of the object.

Transforming - Adding/Deleting data, changes to data types, and other such transformations are quite common during the development life-cycle.

Querying - In a way I see this as a related issue. In dealing with Managed objects it would be nice to have other ways of getting them besides a named lookup. Maybe a way to iterate through a sequence by class or some other arbitrary properties.

While I really like the goal of not burdening the developer with pre-processing code or maintaining explicit OR schema having the option for more control over managed objects is going to be more and more important as people start digging into Darkstar. As for myself, I’m going to start digging more into Darkstar and see what issues I run into as I approach it for the first time. It should be fun!

#11

Hmm. This is an interesting notion… having a “versioning” class for each version that knows the translation…

#12

One thing I might suggest is a shutdown() method on the AppListener that is called when the server is shutdown properly and or if the app is shutdown properly if there is such as thing… This would allow game developers to do some things for themselves if their application allows it

Another possible change would be to add a boolean to initialize on the AppListener. It would be true if the object was being loaded for the first time and false if the object had been loaded from the store. Obviously if this was the case then initialize would need to be called everytime it was loaded. Anyway, this could allow the upgraded version per-sey to go through its objects and do something useful…

#13

can’t step 1 be taken over by darkstar; if it where to cache the old class signature?(that way it can still deserialise the old data) iteresting that the schema of your datastore is starting to look like a bunch of class signatures. the problem with the version class, what are you going to use as a method signature? I’d hate to abuse the package- or classname for that, as that would make it tedious imo.

now that I’ve finished typing this message other stuff came to mind. if darkstar where to cache it you should be able to backup that cache. (moving to a new datacenter… etc)

how are you going to bind the managed object to the versioning/tranformation class? annotations? perhaps you can do something funky with an annotation pointing to the older class signature. that would require ide support to not become something annoying to do. (considering how spring ide is crawling going down the road of something that would need ide support.) this is all pritty tricky stuff.

#14

Yes, I was kinda thinking along the lines of registering a class with the Datamanager.



   public interface ManagedObjectTranslator
   {
         public Object translateObject(ObjectInputStream in)
   }

   AppContext.getDataManager.registerObjectTranslator("com.foo.CoolManagedObject", new CoolTranslator());

   or 

   AppContext.getDataManager.registerObjectTranslator(CoolManagedObject.class, new CoolTranslator());

Then, when you deserialize at startup, and there is a problem the datamanager could pass the OIS to the translator and cross it’s fingers.

#15

Your idea is interesting. But the registration wuld have to occur in the startup properties otherwise there would be no way to translate for the AppListener itself (aka the first object loaded before any programatic registrations could occur)…

I’m not so sure I like the idea of the AppListener being a managed object. Seems it would be more useful if it were constructed and initialized every time the server/app was started. Of course it wouldn’t be useful to store state in if that was the case…

#16

I was thinking about this as well. I think there are some events we need to be aware of, such as cluster startup/shutdown. For a real world example, in another project our game server broadcasts it’s status to a webserver. It tells the server when it goes up, when it comes down, and while it’s running it periodically broadcast it’s state information. This data is displayed to users.

Another use I’ve seen the startup event be used for is reporting user entry point addresses. Instead of having a static address for clients to connect to, the server generates a connection string that may contain more than one entry point. This is given to an external web server and the client takes that value and uses it to connect to the game server. I’m not sure exactly how Darkstar handles client entry points but in a large scale environment, it seems having more than one connection point would be a good idea. Sorry, I think this is getting a bit off topic, just an idea.

Events I could see as being useful:

  • gameAppStartup(boolean gracefulShutdown, Set clientEntryPoints); // Called on startup and the flag tells us if we shutdown gracefully and where the clients should connect
  • gameAppShutdown(); // Called on graceful shutdown
#17

Good point about the AppListener, It would certainly need to happen before it was loaded. I suppose they would need to add some sort of new startup hook to allow programmatic access.

#18

We considered this as well, but here is the problem: what if you trip on all the power cables in your cluster at once? What happens at restart in that case?

The good news is that the SGS allows you to add any functionality you want to your application object. For example, you can create a special “management” client to poke your objects to tell them about certain events, or (a little slicker) create a service that your application objects can register with as listeners for those events.

The key for the SGS core design is to support a lean, scalable set of primitives that you can extend to do anything you need – and by all means, design that extra functionality as a utility that anyone can use on top of the SGS! :wink:

#19

I think I just put a bunch of developer wish-list items in another thread “n00b Qs”, but here they are as official wish-list items:

  • A way to ask the data-store for every managed object about which it knows. (Facilitates various maintenance.)

    • A way to ask the DS for every managed object of class Foo about which it knows (to facilitate migration.)

  • An API (in AppListener, I think) to notify my code that the AppListener has been restarted. This is IN ADDITION TO the current initialize() method – probably something like AppListener.startup() (called after initialize(), on the 1st run.)

  • An API I can call to gracefully shut-down my AppListener and/or the SGS server. Something to allow me to programatically start a squence where I no loger accept logins, and tell users “The server will be going down in 60/30/15/10/5 seconds”, then boot them all & shut down.

  • A framework that allows source-level debugging (as with XCode) for my AppListener. I think that SGS’s thread-model complicates this, so I guess I’d like a single-threaded box that just lets me step through my code, examine values, etc.

Thanks!