Model design strategies

[Jan]

Hi

I’m having a look to SGS to check if it is able to easily handle large amounts of server-side data. Related to this I’m wondering a number of things. If someone can help me out with them … ;D

• The server tutorial mentions data that wasn’t removed from the object store will not be collected. (Page 3, last paragraph.) Does this mean the total amount of data in the object store is limited to the JVM’s heap size ? Or can it handle anything fitting on the local file system with caching/pooling ? If not, would something like JPA (or even Java EE) be a solution ?
• I would like to implement a kind of Observer pattern where clients subscribe to changes on the server-side model. To do this it would be very handy to be able to catch changes of managed objects - in order to notify the clients a particular object has been modified. Is there a usual design for this in SGS ? Presumably these notifications must be using the server/client messaging mechanism. Correct ?
• I read in this forum the max message size is 64K for messages sent to the clients. Is this also the case for messages sent to the server ?
• Looking for answers I saw a topic about initial model upload to the clients. As my model is way too big to send regularly completely to the clients, I also though about an initial load. Making it only slow at login time. But I would really like to avoid an additional server-side setup, configuration and code development to allow HTTP too. Breaking-up the data into 64K pieces is not difficult. Alas, the resulting number of network requests alone will impact performance. Any suggestion ?

Many Thanks for helping me !

Jan

[Hoyle]

Just some of my thoughts . . .

@Jan

I’m having a look to SGS to check if it is able to easily handle large amounts of server-side data. Related to this I’m wondering a number of things. If someone can help me out with them … ;D

When you say “large amounts of server-side data”, are you meaning static content data or persistance data that changes (as in ManagedObjects)?

@Jan

• I would like to implement a kind of Observer pattern where clients subscribe to changes on the server-side model. To do this it would be very handy to be able to catch changes of managed objects - in order to notify the clients a particular object has been modified. Is there a usual design for this in SGS ? Presumably these notifications must be using the server/client messaging mechanism. Correct ?

Remember, SGS is client agnostic, so any type of notification system that SGS ever provided would need to be as well. It might be easier to wrap your server data in proxy objects that are also ManagedObjects. These proxies could then perform the notification for you when set methods were called, in whatever manner you see fit.

In my experience it’s often bad to wholesale replicate data from the server to the client, often much of the state that the server knows about isn’t required within the client.

@Jan

• Looking for answers I saw a topic about initial model upload to the clients. As my model is way too big to send regularly completely to the clients, I also though about an initial load. Making it only slow at login time. But I would really like to avoid an additional server-side setup, configuration and code development to allow HTTP too. Breaking-up the data into 64K pieces is not difficult. Alas, the resulting number of network requests alone will impact performance. Any suggestion ?

You don’t have to have the client make tons of 64K chunk requests to the server. Simply have the client send a single message to the server telling it what it wants to download, and then the server can send data back to the client in 64K chunks. Just make your client be able to recieve the data in chunked form and piece it back together. This seems pretty easy with reliable ordered delivery.

[Jan]

Thank for replying Hoyle !

@Hoyle

When you say “large amounts of server-side data”, are you meaning static content data or persistance data that changes (as in ManagedObjects)?

As in ManagedObjects indeed. Other data, like static background, bitmaps, etc … I can live with it to put it into a separate file format on a separate machine. It’s the content and limitations of the object store I’m wondering about. Is it residing completely in memory all the time ? Is it partially swapped-out ?

@Hoyle

Remember, SGS is client agnostic, so any type of notification system that SGS ever provided would need to be as well. It might be easier to wrap your server data in proxy objects that are also ManagedObjects. These proxies could then perform the notification for you when set methods were called, in whatever manner you see fit.

That’s a solution. Or use an AOP framework to wrap the methods with code that would create these events. This way you would bring down the coding to a limited number of aspects.

@Hoyle

In my experience it’s often bad to wholesale replicate data from the server to the client, often much of the state that the server knows about isn’t required within the client.

The replication would be limited to the “region” the client is using of course. Nonetheless, if the region is big, the client needs to get a complete copy at least once.

@Hoyle

You don’t have to have the client make tons of 64K chunk requests to the server. Simply have the client send a single message to the server telling it what it wants to download, and then the server can send data back to the client in 64K chunks. Just make your client be able to recieve the data in chunked form and piece it back together. This seems pretty easy with reliable ordered delivery.

Well, that’s the limit and we have to live with it. The only worry I have is related to a project I did recently - with another product. The first solution was put one object into one message. But just the number of network requests was slowing the application down. We chose for less- and bigger messages. Which did help to transfer more data faster to the clients. Hence this question.

Btw, do you know whether there’s something planned like best practices, design patterns, etc … for SGS ?

[Hoyle]

@Jan

As in ManagedObjects indeed. Other data, like static background, bitmaps, etc … I can live with it to put it into a separate file format on a separate machine. It’s the content and limitations of the object store I’m wondering about. Is it residing completely in memory all the time ? Is it partially swapped-out ?

The current SDK is using Berkeley DB, so I’m pretty sure it is partially swapped-out. As for the final multi-node stack, I’d have to guess that they will support storage of data greater than could be managed on any one JVM. Maybe Jeff can comment on the expected data limits of the final system.

@Jan

The replication would be limited to the “region” the client is using of course. Nonetheless, if the region is big, the client needs to get a complete copy at least once.

One thing you might consider doing is splitting your data objects into two groups. Client shared data objects and server data objects. This way you can send to the client only data you are sure it cares about and keep everything else in a seperate data object. For example, if you have a virtual world you are simulating with hundreds of players walking around your player, there is no need to ever send him the inventories of the hundred other players. In fact, when he requests his inventory, you may only need to send him a small subset of the data to him, while keeping more detailed data on the server.

@Jan

Btw, do you know whether there’s something planned like best practices, design patterns, etc … for SGS ?

I’ve been curious about that myself. If you browse the forums you will pick up suggestions and comments that could be the beginnings of such a thing. I think part of “best practices” can be put together by the community, especially after they open source it.

In the mean time, I think the best thing is to just dive in and start experimenting with SGS to see what works and what doesn’t.

[Jeff]

@Jan

Hi

I’m having a look to SGS to check if it is able to easily handle large amounts of server-side data. Related to this I’m wondering a number of things. If someone can help me out with them … ;D

• The server tutorial mentions data that wasn’t removed from the object store will not be collected. (Page 3, last paragraph.) Does this mean the total amount of data in the object store is limited to the JVM’s heap size ?

No, it means its limited to the maximum size of your dataabse. In the SDK this woudl be disc storage where the dsdb is located.

[list][li]I would like to implement a kind of Observer pattern where clients subscribe to changes on the server-side model. To do this it would be very handy to be able to catch changes of managed objects - in order to notify the clients a particular object has been modified. Is there a usual design for this in SGS ? Presumably these notifications must be using the server/client messaging mechanism. Correct ?

Yes. For a system that does this, see the Wonderland code base.

• I read in this forum the max message size is 64K for messages sent to the clients. Is this also the case for messages sent to the server ?

Yes this is the current design limit either direction. Our gut feelign was that anything that even approaches that size is leaving the realm of interactive communicayion. If you have a real-world counter example though we’d like to hear about it/

• Looking for answers I saw a topic about initial model upload to the clients. As my model is way too big to send regularly completely to the clients, I also though about an initial load. Making it only slow at login time. But I would really like to avoid an additional server-side setup, configuration and code development to allow HTTP too. Breaking-up the data into 64K pieces is not difficult. Alas, the resulting number of network requests alone will impact performance. Any suggestion ?

Not realy. HTTP servers and Streaming servers are built for this kind of massive data distribution. We chose not to try to replicate that kind of functionality because it already existed and it allowed us to focus on the space the SGS really needed to perform in-- interactive communication.

If you rule out my suggestions to begin with then no, I don’t have any other. I’m not sure how you find an HTTP server complex though…

Many Thanks for helping me !

Jan

Thats what Im here for!

JK