Could a game like EVE Online be implemented on the SGS?

Networking & Multiplayer
#1

My complaints about the SGS are basically that database queries are not supported by the SGS itself. Now imagine you would need to implement something like EVE Online on the SGS. SGS has a huge market and mission database with millions of entries and you would need a database system for it to manage all the queries needed for proper access to the markets like EVE provides it (local, regional and global markets and mission data). If you would do that with just GLOs you cant expect any sophisticated answers in decent time and the game would be broken. So you would need to use an external database and the SGS currently just provides this using raw sockets with asynchron query/result access. This means you would need to do a query or database manipulation operation on one SGS task and get the results on a different task. What about transactional safety then?- It could happen that it is broken because the manipulation operation is send from one task (which could be discarded afterwards) and the result is given to a different task (and this result is maybe invalid due to the discard of the task before); or the sending task commits but the database fails and the state of the SGS objectStore is not valid and needs to be fixed.

So, how would you implement an EVE Online like database with the SGS?- IMO the SGS model is broken for games which needs to deal with some huge databases as far as transactional safety is concerned, but maybe you have a better idea I am not thinking of.

If you have never played EVE to answer this question: EVE offers a 14-day free trial currently where you could get an idea about this huge database and the way EVE provides access to it.

Ragosch

#2

Erm, havn’t we had this discussion before?

Jeff has reported that he and his crew are looking at solutions to just this problem.

Endolf

#3

Yes, we had this in a way before - but I feel it is not taken serious as a real problem. I wanted to point out in a different thread, that it is not only our game which needs this but that there are a lot of other games which need this also … and to give a good example of a game which is currently top ranked and has a constantly increasing number of subscribers which would need that also, I made this example. It is vital for the success of the SGS IMO to take this problem really serious. That is why I made a new topic about it and point out the problems regarding to transactional safety if you would use an external database for it using the raw sockets from the SimTask object.

Ragosch

#4

IIRC, Jeff mentioned that people were investigating this within his team, how more seriously did you want it taking? :slight_smile:

Endolf

#5

Its on our Radar. I’d call it a tier-2 issue. A lot of how we might adress it depends on some final architectural issues we’re looking at now on the big back end.
Some thing we are looking at include

(a) a formal lock down of the ObjectStore’s internal interface to the rest of the system
(b) the idea of a well defined and pluggable “event generator” interface to the system for extensions

Dpending onwhat we do here I could see this being attacked in different ways. The thirs approach, which is ofcourse abvailable today, is to put that database ona seperate server and call to it with the raw socket interface but I see this as the least desirable solution.

SO… for teh moment I am going to tunr this around asn ask this question…

If you could have this any way you wanted, what would it look like? A more compelx and felixible index system then the name index? Generic JavaSpaces style object searching? A seperate info data base with an SQL interface? A virtualized generic JDBC connector that works asynchronously and generaters events? Some other idea?

Id love to see some debates on the pros and cons on these approaches.

#6

As someone that is very interrested in this as well ( but not nearly far enough along to be concerned with it not yet being available Cheesy ), I would have to say that an “event generator” sounds initially appealing, but I would be worried about it’s flexibility. An economic model is different than a weather model which is different than a political model which is different than, well, you get it.

One option I am thinking about would be to have a second database deployed in parallel to the SGS’s ObjectStore to ensure the same level of scalability & failover as the primary ObjectStore but without the same level of restrictions as the ObjectStore. However, to minimize risks it would still seem to need a similar level of access control as you are placing around the ObjectStore. Perhaps ELOs ( Event Level Objects ) which actually provide the interraction w/ the EventModelStore in a manner similar to how message driven beans can be used in J2EE development .

Example of an EVE like market system, ELOs can be written purely as market managers, a GLO sends a request to an ELO for prices of a given good at a given location. The ELO can PEEK the EventModelStore database for availability of the good, recent and average sale and purchase prices at that location, etc., and calculate a new sale price for the player. The player accepts the price and makes the purchse, the ELO would GET the quantity of goods at the location to reflect the sale and also update the database with the new sales prices.

Another ELO might be triggered on a scheduled basis to review each regional market for supply/demand trends and update prices over a larger market segement without being specifically driven by a single player’s actions ( if three out of five ports in the area are all selling large quantities of Robots, then the market might adjust the cost of Robots upward at all five ports to better reflect demand ).

Since the EventModelStore/ELO/EventTask is a generally seperate set of functionality it can be layered in on an as-needed basis. If a game is more basic and does not require such a model, then the standard ObjectStore/GLO/Task implementation would be enough.

Talking thru my fingers a bit there, so I may need to formalize my thoughts later, but you may get the idea. It’s probably too complex to be reasonable, but I was trying to keep it in line with similar direction that SGS is going already and maintain the power and flexibility the current SGS model already provides.

#7

[quote]As someone that is very interrested in this as well ( but not nearly far enough along to be concerned with it not yet being available Cheesy ), I would have to say that an “event generator” sounds initially appealing, but I would be worried about it’s flexibility. An economic model is different than a weather model which is different than a political model which is different than, well, you get it.
[/quote]
Okay let me make that one clearer. What I was talkign about was GLO events. basically so you can plug any arbitrary code directly into the kernel of the system and generate what look like SGS events to the system.

Though if you have another idea the name spurred, id be curious to hear it.

The rest of this is very intresting… but i need to read and really digest it before I ask any more questions Cool

#8

Pluggable events would definately be good - no doubt. But for a market-database like EVE has you need a real database system which can be accessed in a transactional safe manner seen from the SGS point of view (it needs to be safe inside the current SimTask). That is why you cant do this in an asychron way because it would break the transactional safety as I have pointed out in the OP; it needs to be synchron to the SimTask. The transaction should database-side not be commited until the SimTask is ready to be commited also. This way the results in the objectStore and the database can be seen as either both valid or both rolled back, i.e. both transactional safe in relation to the same context (the SimTask).

Seen from a programmatical point of view SQL command would be good to manipulate and query the database. An SQL query is optimized by the database system and leads to an efficient access (this is the job of a database system, so let us let it do it’s job), which is needed for sophisticated and fast results. I am totally against any form of doing it by hand while there is a database system which can do it even better because it is specialized to do so.

Ragosch

#9

Sounds like premature optimalisation.

sync, transaction, locking, connection stratigies certainly wouldn’t be my entry point in this. Hibernate, EJB3, JPA projects have all asked these questions too, but perhaps lead into differend conclutions because of differend criteria, wouldn’t it pay to look at that documentation/those discussions? We are reinventing the wheel here, surely a more slick one that runs better on our projected terrain. But it will still be a wheel, and thus will have similarities in solutions and problems. And not even tied alone to the techical aspect of it, but also to stuff like ease of use to the programmer, and esp. on that point we could learn alot from other projects.

#10

But that is exactly what i dont want to do … reinvent the (database) wheel again - I want to use it because it solves the problem of manipulating and query data which is hold in several tables in a transactional safe manner. The issue is here to embed a database nicely into the SGS to ensure transactional safety for the whole database (consisting of the database and the objectStore, which itself is just a table inside the database).

Or did I misunderstand you?

Ragosch

#11

JavaSpace database pls, ta.

#12

Ah so this is the debate I wanted to get started…

SQL v. JavaSpaces model v. …

Keep in mind that JavaSpaces put more limits on objects then the current GLO model.

So IF you are talking JavaSpaces, do you want GLOs restriced with the JavaSpaces limitations? Do you want it as a seperate parallel structure (could be tricky and error prone as now you have two different object models in the system…) OR do you just want search abilities on GLOs simialr to JavaSpaces? (might be tricky to implement without limitations similar to JavaSpaces… )

#13

Search abilities on GLOs - a database is so much more than just some search abilities on GLOs. What about complex queries which do joins over several tables filtering out the requested data grouped and sorted in the desired way?- The last is what is needed if you think of a database like EVE has it … do you really want to return to information retrieval like in the pre-database area?
Ragosch

#14

Interesting discussion guys… I have some ideas starting to form here but Id like to see where the discussion goes still…

#15

Hi

I’m going to vote +1 for Ragosch regarding complex queires with joins over multiple tables. I have no imediate need for this in SGS, but my long term plans also include a trading system which, having worked on real finacial applications, I can bet will get complex quite fast. On the other hand, that sort of system I could manage externally if addition of this feature was going to adversly effect the performance of the GLO based operations.

A thought I’ve just had (and I’m typeing as I think, so excuse the mess) is that maybe we partition the data, and one set of data in SGS is SQL based tables and queries, and one part is accessed via GLOs, in the SQL a type of GLOID is introduced so we can return rows from SQL queries that have the ID’s of GLOs in them. On the other hand, the GLO id’s are just strings, so it’s perfectly acceptable to have 2 interfaces to the database that are seperate data tables, one with serialised GLOs, and one thats ‘normal’ tables. As long as they are in the same database transactional integrity might be easy to maintain, which was Ragosch main objection (as I understand it) to having 2 data sources.

Endolf

#16

i think Javaspaces is much more then just a database of fields (relationships can he added to objects)- you can also have Works to do stuff for you. Its also a much more modern way of storing data, its also easyer to get your head around (after you get the hang of it). GLO’s are much more then a Entry in a Javaspace (custom Entry?). The cool thing i find with SGS is the comms layer, also the way you can make rooms etc… The SGS team have done an amazing job here. GLOs and Entrys feel like the same, SQL is just nasty, and can get out of control fast. Storing backups of the SGS world into a Javaspace feels like a nice - i have made a game server that has all the data stored in a few Javaspaces. SGS is way better then what i did (game worlds need to be in memory when they get BIG). And if banking uses Javaspaces for the biggest game in the world (the stockmarket) then i guess is ok for us ;).

I’m supriesed that people think a game like EvE-Onlines persistent world cant be done in SGS, with game items being stored in a Javaspace- if SGS can grid its world over lots of computers for more power (javaspaces can also do this) then theres no big deal (part from the hidden problems you will never find till its to late) :slight_smile:

I dont do much highend database modeling, so i cant say if SQL would be better- its just different… Javaspaces just tasts nicer…

#17

Hi

I haven’t looked in to java spaces, so i don’t know what it’s capabilities are.

The reason I suggested SQL is that the database providers like oracle have spent years and years creating algorithms for analysing the queries and finding out how to optimise them, you also have the ability to add indexes etc in to the tables themselves when you know certain fields will be searched on frequently to avoid having to do table scans.

An example query I thought of, which links in to eve and the trading system, is something like this.
1 table contains all the currently offered items on the market, the item instance id, the location it’s for sale at, and the price.
The item table contains a mapping from the id to the type, and instance stats of that object, things that can change, like wear, no usages etc. The location is an id to a particular shop or space station, the sector/area it’s in, maybe some higher grained geographic area structure information too.

I want to find, all the type 4 plasma drives, with condition >50% in the same galaxy as me with a retail price of < 100,000,000 credits.

in SQL I can do joins, inner queries, indexes on tables, to try and reduce the number items I have to transfer across the bus, and reduce the amount of data in RAM, also, improve performance of this queries, especially when there are large numbers of objects on the market or in the object table.

I think Ragosch feels that trying to do this sort of query with GLOs could be not very performant. Jeff may be able to suggest ways to use GLOs that I hadn’t thought of that would satisfy the issues above.

Would javaspaces be able to cope with say 2 million objects in the objects table and say 10,000 items for sale in a performant maner?, a very quick look at this article suggests that this really isn’t a good use of them. (I’ve never used java spaces, correct me if I’m wrong, I like learning new things so I won’t take offence :))

Endolf

#18

have a look at a very hardcore jini/javaspace. there also the very cool blitz:

http://www.dancres.org/blitz/

no point me covering the whole thing on a forum when theres loads of info
out there- u can make a much more complex data models then just a few Entrys
hanging around a space ;P- dont want a flame war; i gave up on SQL a while
back- oracle:anything thats been around a while, doesnt make it the correct
thing to use today.

#19

I use SQL-like databases since the early days when it was called SEQUEL in the 80s; if you are used to real database systems and a powerful data query and manipulation language like SQL, you do not want to do that on your own again ever in your life. Is there any proof that javaspaces can provide all the beneficial features of a database system and performs well in comparision to SQL databases?- I think especially of fast joins between several normalized tables, complex selections over those joined tables, and result-sets which are grouped and sorted in the desired way (including standard functions like average, sum and such). If you know of any performance contests between databases and javaspaces, let me know and post a link here.

Those features are not trivial, a good database represents about 30 years of experience using relational databases and speed up of database queries by optimization of the query before it is performed on the data. I remember those days when I studied computer science in the early 80s where these database manipulation and query languages were born. I can remember the problems with optimization of database queries and therefore I dont want to use anything “new and modern” which is not proven safe, fast and reliable as far as database management and database query is concerned.

Ragosch

#20

Then why bother with SGS, or Java for that matter?.. EvE runs over Python for its serverside i think- its an open source framework, maybe go play with that, i guess thats ‘proven and safe’ now, took a lot of fixing/work by CPP to get it where it is today mind, not sure all there patches are publicly free…

Its a case of thinking about what Javaspace gives to SGS and the Java programer. I DO understand what you are saying. im not backing Javaspaces, It just seems to be a better fit then SQL- being Javaspaces is also a disturbuted system of objects, and can be expanded across many computers within the same mindset as SGS. Heres a nice webcast/podcast (its just mp3 whatever).:

Javaspaces is a newish mindset; Its infact been around a long time, and has been proven and is being used today. I’m sure there is a whole world of people out there that will give u performance mumble, if that makes you feel better… Google ‘Tuple space’s’ maybe you seem to like history, but thanks for your history lesson on databases (i didnt no SQL was also SEQUEL, thanks for that, well done; Billy). All in all it seems we can all write our own storage system regardless of the SGS team doing it for us- so use whatever you like, i just thought saving an object to your database was nicer then writing an SQL call… Looking up object by using matching would also be nice- if you want details on speed and stuff- i’m sure the JINI.org dudes can tell you that info my friend… I still havnt seen anything here that makes me feel writing aload of SQL is the way to go… But making a Player object and 'write’ing it to a space sounds cool to me…