How to manage very large 3D arrays? (OutOfMemoryError)

Newbie & Debugging Questions
#1

Hi,

I just started developing my first game, a learning project of sorts. It’d describe the envisioned finished version as something between Minecraft, Dwarf Fortress, and Battle of Wesnoth; a blocky, tick-based simulation of a randomly generated fantasy world with thousands of unique, detailed characters in it, in which you can do pretty much anything. I got to a problem right at the beginning. I was trying to generate a basic random world, so that I have something to work with when setting up the display, but when I tried to represent the game world as a 24,576 x 24,576 x 150 3D array of Element objects(a class I made to eventually represent everything in the game), I kept getting the “Exception in thread “Display” java.lang.OutOfMemoryError: Java heap space”.

I got those numbers from Dwarf Fortress, and that’s not even a large world. And Minecraft worlds can be up to 6,000,000x6,000,000x256, so it should be possible, but how? I tried to google for how to save it in a file, but all the advice there was a bit off from what I wanted and I couldn’t divine anything useful from it. So, how would you go aroung managing a game world like this?

#2

Split it up. I’m sure there is a lot more details on this if you look up ‘chunking’.

#3

One way to do chunking is to store smaller cubes in a Map, using the cube’s offset coordinates as the key. Wrap it up in a facade that makes it look like one giant cube, but which handles “missing” cubes by constructing/generating/loading them on the fly.

Cas :slight_smile:

#4

As previous posters have already suggested, divide your ‘world’ into chunks and create some sort of service or facade that loads chunks as required on the fly.

Some other things you might want to consider adding to this idea:

  • caching: as the user moves around you will loading new chunks into memory (on a background thread) and discarding stale chunks, this is essentially a cache. If your cache is slightly larger than the extent of your world you can retain chunks even though they are not currently in view since they are likely to be needed again due to locality (i.e. user will move back). When you do need to discard from the cache (you will need to apply some sort of limit(s)) you could use an LRU algorithm, or distance to user, or cost-to-load, some combination of these, etc.

  • pre-loading: similar to the above, you could pre-load chunks that the user is likely to see soon, e.g. you display (say) all chunks within 3 units-of-distance from the user, but you load all chunks to a radius of 5, or load the ones in the direction the user is facing, or some such algorithm.

  • LOD (level of detail): not sure if this is relevant to your game, but if you are (for example) rendering a 3D environment the more distant chunks do not need to be at the maximum LOD. Therefore you can load less detailed data / textures initially and progressively load more detailed data as the user gets closer.

Just some thoughts to consider.

  • stride
#5

My advice is to makes some some small games first.

#6

Oh yes, and that :slight_smile:

For yours and everyone else’s sanity, please, please make Space Invaders first.

Cas :slight_smile:

#7

If your entities don’t have any special data, you can use byte arrays inside class with proper getters/setters instead of object arrays - this solution is using from 4 to 8 times less memory. If you need to add special data to some tiles, then you can just add them to proper list/map.

#8

Procedural content. Read this: http://www.java-gaming.org/topics/uniform-feature-points/28000/view.html. Schemes of this nature are the basic building blocks.

You can store an (effectively) infinite amount of content in zero persistent memory. The only maintained memory you need is for whatever schemes you want to support PC changes to the environment.

#9

Alright, first off, thanks for all the advice, much appreciated. Here are the things I checked out and how they turned out:

Chunking

The only way to divide the world into chunks that I could come up with was to create a 5D array, with the first two coordinates being coord of a 2D chunk grid comprising of 32x32(chosen arbitrarily) chunks, and the rest a normal 3D coords withing the chunk. When I tried to randomly generate a world like that(nothing too complicated, just a random number from 0 to 9 to represetnt ten different kinds of blocks), I got the same error. I think I’m missing something important here. I get the error because the array is too large, right? So how exactly do I go around chunking it into smaller pieces so that it doesn’t crash when I try to generate a world? I could save each chunk in a separate variable intead of an array, but then I couldn’t make worlds of arbitrary sizes, or could I? Here is the code for the chunked world, for reference:


		for (int chx = 0; chx < currentWorld.worldArray.length; chx++) {
			for (int chy = 0; chy < currentWorld.worldArray[0].length; chy++) { 
				for (int ix = 0; ix < currentWorld.worldArray[0][0].length; ix++) {
					for (int iy = 0; iy < currentWorld.worldArray[0][0][0].length; iy++) {
						for (int iz = 0; iz < currentWorld.worldArray[0][0][0][0].length; iz++) {
							currentWorld.worldArray[chx][chy][ix][iy][iz] = new Element(random.nextInt(10));
							System.out.println(currentWorld.worldArray[chx][chy][ix][iy][iz].ID);
						}
					}
				}
			}
		}

Byte arrays
Tried to google and read up on them a bit, but I’m still not sure what do you mean or how to implement it. Are you saying that I should store the data about what’s in each block in a byte instead of in other data types? So far, I’ve been planning on having either blocks or mobs filling the game world; Blocks will just have an integer ID referring to a definition of that block’s properties(texture, hardness, drop, an so on), and mobs will have tons of properties like species, inventory, health, etc., etc. They will also be subject to a regular AI check, as soon as I set up the rest of the basics. So how would you implement this using byte arrays instead of a more straighforward solution like ints and Strings?

Simpler game
I do plan on making it simple… for now. It should look more or less like Dwarf Fortress with a graphic pack:

[spoiler]

[/spoiler]

I don’t want any of that 3D or real-time stuff. Just 2D graphics, 3D block-based space, and tick-based time. I don’t plan on making it super-complex righ now, either. A hero slaughtering sheep on a giant, randomly generated cube world composed solely of “Rock” and “Dirt” blocks would be enough to make me happy for now :slight_smile: I’d like to add a ton of things later(that’s what I like about this porject the most, I can keep adding stuff indefinitely), but for now, I just want to have the basics set up.