New File Format for Games

Miscellaneous Topics
#1

Well… after a lot of work and refining, I’ve gotten to a stable point with my file formats. I’ve been creating them for over a week now. They serve the purpose as uncompressed containers for data, which are easy to load. If you don’t want to write the simple load functions, you can just use the standalone library which the program I am making uses. The program is a terrain editor featuring one of my file formats. I’ve also been working on the program over said time period.

My file formats as XMM, XMI, and XMH. They are part of the XM series, obviously. The XM has no real meaning yet. I am open to suggestions on that, but I am thinking about X for uncompressed M for media. I don’t really like the medium part… so there is still stuff on that.

The logo

All file formats

I made a few image files for the formats and I got licenses around. I am using GNU GPL 3, then used their suggested reference for the people to look at GNU GPL 3. I am going to use the logo for the program. I really like the style I came up with. The program so far is a JFrame, which has a menu bar, canvas for opengl, and space to the right of the canvas. I want to fill this stuff up with tools, so with the space I had above the canvas, I put a tool bar. I coded a simple, complex plugin loading system, which took some thought and trial. It was a variation of an old one I made. I am quite happy with how that turned out. I decided to write an article for writing them on JGO and that had got some major hits in just a short amount of time. I currently have 2 project folders for my plugins, as it takes one to lay out the abstraction and another one to extend that abstraction, which links back to my terrain builder.

The goal of these files formats are simple.

XMM is for mipmaps. The file starts with a 2 byte header. The first byte is a boolean (0 or 1), which indicates transparency in any of the mipmaps levels. The second byte is an indicator for how many mipmap levels there are. There can be as many as you want. Then you have the body. The body has two components. A size array and the pixel data. Inside of the array is data that contains width and height information. There are ‘mipmap levels’ (header byte 2) amount of width and heights. The width is 2 bytes. The height is 2 bytes. Each mipmap you add makes this array 4 bytes bigger. They are shorts. After this, there is the image data. Right before the image data, a boolean indicating if that specific image has transparency. This is an extendability thing, and can be useful. As you can see, the file is tightly packed because the header, size data, and image data are all smushed together without byte offset indicators in the header. This file is procedurally loaded. Note the image pixel format is strictly RGBA.

XMI files have been renamed from XMT files. They were originally made out to be textures, but they are just another image file format - a raw one with a 5 byte header. There is no mipmaping or anything going on, so the closest thing to texture here is XMM files. The first byte in the header is a boolean indicating transparency. There is backwards compatibility with XMMs loading XMI files for mipmaps, so basically XMM files are an array of XMI files… with the exception of the next 4 bytes which are two shorts indicating width and height of this image. The width and height are not in that array in the XMM file. So that is that. Following the end of the header is the image data. Note the image pixel format is strictly RGBA.

XMH hasn’t been created, yet. I have all necessary information to make a basic one. The goal for this file format is hold heightmap information. The header is 8 bytes. There are 4 entries in the header. That means each entry is 2 bytes - a short. The first short is for the amount of terrain sections there are in the X direction. The second short is for the Y direction. The 3rd and 4th short are width and length of each triangle. Moving on to the body, there is a bunch of sections, which are loaded in by utilizing for loops and the first 4 bytes of the header. That is as far as structure goes. There is, inside the structures which indicate the terrain sections, a crap load of shorts, which indicate the height of the terrain at that specific point. XMH files are largely speculated by me to be changed a lot to support different, necessary features. I haven’t put much thought into it, but I am now ready to make this file, as I finished the minimum of my program to involve saving to file.

I am using nullsoft scriptable install system (nsis). I gotta say I love it. It is neat and effective. I am using the older version 2.5.1. What it does is load my icons and jar into a program files folder and edits the HKEY_CLASSES_ROOT so that my icons actually show on the files :point: that’s a neat touch that I love. Funny story, while I was doing this, I found my gadgets context option in there somewhere and delete that useless button aswell as a few things that needed to go. The registry offers more than just icons to me. Since I can convert files to my XMI format (currently BMP and PNG are only supported), I can add a right click context menu option to do so. I can also convert from XMI to that format, which is not implemented. I just have the conversion supported. Nullsoft lets me clean up the environment without hassle, and it even refreshes windows icon cache so that my icons actually show on the files :slight_smile:

I am currently enjoying my project, which is really a digression from creating my game engine x)

#2

For an update on this project, I am implementing an all new file type called XMP. P stands for pack. What this file format does is act as a data container for a bunch of XM files packed together. This is good because when transferring data, the installer will just have to run to unpack everything from the file. You save space, and still get the nice load speed of the uncompressed file format! Like the reason I started this project, I found a lot of people to directly load PNG files. This is good if you aren’t batching them/loading them often, and in this case gaming.

The XMP file will feature a header which has one int (4 bytes). These 4 bytes act as an indicator of how many files are stored within, and as a pointer by index to the following section. The following section of data is a bunch of ints. These ints declare files sizes. After this section, which end is told from the first 4 bytes, the files are stored back to back.
So its something like
2221234
Which means there are 2 files. The first file is 2 bytes long. The second file is 2 bytes long. The first file’s bytes are {1, 2}. The second file’s bytes are {3, 4}.
This is a minimalist approach to this and works so well.

As for the other file formats, I have finished XMI and XMM files as far as I can tell.XMM acts as a pack of mipmaps, which you create yourself. I do not feature an auto mipmap generation algorithm, but that is good for now. The reason is if you were to use opengl’s automipmap generation, then you would only need an XMI file. In the case you didn’t want the bulk of opengl’s automipmap generation, and you just want to load the bytes, that is unsupported, sorry. I don’t know any mipmap generation algorithms. :frowning:

It is pretty disgusting loading a PNG file, because there is so much loading, decompressing, recopying of bytes… yeah. XMI and XMM are straight forward and eliminate the decompress overhead with noticeable 1/3-1/2 loading speed (based on TWL’s loader). I have to iterate over all the bytes once more in order to actually detect if the file has anything other than opaque. This boolean is a nice, nice feature.

XMH files aren’t ever started yet, because I have been messing around with my terrain editor adding features for sculpting terrain. I’ve said all there needs to. I feel the XMH files are the bulk of this project, because there aren’t any good file formats openly available. I was also theorizing about an X and Z offset, which allows the user to create geometry inside and modify the X and Z to make cliffs and stuff :stuck_out_tongue: … which fixes a problem with heightmaps!

The code will be up on git sometime in the near future.

#3

I didn’t account for the names of the files. So I rewrote the file a bit.

The header bytes goes as:
int files;{int width, int name_length}…

The body bytes goes as:
{char name_letter…, Raw File Data}…

Where {} are groupings and … denotes there are more than one of the attached segment. ; are separators.
Quick run down: Header is: beginning int telling how many files we will load, and a table of widths and their file names. Body is: the name, then the file data.

Also, made this really quick B)

and added it to OP

#4

I have got everything situated around to be put up on git. WIP

#5

The main branch is, of now, stable. I will be working on the wiki and go through to check if all comments are accurate, aswell as meaningful.

Quick ref:
XMPacker and Zipper classes are for XMP files
XMLoader loads in PNGs, XMIs, and XMMs
Bitwork holds methods for bitshifting bytes to ints/shorts and shorts/ints to bytes, which is used internally so you won’t need it
Image and Mipmap are containers which hold data from XMLoader
Functional PNGDecoder

Loading PNG files is slower than it should be. The reason is I have to iterate through all bytes.length % 4 times to check if there is any alpha per XMI and XMM file formats. The way PNGDecoder is set up I would have to make a method that does the same exact thing or modify like 10 method’s scheme after understanding it. Thats fine because it isn’t recommended to use the XMLoader.loadImagePNG(), but utilize PNGDecoder.decode().
//

I modified PNGDecoder by twl, which is easy to grasp
Added Format enum field
Added overload constructor
Changed constructor from PNGDecoder(InputStream stream) to PNGDecoder(File input, Format target_type)
Added close masking-method for stream.close() (decoder.getInputStream().close())
Added getter for stream
Added getter for format
Added setter for format

Basically, changes that make PNGDecoder class reusable and more functional. There was constraint in the fact that you would actually have to utilize the PNGDecoder in the same place you created the buffer… like intertwined… was gross. It read the width and height on construction of PNGDecoder, so now one can define a PNGDecoder and then later use it to create a buffer.

#6

So you made absolutely no change to the decoder which provide any additional usability … you just made a mess …

#7

So your image format is similar to the ppm format (which is as basic as it gets) and your mipmap format is not really a mipmap format (its just basically a collection of Images). Your packer / zipper essentially uses the java compression libraries which is exactly the same as the PNG file specification, as you can see perfectly well by looking at Matthias’s great PNGDecoder.

What is your selling point for either trying to get people to use this new format or to try and help people learn something? What you have re done already exists else where.

#8

Hi Matthias,
What I did was not make a mess. There is some additional usability. It stores the target format, allowing to allocate only one PNGDecoder class thereof. Good for loading in batch. This is coupled with the stream, which I added a close method. It accepts InputStream, which is dynamic and all, but this library uses only FileInputStream. Its not something that was made to be roled out to your repository. I just shaped it to this library for ease of use. :slight_smile:

It is indeed similar, but the use of whitespace characters and metadata is different.
The mipmap format is indeed a collection of images. What do you think a mipmap is?
Yes. As stated in the wiki, I do use the inflate and deflate methods. They are great. This is per only file transfer over, say network.

This replaces PNG, JPG, and BMP. DDS files are a competitor, if not XM file - use DDS.
Soon there will be a heightmap file.
Soon there will be a image animation file, like gif, but modernized.

I agree, PNGDecoder is great. Constantly decompressing a file format, which is longer than loading the pixels up front, isn’t good. Benchmarks approve around 1/3 speed up.

And thanks for you asking critical questions!