One of the big problems with Java is loading a binary file; for example an md2 model or a tga file. I 've been reading alot of code from the internet lately and it seems the common way to do it is load the file as a byte buffer then cast the bits to the data type you need and fill in our values as you go. This sucks compared to the super easy and simple c method of simply casting a chunk of memory to a struct. So is there a better way to do it? Maybe with nio?
I myself have been wandering the same thing for ages.
Unfortunately the method you mentioned seems to be the only method I’ve seen.
Alternatives? Don’t see any.
Depends what you’re looking for, speed wise there isn’t anything as nice a mapping structs over a block of memory.
If you’re an easy way to manipulate a block of data I’ve found it easier in the past to wrap up the byte array in an input stream and wrap that in a data input stream.
Just writing a read method on each of your objects normally covers this. So you just end up with TGAHeader object that knows how to read itself from a data block.
Kev
Java Land is a warm and fluffy place, where cuddly Objects live in happiness and the Sun never sets (ho ho). Just beyond the borders of Happy Java Land there lies the nasty, spiky, rocky, thorny place called External Systems Land, and it is here that you find all the other programs written in the world by anyone else, ever. Unfortunately Java Land is a tiny little country surrounded on all sides by External Systems Land.
But the benevolent ruler of Java Land has decided that because Java Land is so great, no-one need ever leave, or talk to anyone else outside of the border in their own tongue, and when the Objects do try and leave, they are only allowed to take a hammer with them and are under strict instructions to use said hammer to solve any problems they come across. Rather like America but with hammers instead of M16s.
What this colourful analogy means to you is - because the binary world of external systems is totally incompatible with the managed object heirarchies in Java, the APIs available to you are extremely inefficient at talking to external systems, and torturously complicated if you want to try and make them efficient. Wouldn’t you love to be able to simply map a bytebuffer using NIO onto a BSP file, and simply access it using a BSP class, and it’d Just Work like in C++?
Well - you could do this, but it requires a new feature in the compiler and the VM. Head on down to Bug Parade and Vote for Structs! This feature I consider so important that when I discovered generics would be available in 1.5 I immediately shifted all 3 of my generics votes onto 3 votes for Structs.
Structs are a great solution to this problem as they deal with all the issues of the day neatly:
-
At the end of the day, you’re still dealing with Objects, so no existing APIs or GC policy needs changing whatsoever
-
Super-fast and efficient access can be optimised into the JVM, but the compiler can be made to generate current-standards compatible code as a fallback
-
And finally it’s the first time Java has officially tried to communicate with external systems in their own tongue
Vote! Vote! Vote!
Cas 
In C++ structs have been replaced with enums.
Java 1.5 has enums, why do you want a struct when something better is in place?
how does an enum help with reading a binary file?
[quote]In C++ structs have been replaced with enums.
Java 1.5 has enums, why do you want a struct when something better is in place?
[/quote]
In Polish, we have an idiom (which is of course almost meaningless after translation to english): “What a cinderbread has in common with windmill ?” And no, nobody reply ‘flour’, it is an idiom.
How can you replace struct with enum ??? This is like replacing integers with loops or HashMap with RemoteException. Totally different uses.
[quote]In C++ structs have been replaced with enums.
Java 1.5 has enums, why do you want a struct when something better is in place?
[/quote]
Eh? The closest thing to a C struct in C++ is surely a class, compilers even treat them identically ('cept for default public/private irregularity). The snag in Java being that you have no control over how things are laid out in memory, so you can’t just bung raw binary data straight into an object.
This is nothing to do with platform-independence! You continue writing your portable applications, they will not change.
It is possible to implement structs in a completely platform-independent way - some consideration for data type sizes and use of the NIO ByteOrder system will be needed. The only concern I have for structs is how to handle unsigned data, but that’s what discussion is for!
And to address your concerns that it’s just not needed - while you can import and manipulate binary data without the structs concept, you might just find that things will be a hell of a lot faster if you could use it. We don’t need Collections, generics, static import, NIO, autoboxing, Javadoc, enumerations, more than one integral and one floting-point type, Swing, JoGL etc. We don’t need them, but they do make the platform a lot nicer/easier/faster.
N.B. My three votes are in for structs, but I don’t think it’s going to change anything. Sun are flat-out ignoring this RFE, and I suspect they are hoping it won’t gather much interest and will eventually disappear from view. We’re coming up to ONE YEAR of it being in the top 25 list, and still no Sun evaluation. Gosh.
If you ever really, truly believe in platform independence you will realise that all other platforms that exist aren’t Java, and if you want to interoperate - and you sure better do, because there’s no point in writing programs that live in their own world these days! - you need the tools to do it in your development platform of choice.
Structs aren’t about games programming. They’re about efficient I/O with external systems in those external systems’ formats. If you can’t do that in Java then a powerful reason for using Java, that it is crossplatform and easy to develop in, is moot, as it’s far easier just to keep on using the old native code to manipulate the native data.
Put it this way: with structs you would be able to very efficiently write a TCP/IP stack in pure Java, or a networked filesystem, or a harddisk controller, or an OpenGL driver, or a 3D game engine. It enables the entire world of external systems programming to be within reach of Java.
Cas
Well, aside from the concept of structs which is all very wonderful. The way you do this stuff most efficiently NOW does seem to be with ByteBuffers.
You can read in the number of bytes that make up your ‘struct’ into a ByteBuffer. You must explicitly set the ByteOrder of the ByteBuffer to match that of your data file. Then you use get() getInt(), getDouble(), getShort()… to initialize class members.
It is tedious, but I bet you could use reflection to automate most of it… then it would be less tedious - just slow
Instead of structs it might even make sense to develop a simple API that works now using some sort of mapping to members and a method that initializes an object using that mapping and a ByteBuffer.
Structs live only inside ByteBuffers. That was the reason they existed!
The whole mapping-things-to-bytebuffers idea is basically fraught with hassles and very inefficient in a JVM, requiring many, many boundschecks, and also many, many lines of code to achieve something which could be very simply achieved with some language enhancements.
Structs is all about making things easier for developers that need to do this kind of stuff. There are a surprising number of developers who need this functionality; the problem is that the vast majority of Java developers that exist right now are ones who have come from a completely different application domain and so when you say “we need structs in the language!” they say “Huh? Why? What’s the point in that?”
Cas
I got that problem too. I tried to use reflection to automate reads from a DataInputStream, as swpalmer suggests in his post, but this failed because :
-
Class.getDeclaredFields is not bound to return the fields in any particular order, so there is no reliable way to know what field should be read first ;
-
unsigned types cause troubles : if you store an ushort in an int, reflection will see an int and read 4 bytes instead of 2 ;
I also had problems with array lengthes, but I can’t remember whether I solved them or not
Now I am wondering if annotations could be used to solve those issues. Since it is a new feature, I don’t fully understand how to use them, but runtime accessible annotations could probably contain information on the order and sign of the fields.
As of performance issues, one could analyze the structure of a class only once, and generate optimized bytecode to read / write the class in binary. Never made that before, but there are a few good tutorials on bytecode modification libraries.
[quote]If you ever really, truly believe in platform independence you will realise that all other platforms that exist aren’t Java, and if you want to interoperate
…
Structs [are] about efficient I/O with external systems in those external systems’ formats.
[/quote]
…I wish you’d described them like this way back when you first brought them up (a year ago?).
“Structs as a means of communicating with external binary systems in a neat way” is an argument I can agree with :). Your early descriptions had me thinking of you wanting them mainly for internal JVM use (…I still don’t see very compelling reasons for them purely for internal use…).
Although this quote misses your other key point, perhaps it should say: “…external systems’ formats, in a fully OOP manner without yuckiness, and with all the maintenance advantages of OOP” :).
That’s basically what the RFE describes. Damn, I thought you’d understood what I was after from the beginning Go place your 3 votes now!
Cas
Handy link for those who otherwise wouldn’t vote because they don’t know where to look:
http://developer.java.sun.com/developer/bugParade/bugs/4820062.html
;D
Thanks Charlie
Cas
jean-marie dautelle has a very handy set of classes to do exactly what you are asking: http://jade.dautelle.com/
jim
Almost but not quite. There’s no performance gain: multiple bounds checks still, multiple gets and sets with all the associated performance loss 
It strikes me that the new Metadata feature in 1.5 might be a good way to implement Structs. More investigation required.
Cas
Cas is half right. Javaland is a happy palce where people play together. the outside world is an evil places where NOTHING is compatable.
Now in english:
Java defines important standards across systems such as endianess and the size of types.
The rest of the world doesnt have this. If you took an arbitrary structure and wrote it out from C under Win32 and read it back into a C program under Sparc you would find the structure hopelessly garbled.
This is the fundamental problem. Although the Java side of the equation isd defined the “binary” (or C) side of the equation mist emphatically is not. Therefor any attempt to automatically read such into Java is doomed to failure. We provide tools for doing thinsg like setting endianess conversions for ByteBuffers but you need to provide the info on how all those conversions need to be done because there IS no apriori answer.
And thats why it looks the way it does.