Loading a binary file

Newbie & Debugging Questions
#41

[quote]Yes, that’s exactly the point of structs - directly modifying data in Buffers by mapping normal Java fields over them.
[/quote]
I was looking closer at your proposal Cas, and one of the problems I have is that the Struct class can’t have non-buffer data members. Wouldn’t we better off adding some kind of field access specifier or field metadata attribute - like @structmember, that indicates that a field is actually read from the buffer? The structmember attribute could be a little more flexible too - like specifying the number of bytes that get read from the buffer for that data member.

God bless,
-Toby Reyelts

#42

[quote]I’m waiting for someone to come up with an implementation… /me taps fingers impatiently…
[/quote]
Hey swpalmer,

You could probably do something like:

MyStruct extends Struct {

  private @structfield( 1, 4 ) long val1;
  private @structfield( 2 ) int val2;
  private @structfield( 3 ) int val3;

  // some other fields 

  public long val1() {
    return val1;
  }

  public val1( long val1 ) {
    this.val1= val1; 
  }
}

Some thoughts:

  1. You need the structfield attribute to tell you what order the fields should be read/written, because fields aren’t necessarily stored in order in bytecode (unfortunately).

  2. Like Cas said, this won’t remove any of the array-bounds checks that exist in the current code. It’ll just make things a little simpler.

This would probably be pretty simple code to write. Maybe a couple days worth of effort.

God bless,
-Toby Reyelts

#43

I reckon that the whole struct thing can be done with metadata and a specially tuned VM now. I don’t quite know exactly how metadata works yet but I have a hunch it does what we want, ie. mark a class as having its fields laid out in memory in order, etc. and the VM can generate special case code when it encounters the metadata tags.

Is there anywhere that explains 1.5 metadata and how to use it on the web that’s easy to get at?

Cas :slight_smile:

#44

[quote], but I would still expect the JVM to deal with it and produce correct results by generating code that manipulated non-native-endian mapped fields according to the JLS instead of just getting it wrong.
[/quote]
Im not sure I understand you. From what I THINK I understand, this IS what Java does today. if you use DataInput/DataOutput then it twiddles the bytes to be a standard that any Java VM can correctly read in.

#45

[quote]I reckon that the whole struct thing can be done with metadata and a specially tuned VM now. I don’t quite know exactly how metadata works yet but I have a hunch it does what we want, ie. mark a class as having its fields laid out in memory in order, etc. and the VM can generate special case code when it encounters the metadata tags.

Is there anywhere that explains 1.5 metadata and how to use it on the web that’s easy to get at?
[/quote]
Sun is very reluctant to attach any semantics to metadata outside of emitting compiler warnings or errors when it comes to the Java compiler and VM:

http://forum.java.sun.com/thread.jsp?forum=316&thread=431398&message=1957008#1957008

If you want information about metadata, you should download the 2.4 prototype, which contains draft specs for all sorts of things, including JSR 175:

http://java.sun.com/developer/earlyAccess/adding_generics/

God bless,
-Toby Reyelts

#46

[quote]Why is the BSP example pointless? I’d simply map my BSP file directly from disk into memory and then when I needed to walk it I’d just have a sliding Node struct to follow the tree, and sliding Vector3fs to find coordinates in it, etc.

Cas :slight_smile:
[/quote]
You said structs were not to be used for lw objects. I assumed you would want the whole BSP in memory at once, in which case I’d expect you to be using lw objects.

But it hadn’t occurred to me that you might want to only selectively/lazily load the file.

How tricky is it to implement deformable maps in this scenario? (and I don’t mean just a single elevator, or on/off hole-in-the-wall. I mean interesting changes :))

#47

[quote]You said structs were not to be used for lw objects. I assumed you would want the whole BSP in memory at once, in which case I’d expect you to be using lw objects.

But it hadn’t occurred to me that you might want to only selectively/lazily load the file.
[/quote]
I think Cas wants to reuse a single ‘struct’ to give some “structure” to different areas of the ByteBuffer on-the-fly. This isn’t about many objects or lazy loading. It’s just about making the data accessible in an efficient way. It is simply the “view” of the data in the ByteBuffer that we want control over.
(I’m sure Cas will correct me if I got that wrong.)

#48

Yes, that’s what I, and many games developers, need to be able to do. There are so many overheads to doing it with Java objects it’s just not feasible.

There are plenty of other uses too such as a vertex buffer processor. Say you needed to write interleaved data out to an OpenGL vertex buffer. You might have tons and tons of different data to deal with and you can’t very well store it all in objects and keep reading and writing objects and constructing and waiting for GC as it’s discarded every frame for a completely fresh set. Far better just to point a struct at it and slide it along the buffer to do your processing.

Cas :slight_smile:

#49

So how is that different from the efficiencies gained from immutable objects?

#50

Because we’re talking about data that’s so mutable you process megabytes of it 85 times a second. Or data that’s so large and complex that you’d need to double the memory storage to use it in Java and pointlessly get the garbage collector to traverse the whole thing periodically only to discover it’s all still referenced. Or data that takes ages and ages to load in using the Serializable interface because it’s so big but is mapped in the blink of an eye with a MappedByteBuffer.

Cas :slight_smile:

#51

there is type problem too, how annoying is to manipulate C unsigned char with java signed bytes …

#52

[quote]Or data that takes ages and ages to load in using the Serializable interface because it’s so big but is mapped in the blink of an eye with a MappedByteBuffer.
[/quote]
The mapping may not take long, but if you then go and read that data sequentially from beginning to end, it can be slower than using ordinary file read operations. This happens when the mapped pages remain in memory after you have been past them and cause parts of your heap to be kicked out to swap file. So memory mapping is great when you randomly read only a small part of the data, but if you actually read the whole lot it can be quite bad (at least on Windows in my experience).
In my case I have 600MB heap and a 1GB data file. That brings me quite close to the address space available to ordinary applications on 32 bit windows.
You also have noticed my RFE relating to some of the problems with memory mapping:
http://developer.java.sun.com/developer/bugParade/bugs/4724038.html