Java .Blend, Blender DNA fully implemented in Java /w R/W file access -RELEASE

Engines, Libraries and Tools
#1

Edit (Jan. 10th 2016): Update Version 1.0.2 available!

Blender files are basically dumps of Blenders native heap memory content (C structs and scalar types of the Blender DNA (its core data model). A .blend file is organised in blocks of arbitrary length, and the type of data is described in a special block, which contains type information (meta data) for all C types which can exist in a .blend file.

The Blender DNA is huge and new aspects get added frequently. Thus, it is almost impossible to provide a library which supports all aspects of it without breaking features in the next Blender release.

When I was analysing Blender’s data model, I noticed the potential of the extra meta data in each blender file and decided to write a library which leverages it to achieve full coverage of the Blender DNA. The result is Java Blend (open source, now beta http://homac.cakelab.org/projects/JavaBlend). It is a combination of a code generator and a runtime I/O package. Together they provide generation of Java classes (facades to native data) for all DNA types at development time and random R/W access to Blender data at runtime. Thus, a generic any-version blender file import/export in Java which features all aspects of the Blender DNA data model.

I designed this toolkit with the ambitious goal to serve for any kind of interaction with blender. Whether just importing data into another graphics engine, conversion to a different data format or external processing of data to add features to blender. The fully generic mapping of the entire Blender DNA core data model combined with both import and export supports e.v.e.r.y.t.h.i.n.g! Also the generic approach reduces maintenance effort to a minimum. New details in blender’s DNA will be instantly reflected in the generated data model and implicitly supported by the I/O module.

I’m posting this here (1) obviously to share it, so it may find its users and (2) to request for comments/feedback on its design/implementation/use. I put a lot of work in it but there may be some things I didn’t think of.

#2

This is pretty awesome. I haven’t looked at it but it sounds great.

#3

Hi

It might be very helpful in some 3D engines to implement import and export to Blender. I’m going to look at your source code.

#4

Thanks a ton for this great library!
Yes, Blender’s serialization is pretty awesome and their process with that python script running over all C sources and extracting the structs is just super-awesome. :slight_smile:
I would have liked for your library to be available one year ago :), where I needed it to use Blender/Cycles for rendering a procedurally generated scene from out of Java. So I am more interested in the process of procedurally generating static scenes (or possibly videos) as .blend files and then letting Blender render that.
Do you know whether the .blend format actually supports text? Or is text just produced by API functions of Blender and then tessellated to be a triangle mesh?
Also, back then, I had a lot of trouble finding the semantics of the structs and fields and where ‘int’ indices point to and to know how, for example, a keyframe is stored in Blender and what is associated with it.
But I guess I have to peek into the C sources to get to know that.

#5

Thanks for the feedback. ;D

But honestly, it’s not that much of a big deal … someone just had to do the work :wink:

#6

Sorry, can’t answer your question about the text (at least not yet). But you are addressing an important point, which is documentation.
I’ve build a documentation system which provides documentation to the code generator. Provided documentation is (1) gathered from source code using doxygen, (2) gathered from python API using a python script and (3) added manually by myself. There are still a lot of gaps in the documentation and my idea was, to setup some kind of community driven documentation system, so everybody can easily contribute to it, like in a wiki. I thought the easiest way to set it up is to put the documentation (which is now stored in JSON files) as a separate project on github, so everybody can get it and submit changes. What do you think?

#7

Having a more thorough documentation is an excellent idea. I think I did not see you post about your project idea on the blender mailing list. You might want to inform those people about your idea and gather some more ideas and collaboration about a general documentation that also the C language guys can benefit from.

#8

I’ll do. But, I just came back from a long trip and feel a bit dizzy. Did the post right before I left and was now curious about the feedback :stuck_out_tongue:

#9

Hi

My suggestions are these furthers:

  • Choose a license (I know that you plan to do it)
  • Use versioning (Subversion, GIT, …)
  • Support input streams in BlenderFile (Have I missed anything?)
#10

Thank’s a lot, gouessej! I really appreciate, if someone actually spends his time to review the source code and provides professional feedback!

I think, I will go with a dual license with this. Still did not read enough about it (actually I hate the whole licensing topic :P)

Of course, I use a version control system, just not publicly for security and privacy reasons. But I will always provide a detailed version number, so I can merge patches submitted from fellow open source developers.

I also considered InputStream interface. (I think you mean the way the InputStream interface is used by archive readers such as ZipInputStream etc.) When I looked closer at Blender’s I/O concept, I noticed the potential to just map file contents into memory, in terms of memory mapped files. That’s why I dumped the streaming approach early on and used an abstraction which allows direct read write access to raw data (regardless whether data is on disk or in memory). And my goal was, that I will introduce memory mapped files behind that interface, when I’ve sorted out the whole code generation and block management part. Just forgot to write that task down - so, thank you for the reminder :wink: I’ll implement it right away!

Regarding the introduction in game engines: That would be nice. I’ll ask them if the time has come, but proper game engines like jMonkeyEngine and libGDX already have their own blender file reader implementation well integrated in their data model. Will see.

#11

As far as I know neither JMonkeyEngine nor libGDX supports loading “native” .blend files.
Not even Ogre3D can do that. But Ogre3D provided a Blender plugin to export scenes into the ogre format.

Additionally, I have a question: Do you know why the normals in the mesh are short values?
Running your demo with the cube mesh prints strange short values of 18918.
This is not MAX_SHORT and also not MAX_SHORT/2.
How would I map those to the [-1…+1] range?

#12

Oh really? You mean they read the .obj format? I was sure that I read somewhere, that they use .blend files. Especially JMonkeyEngine, since they have their own modified Blender version, too.

Normals are normalized and if represented as short, they are mapped to the range [-32767,+32767].

These are the conversion functions from blender (see math_vector_inline.c):

#13

What do you mean exactly? I don’t see the point.

#14

means, there are private and unsafe things stored at his repository. which are safe as long as private.

#15

Why not separating what can be safely published from the rest?

#16

Does anyone here know whether there are Java or non-Java rendering engines that can import Blender’s ‘.blend’ files or another scene (and material!) format exported from it while at the same time more or less read in the material node graph as used by the Cycles rendering engine in Blender?
What I am searching for now is a realtime rendering engine that can basically read in the meshes and materials and display that (of course only locally shaded).
Ideally, this particular rendering engine (or its Blender importer) would convert the material graph nodes to GLSL shader code, such as when having a simple material consisting of only a diffuse shader, a glossy/specular shader and a mix shader, it would simply convert that to GLSL, which should not be that much of a deal, hopefully. :slight_smile:
Otherwise I would tackle that problem on my own.

#17

Man you must’ve had to read a ton to get this up and running. Good stuff!

Also, JMonkey does in fact load .blender files.

#18

Sorry that I kind of abandoned this thread in the last weeks. Release was serious business due to licensing issues. I wanted to get this as free as possible but parts of the documentation are copied from Blender sources … so … just read it on my website :stuck_out_tongue:

Java .Blend is now released. I added a couple of examples especially for the generation of new blender files and fixed some remaining bugs etc. pp…

Have fun trying it out!

KaiHH: I thought about implementing something similar but more focused on modifiers to get mirror, subdivision etc. going. Are you interested in a cooperation than pm me. Maybe we can benefit from each other.

gouessej: I’m generally not a fan of github, sourceforge etc. Who knows when they decide to modify their conditions? And, I have my own server, why would I put my stuff on a public server, where I have no control over it.

#19

I strongly agree with @gouessej here. GitHub is for when you want to collaborate with people. If on the other side you want to work on your project for your own and only let people look at the sources or download a zip of it when they know you and probably ask you, you may stick with your own hosting solution. GitHub is about reaching people when they search for something like “Blender Java” and then see your project.
Also GitHub has excellent uptimes. Who knows when your server will be up/down or when you decide to apt-get update/upgrade it or change its domain or change its repository path or change the SCM system ? How save is your own server against attacks?
These are a lot of questions for possible collaborators that a service like GitHub likely has better answers for than an own hosting.
The collaboration tools in GitHub are excellent. Commenting on commits, providing pull requests. You even have a Wiki and a small but dandy Issue tracker in it. These are all reasons why GitHub got so popular in short time.

#20

i’m with homac.

keep your private git repo on your own server. yet, you can simply create a new one on github and push/pull from/into your private and use both worlds.