Since its hard to describe, so I drew it out:
Can someone help me code this pleease?
You could find the “surface normal” vector as a cross b and then rotate a around that normal vector by PI/2.
Using LWJGL’s vector classes, something vaguely like:
Vector3f surfaceNormal = new Vector3f();
Vector3f.cross(a, b, surfaceNormal);
Matrix4f rotateAroundNormal = new Matrix4f();
Matrix4f.rotate((float)Math.PI/2.0f, surfaceNormal, rotateAroundNormal);
Vector n = new Vector3f();
Matrix4f.mul(a, rotateAroundNormal, n);
(I don’t expect the code above to work first time, I haven’t tested (or even compiled) it 
Or:
calculate the cross product m := a x b. This is pointing out of the plane you want your vector in. Now you need something to make it perpendicular to both m (so it is in your desired plane) and a. One such candidate is n := a x m.
In total, n = a x (a x b).
Ever heard of the “BAC-CAB” rule? It allows any double cross product (i.e. A x (B x C)) which would ordinarily involve the calculation of around the same complexity as two 3x3 matrices, to be expressed as simple inner products. In general, A x (BxC) = B(A.C) - C(A.B), where “.” denotes the inner product (perhaps the term “dot” product is more familiar). Therefore,
n = a(a.b) - b(a.a).
That’s a neat solution, much more efficient than mine.
Hehe, thanks. But really it’s less general (and no more simple) than the Gram-Schmidt orthogonalization procedure which would also have been applicable. The cross products are cooler, though, because the method works even if one of the vectors is zero. Basically, the Gram-Schmidt method works by subtracting the projection of b onto a from b (which is not defined if a = 0):
n := b - [projection of b onto a] = b - a (a.b) /(a.a).
Then the previous result would follow directly from multiplying by (a.a). Normally, however, one would divide n by its length to obtain a unit vector. The Gram-Schmidt orthogonalization procedure consists of repeating this method for any number of vectors, so you can slowly form an orthonormal (i.e. all unit vectors and set and all perpendicular) set. Works in (vector) spaces of any (finite) dimension.
Sorry about all the math stuff, but no one ever talks about anything interesting in the physics forum.
I would, if someone would have nice questions, too.
Maybe you’d like help us with JOODE? There are many parts that have to do with math 
the bac-cab trick is nice - I’ll remember it