I just reviewed and tested Kenji version of vecmath modified by Artur Biesiadowski and only I can say - this is good and compatible with Xith3D engine.
I was able to compile engine and the apps based on it without any problems.
BTW, to make it running with my apps I had to add explicit declarations of serialVersionUID to all the serializable classes [except of GMatrix, GVector (incompatible!) and TexCoord4d] to make it really working. Also one incompatibility is that some objects do not implement Cloneable, and one may encounter Java3D->Xith3D porting problems because of that.
So, my suggestion is to switch at least to this library because of it is open source. And, of course, spend some short time to make it compatible in serialization [99% done] and cloneability [todo].
Yuri