Is this possible and has anyone done it?
As far as I know, it is not possible, JMonkeyEngine relies on hardware acceleration through OpenGL via LWJGL and JOGL (as an option in 2.0).
If you want a 3d engine which can do software and hardware rendering then your best bet is probably jPCT
+1
or Java3D ?
Are you sure it can work in software rendering?
Java3D do not have software rendering. It has DirectX, OpenGL (custom JNI) and JOGL renderer.