One of Java3D’s strengths should be its scalability, so maybe you’re experiencing another problem which is not caused by Java3D but perhaps by your use of it.
That said, you could try Xith3D, which is much like Java3D but specifically made with games in mind. Another option is the excellent and feature rich jME.
I guess which is fastest is still a matter of debate and probably depends mostly on what you do with it. I personally like jME, but that’s mostly a matter of taste, not because the other libs are of lesser quality.
JOGL and LWJGL are just OpenGL bindings (well, LWJGL binds to other libs as well but since we’re talking 3D graphics…), and don’t provide more functionality than bare bones OpenGL. So if you want to use that, you’ll have more control but you’ll need a lot more knowledge and to put a lot more effort in to get the high level functionality you get from Java3D, Xith and jME. Again, it depends on what you want to do.