You’re right, JavaFX’s rendering capabilities are indeed very limited. They don’t go beyond a simple phong shader, actually. It’s more a UI library, not a game engine.
So if you want a high level, but still customisable 3D engine, use JMonkey. But it might be less active and have a smaller community than Libgdx. The latter however is not as good as JMonkey when it comes to high level 3D, some features are missing or badly documented etc.
I personally like the non-3D related features of Libgdx more than the ones of JMonkey. Like the 2D, filesystem, sound, resources/assets API and so on.
So in the end it depends on whether your graphical demands are high. Does “retro” mean retro graphics as well?