http://docs.oracle.com/javase/7/docs/technotes/guides/2d/flags.html#opengl
Java2D isn’t strictly software based rendering. There is a lot of hardware acceleration that can occur (if Java2D determine the most optimal manner in which to render them is via hardware acceleration.) Java can even be using OpenGL or DirectX in the backend to do its rendering (hence the advantage of abstraction)
There are places where the abstractions sucks though (pixel shaders, etc which are platform specific. Java is looking to develop its own shading language sometime soon though IIRC)
The Java2D games you converted to OpenGL likely performed poorer because Java2D knows more about its environment and how to (and can) properly dynamically configure itself for its environment. You cannot do that when you have a static native dependency on OpenGL. Java2D engineers also know a lot more than you do about OpenGl and DirectX and how to use them properly (no offense, but the Java2D developers are seasoned engineers.) Java2D also probably made more efficient use of the OpenGL rendering subsystem (or switched to DirectX due to better driver support)
OpenGL has a million ways to do the same thing, half of those ways are deprecated, and only few of them are actually the good way to do it.
This is really why you shouldn’t be using a native dependency like OpenGl for rendering if you don’t need it (outside of the fact that you now have a native dependency to carry around with you.)
The abstraction can suck, sometimes the rendering API seems too general purpose, but if you can pull it off, there is no reason in the world to opt out to explicitly use a graphics layer.