Does the problem persist when you disable the Matrox board in Display properties, and run java with d3d on GF2mx?
I have almost exact configuration as my home system (P3 750, GF2MX, W2k), and I haven’t seen problems like you describe (I havent’ run your app, though =)