Hi,
after quite a lot of debugging until 2.30am last night I have possibly discovered the cause of the problem:
The Texture was the suspected culprit :o
I first checked the HelloXith3D demo and extended it to use two Canvas3D and it worked fine (just as the JOGL demo extension did).
Then I changed my own code to match the modified HelloXith3D demo step by step until they matched almost line by line but I still had my memory leak.
The only difference was that my object used a texture (mapped to 6 separate faces on a cube).
The problem occurred when the same texture (instance) was used in both displays simultaneously. When I used a different Texture on the object in the second display (Canvas3D) the memory leak was no more.
As a workaround to get two identical textures in both displays I tried to load the same texture (same by filename) twice into two Texture instances but this seems not to work. Perhaps Xith is keeping a hash of loaded texture filenames to prevent double loading. Duplicating the texture file with a different filename does work (perhaps a link would do the trick too).
@Will:
The memory leak could not be on the java side since I use default heap size and this was one hell of a memory leak. After a few seconds it was getting difficult to move the mouse to reach the shell to end the process. And there was no OutOfMemoryError thrown by the java VM.
Kudos for the Xith3D coders as I checked a lot of well written code last night. The documentation is a bit sparse but the code is well readable 
Ca$
(Starship Trooper mode: Only a dead bug is a good bug !)