What does LogCat say?
There are two likely scenarios:
- The gl11 object is null and you are getting a null pointer exception. If useGL20 is enabled, then gl10 and gl11 will be null. So the proper code is to do this:
//enable stencil testing
Gdx.gl.glEnable(GL10.GL_STENCIL_TEST);
//set the stencil clear color -- it is zero by default
//so this line is just to be safe
Gdx.gl.glClearStencil(0);
//clear the stencil buffer
Gdx.gl.glClear(GL10.GL_STENCIL_BUFFER_BIT);
If you need something specific to GL20 or GL10 you should use [icode]Gdx.graphics.isGL20Available()[/icode]
- You didn’t enable the stencil buffer during your application initialization:
Android:
AndroidApplicationConfiguration Configuration = new AndroidApplicationConfiguration();
Configuration.stencil = 8; //stencil buffer size
initialize(new Game(), Configuration); //pass it as parameter
Desktop:
LwjglApplicationConfiguration Configuration = new LwjglApplicationConfiguration();
Configuration.stencil = 8;
new LwjglApplication(new Game(), Configuration);
Regarding your problem. Scissor stack does not use the stencil buffer. What kind of masking do you need? Arbitrary shapes? You can always use images and mask via a shader; this allows for anti-aliasing (unlike stencil testing).
Here is an example of using a shader to mask arbitrary shapes, with a mask texture:
If you’re new to shaders, you should start from the beginning:
And here is an example of computing a mask in the shader, which means it is resolution-independent:
http://www.badlogicgames.com/forum/viewtopic.php?f=11&t=8540&p=38862&hilit=rotated+rectangle+mask#p38862