Arrrgggg

[DavidYazel]

I hate it when I stare at code for hours and cannot figure out what is wrong.

I have implemented shadow volumes in the xith3d engine. The scenegraph can construct a shadow occluder for any branch in the scenegraph by examining all the geometry below it.

The occluders calculate a connectivity list and can be queried against lightsources to produce shadow volumes. The volumes are perfect because when I render them to the color buffer they are correct.

But for some reason I cannot figure out, when I render them to the stencil buffer they are not updating the buffer, or when I test against it the tests are not working. In fact if all I do is clear the stencil and then write geometry with a :

        gl.glStencilFunc(GL.GL_NOTEQUAL, 0, FULLMASK);
        gl.glStencilOp(GL.GL_KEEP, GL.GL_KEEP, GL.GL_KEEP);

The geometry STILL RENDERS when it should be masked!

In this image I show the shadow volume in red. The alpha blended square should only be drawing in the red areas.

Anyone have any ideas?

I will post the routine below.

[DavidYazel]

    /**
     * draws the shadow volumns in the render bin
     * @param bin
     */
    protected void drawShadows(ShadowBin bin) {

        final int FULLMASK = 0xffffffff;
        // determine edges

        for (int i = 0; i < bin.curSize; i++)
            determineVisibleEdges(bin.buckets[i], bin.lightsource);

        gl.glPushAttrib(GL.GL_COLOR_BUFFER_BIT | GL.GL_DEPTH_BUFFER_BIT |
                        GL.GL_ENABLE_BIT | GL.GL_POLYGON_BIT |
                        GL.GL_STENCIL_BUFFER_BIT);

        gl.glPushMatrix();
        gl.glLoadIdentity();
        gl.glDisable(GL.GL_LIGHTING); // Turn Off Lighting
        gl.glDisable(GL.GL_TEXTURE_2D);
        gl.glDisable(GL.GL_TEXTURE_3D);
        gl.glStencilFunc(GL.GL_NEVER, 0xff,FULLMASK);

        gl.glDepthMask(true);
        gl.glDepthFunc(GL.GL_LEQUAL);
        gl.glEnable(GL.GL_DEPTH_TEST);

        // (test)draw a floor
        gl.glColor3f(0.2f,0.4f,0.2f);
        gl.glBegin(GL.GL_QUADS);                                    // Begin Drawing Quads
                // Floor
                gl.glNormal3f(0.0f, 1.0f, 0.0f);                        // Normal Pointing Up
                gl.glVertex3f(-100.0f,-10.0f,-100.0f);                  // Back Left
                gl.glVertex3f(-100.0f,-10.0f, 100.0f);                  // Front Left
                gl.glVertex3f( 100.0f,-10.0f, 100.0f);                  // Front Right
                gl.glVertex3f( 100.0f,-10.0f,-100.0f);
        gl.glEnd();
        gl.glPopMatrix();

        gl.glColor3f(1f,0f,0f);
        gl.glClearStencil(0);

        gl.glDisable(GL.GL_LIGHTING); // Turn Off Lighting

        gl.glEnable(GL.GL_CULL_FACE);
        gl.glCullFace(GL.GL_FRONT);
        gl.glDepthMask(false); // Turn Off Writing To The Depth-Buffer
        gl.glDepthFunc(GL.GL_LEQUAL);
        gl.glEnable(GL.GL_DEPTH_TEST);
        gl.glEnable(GL.GL_STENCIL_TEST); // Turn On Stencil Buffer Testing
//        gl.glColorMask(false, false, false, false); // Don't Draw Into The Colour Buffer
//        gl.glStencilFunc(GL.GL_ALWAYS, 0xff,FULLMASK);
        gl.glStencilFunc( GL.GL_ALWAYS, 1, 0xFFFFFFFF );


        // First Pass. Increase Stencil Value In The Shadow

        gl.glStencilOp(GL.GL_KEEP, GL.GL_KEEP, GL.GL_INCR);

        for (int i = 0; i < bin.curSize; i++) {
            drawObjectShadow(bin.buckets[i], bin.lightsource);
        }

        // Second Pass. Decrease Stencil Value In The Shadow
        gl.glCullFace(GL.GL_BACK);
        gl.glStencilOp(GL.GL_KEEP, GL.GL_KEEP, GL.GL_DECR);

        for (int i = 0; i < bin.curSize; i++) {
            drawObjectShadow(bin.buckets[i], bin.lightsource);
        }

//        gl.glFrontFace(GL.GL_CCW);
        gl.glColorMask(true, true, true, true); // Enable Rendering To Colour Buffer For All Components

        // Draw A Shadowing Rectangl.gle Covering The Entire Screen
        gl.glColor4f(0.0f, 0.0f, 0.0f, 0.4f);
        gl.glEnable(GL.GL_BLEND);
        gl.glEnable(GL.GL_DEPTH_TEST);
        gl.glBlendFunc(GL.GL_SRC_ALPHA, GL.GL_ONE_MINUS_SRC_ALPHA);

        /*
        for (int i = 0; i < bin.curSize; i++) {
            drawObjectShadow(bin.buckets[i], bin.lightsource);
        }
        */

        // get rid of the view matrix, only have the projection matrix

        gl.glMatrixMode(GL.GL_PROJECTION);
        gl.glLoadIdentity();
        renderFrame.getView().getProjection().get(trans);
        gl.glLoadMatrixf(trans);

        gl.glMatrixMode(GL.GL_MODELVIEW);
        gl.glPushMatrix();
        gl.glLoadIdentity();


        gl.glDepthMask(false);
//        gl.glColor3f(0,0,1);

        gl.glEnable(GL.GL_STENCIL_TEST); // Turn On Stencil Buffer Testing
        gl.glStencilFunc(GL.GL_EQUAL, 0, FULLMASK);
        gl.glStencilOp(GL.GL_KEEP, GL.GL_KEEP, GL.GL_KEEP);

        gl.glBegin(GL.GL_TRIANGLE_STRIP);

        gl.glBegin(GL.GL_TRIANGLE_STRIP);
                gl.glVertex3f(-0.1f, 0.1f,-1f);
                gl.glVertex3f(-0.1f,-0.1f,-1f);
                gl.glVertex3f( 0.1f, 0.1f,-1f);
                gl.glVertex3f( 0.1f,-0.1f,-1f);
            gl.glEnd();
        /*
        gl.glVertex3f( -100f, 100f, -2f);
        gl.glVertex3f( -100f, -1000f, -2f);
        gl.glVertex3f(100f, 100f, -2f);
        gl.glVertex3f(100f, -1000f, -2f);
        gl.glEnd();
        */
        gl.glPopMatrix();

        gl.glPopAttrib();

    }

[DavidYazel]

Ok so this code draws a blue square when it should be impossible. I clear the stencil buffer, then only draw where it is not zero… yet it draws!

   gl.glClearStencil(0);
        gl.glEnable(GL.GL_STENCIL_TEST); // Turn On Stencil Buffer Testing
        gl.glStencilFunc(GL.GL_NOTEQUAL, 0, FULLMASK);
        gl.glStencilOp(GL.GL_KEEP, GL.GL_KEEP, GL.GL_KEEP);

        gl.glColor3f(0,0,1);
      

        gl.glBegin(GL.GL_TRIANGLE_STRIP);
                gl.glVertex3f(-0.1f, 0.1f,-1f);
                gl.glVertex3f(-0.1f,-0.1f,-1f);
                gl.glVertex3f( 0.1f, 0.1f,-1f);
                gl.glVertex3f( 0.1f,-0.1f,-1f);
            gl.glEnd();

[DavidYazel]

wooooot

I figured it out!

I forgot to set the GLCapabilities to give me a stencil buffer!

HAHAHAHHAHHA

Man I spent more time debugging that than it took to write the shadow volume code

[TarmTarm]

I had the same problem yesterday with the stencil buffer. I almost threw my computer out of the window before I found the solution

[swpalmer]

I always find these sorts of errors about half way through explaining it to someone else. There I am trying to describe the problem, all worked up because I can’t find the error, only to shrink away with a “never-mind” and go back to my desk.

They should just build a simple “Eliza-like” AI into IDEs so you can complain about bugs to them for a while. Half way through chatting with the AI about how stupid some API is because it doesn’t work the light bulb will turn on in your head and you can just click the close button on the chat interface and get back to work.

[aNt]

but wheres the fun in that?.. u need that shocked confused look in other peoples faces… where they have no clue what the hell u are on about… ;D

[Markus_Persson]

Haha, it’s funny because it’s true.

We’ve even put a system into place at work, where we try to explain our problems to people who are not involved in it. Most of the time you come up with the solution and/or realize you’ve missed a tiny detail while explaining it, and the rest of the time the listener can provide some very good out of the box thinking that helps you solve it.

Of course, some of the time it turns out to really be a REAL showstopper, like when SavaJe didn’t let us change the id for the 802.11 card from the default value, and that value turned out to be the only setting our access point was hardcoded to reject, for security reasons.

That one drove us very much insane until we figured it out.