The community was pretty helpful the last time I posted so I’ll give it another go with this other problem I have.
I’m still working on a topdown game (orthographic projection) using deferred shading.
I render my diffuse scene to an FBO then render the same secne with normal texutres to an other FBO then I bind these 2 FBOs and render the light meshes to create the final scene.
The problem is shown on this picture:
The black circle is the player, the blue lines are the shadow geometry, the light grey area inside it is the “problem area”. It is part of the shadow but I’d like it blocked by the player as shown on the “Desired effect” part of the picture.
Since both the diffuse and the normal FBOs are basically one-one textures and the shadows are rendered over them in the final phase I couldn’t figure out a way to fix this.
Basically I want my player to be normal mapped and not have shadows on it’s texture.
I hope I managed to describe my problem well enough.
Thanks in advance for any help given!
Essentially you have to render your player writing a specific value (i.e: 1) to the stencil buffer. Then, later in the final phase, you will render only where you don’t have that stencil value (1).
I have never really used stencil or depth buffers so could you help me out with some pseudo code please?
// Diffuse FBO
diffuseFBO.begin();
drawrMap();
drawEntities();
diffuseFBO.end();
// Same for Normal FBO
// ...
// Stencil
// Disabling color and depth writing
glColorMask(false, false, false, false);
glDepthMask(false);
// Clearing the whole buffer with 0
glEnable(GL_STENCIL_TEST);
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);
// Draw the map with 1 stencil value
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilMask(0xFF);
drawMap();
// Draw the player with 2 stencil value
glStencilFunc(GL_ALWAYS, 2, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilMask(0xFF);
drawEntities();
// Rendering the final scene
// ??? :(
diffuseFBO.bind(0);
normalFBO.bind(1);
renderFinalScene();
I made a little pseudoish code based on the article you linked but I’m not sure if it is even good not to mention a don’t know what to do exactly when I’m rednering the final scene.
It looks like your player then should not be a sphere (as it seems in your image) but rather a cylinder.
That would have the desired effect, since no two faces on the top of the cylinder can be one front and the other back facing the light.
Another option: If the only shadow receiver is always ever going to be a plane (your ground) then you can simply use projection shadows without having to build a shadow volume or doing with depth or stencil buffer: ftp://ftp.sgi.com/opengl/contrib/blythe/advanced99/notes/node192.html
(Demo of projection shadows with LWJGL 3)
I see. In this case you do not have an orthographic projection, because there is nothing to “project” (i.e. 3D -> 2D), since you are completely in 2D.
In this case, I’d definitely stay with @elect’s advice to stencil-out your player. I am wondering though, how you build your shadow “polygon.”
Could you elaborate on how to “stencil-out the player” with a little code snippet maybe?
Also I’m using LibGDX with Box2D and I use it’s raycast methodto build a mesh.
If you were refering to the player’s shadow it’s 2 tangent lines on each side of the circle.
The problem here is that the stencil is set to 1 even where the alpha value of the texture is 0 but that aside, the texture got masked completely anyway.
Anyone have any solution for that?
glStencilFunc(GL20.GL_NOTEQUAL, 0, ~0); or glStencilFunc(GL20.GL_EQUAL, 1, ~0);
instead
glStencilFunc(GL20.GL_NOTEQUAL, 1, ~0);
Because the stencil test has to pass when different from 0 or equal 1
You also want to combine it with the alpha test in order to render properly the transparent texture parts
Anyway, don’t underestimate setting up a small sample, because it gives you the possibility to test all of this (in all the infinity possibilities you want) in a controlled scenario. Once you are confident you can debug it in the future by yourself because you know exactly how it works.
I may not understand what you are saying, but if I use that stencil function then it just inverts my original picture.
Not to mention the player itself still has shadows on it which was the original problem I tried to avoid. ???
// Diffuse FBO
diffuseFBO.begin();
drawrMap();
drawEntities();
diffuseFBO.end();
// Same for Normal FBO
// ...
// Rendering the final scene
diffuseFBO.bind(0);
normalFBO.bind(1);
renderFinalScene();
In the ‘renderFinalScene()’ I loop through all the visible lights’ meshes and render them (these are the lit areas) calculating the amount of light each pixel has based on the normal and diffuse FBOs’ textures.
So basically both the map and the player are on the same texture (diffuse FBO’s texture).
[quote]Draw map normally, then clear the stencil buffer to 0 and render the player with 1
glStencilFunc(GL_ALWAYS, 1, 0xFF);
Finally render the final scene checking for stencil != 1
glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
If you don’t care anymore about the stencil values, skip stencilOp and stencilMask
Anyway, you play with a short sample to get confident with stencil
[/quote]
Since it looks you don’t want to experiment/understand how it works on a simple scenario and try the hard way, try to proceed step by step to detect where it gets different than expected:
clear the stencil buffer after the map, before the player, you should get the player shadowed as mentioned by you in the first post
renderMap();
glEnable(GL_STENCIL_TEST);
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_ALWAYS, 1, 0xFF); // Set stencil to 1
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); // don’t touch anything
glStencilMask(0xFF); // write to stencil buffer
renderPlayer();
glDisable(GL_STENCIL_TEST);
final();
check if player rendering get skipped, because nothing has stencil == 1
renderMap();
glEnable(GL_STENCIL_TEST);
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_EQUAL, 1, 0xFF); // pass if stencil == 1
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); // replace if depth and stencil pass
glStencilMask(0xFF); // write to stencil buffer
renderPlayer(); // nothing pass
glDisable(GL_STENCIL_TEST);
final();
try to get it right
renderMap();
glEnable(GL_STENCIL_TEST);
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_ALWAYS, 1, 0xFF); // set stencil to 1
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); // replace if depth and stencil pass
glStencilMask(0xFF); // write to stencil buffer
renderPlayer(); // player rendered and written 1
glStencilFunc(GL_EQUAL, 0, 0xFF); // pass if stencil == 0
glStencilMask(0x00); // don’t write to stencil buffer
// glStencilOp is redundant, we want to keep replacing if depth and stencil pass and keep otherwise
final();
glDisable(GL_STENCIL_TEST);
I’ve been messing around with stencil writing a bit and I think you misunderstood how I render my final scene (or I still don’t get stencil afterall )
So if I do what you suggest then I’d first draw my diffuse scene to the FBO while filling the stencil buffer.
This way I’d get a stencil buffer full of 0’s everywhere excpect at the player’s position where it would be 1’s.
So when I render my final scene combining the 2 FBO textures I use a 3rd FBO to draw it to.
If I use the glStencilFunc(GL_EQUAL, 0, 0xFF); that means I only draw a pixel where the stencil value is 0 and since the player has a value of 1 it will not be drawn at all.
The main problem here is that I don’t just render the shadows over the scene, I create a completely new one based on the diffuse and normal FBO textures.
