EXT_packed_depth_stencil and fragment shaders

Newbie & Debugging Questions
#1

Hi all,
I honestly dont know where else to post this, since it probably is a n00b question, it has its home here. In any case…

I am using EXT_packed_depth_stencil to render to DEPTH24_STENCIL8 texture target and I wish to access the stencil value from fragment shaders in order to do some funky shadow volume/map mixing of their ambient factor.

Anyway, how can I access the stencil? The specs for EXT_packed_depth_stencil say that a sampler2D has an undefined operation on a DEPTH24_STENCIL8 format, only sampler2DShadow has defined operations. How does the sampler2DShadow deal with stencil values?

Any help on this matter is appreciated.

DP :slight_smile:

#2

Stencil values are not accessible from fragment shaders (not sure if they’ve changed this in DX10). You can try using the alpha channel to accomplish stencil like functionality.

#3

The idea was to access the stencil buffer to unify shadow maps/shadow volumes in their ambient factor so that I can mix and match them. As you probably know, the ambient factor of shadows in maps is just a constant you add at the end of the lighting equation, whereas in shadow volumes, the alpha of the full screen quad changes the ambient factor of the shadows.

I was hoping to be able to access both the shadow map and the stencil buffer from fragment shader so that I can treat both colour values the same…

Any ideas for this?

DP

#4

Render a big-white quad using the stencil to a texture, then sample the texture?

It’s slower, but… it’s obvious you can’t access sstencil-values from the fragment-shader, as the fragment-shaders are not run if the stencil-test fails. Besides… stencil-buffers are plain bits, and i don’t think there is a sample-function for bits out there :slight_smile: (at least not in DX9 cards)

#5

Umm, your misunderstanding the point there riven…

I am using EXT_packed_depth_stencil as a texture format for the stencil buffer to be rendered to as a target. The texture should hold the bits of the stencil. When passing this texture to a fragment shader as a sampler2D, you cannot access the stencil information through any current means. Your slightly right in the sense that there isn’t a sample function out there, but the fragment shaders that I need will run as they are not stencil tested.

I was hoping to pack the shadow map inside the 24bits of the texture and the stencil shadows in the other 8 bits and use 1 texture for both shadow maps and shadow volumes. But It seems like im going to have to forget about EXT_packed_depth_stencil for the time being as it doesn’t give me enough functionality to work with.

DP

#6

[quote=“darkprophet,post:3,topic:29190”]
What you’re trying to do sounds wrong. Shadow maps contain depth values, stencil buffers contain integers in the range 0-255. There’s no color value.

[quote=""]
Not exactly. The lighting equation looks like this in Marathon:

half3 color = (normalMapZ * AMBIENT_PRODUCT + diffuse * shadowTerm * DIFFUSE_PRODUCT) * decal.rgb;

Then you have specular, fog, etc.

[quote=""]
There’s no full screen quad in any shadow volume algorithm I’ve seen.

[quote=""]
That’s easy. Render an ambient only pass first (that also inits your depth buffer, nicely and fast), then render each light using either shadow maps or stencil volumes or both, without worrying about ambient (the shader should only add more light to the scene). :slight_smile:

#7

Yup, i know :). I would render into the depth attachment from the light’s perspective. Then from the camera’s perspective render the to the stencil attachment and send to the fp shader. The sampler2DShadow would then figure out which pixel is in shadow regarding the SM. With the stencil, any stencil bit/pixel thats not 0 is in shadow. You then combine the values of the stencil (clamp to between 0 and 1) and add to the SM value. You then obtain a final value to say whether this pixel is in shadow or not.

Umm, there is. After you have the stencil buffer ready and before you render all all shadow volumes, you render a fullscreen quad so that you store it to give some ambient after wards.

[quote]The final section of this function draws one blended rectangle over the whole screen, to cast a shadow. The darker you make this rectangle, the darker the shadows will be. So to change the properties of the shadow, change the glColor4f statement. Higher alpha will make it more black. Or you can make it red, green, purple, …!
[/quote]
Edit: Completed one of the quote thingies to make it look nice

DP

#8

I know that some people do it this way, but it’s wrong and really not needed.

It’s wrong, because it doesn’t work correctly on surfaces with specular highlights. You can’t take a surface with specular and just darken it. The specular shouldn’t have been there at all.

It’s not needed, because an ambient-only pass is easier, faster and more flexible (different surfaces may have different ambient colors).

#9

I will be very surprised if rendering the entire scene with colours and depth as an ambient pass is faster than rendering a fullscreen quad with alpha. But I guess your right about the specular highlights…

I still think that my way is faster and better than both, but its a shame that I need to render the stencil texture to another texture to get values…

#10

Rendering an ambient pass first is considered good practice and may actually make subsequent passes faster (because the depth buffer values are already there).