[Solved (mostly)] Dealing with billboards and Z buffers

General Discussions
#1

I’ve been playing around with some things trying to maybe get a 3D engine made. Not really a game yet, just messing around. Right now I have a small map and a guy who walks around. The map is full 3D but the character is a 2D sprite which is rotated to face the camera. Unfortunately, this is giving a weird result which you can see here (try to ignore the awful awful art at this time, I am no artist):

The only thing I can think to do about this to pass some uniform to the fragment shader to force the Z buffer to be something else, but I’m not exactly sure how to do this or if it is the best solution. Anyone have any experience with this sort of thing? I’m mostly going this route because I don’t know anything about 3D animation and it seemed a little easier to get started with 2D sprites for characters.

#2

Can you turn off depth testing before drawing the sprite? Otherwise your sprite is always going to get obscured by the models it intersects. …but then your sprite won’t be able to pass behind anything… Hmmm…

Alternatively prevent the sprite from intersecting world objects? I.e. collision detection??

#3

The problem is certainly due to the fact that the character is facing the camera.
But the character should stay perpendicular to ground,
so I think you should try to rotate the character only on the Y axis and cancel rotations on X and Z axis.

#4

In fragment shaders, you can set the fragment’s depth value via the output variable gl_FragDepth. So if you set every fragment of the sprite to have the depth of the centre (or base) of the sprite then it will all get rendered in front or all get rendered behind but this does mean you would have to work out the depth component yourself. Not so simple, especially if you are doing perspective projection.

As I understand it (and the second part of this may be flawed since I’ve never actually done it myself). You take the position of the point whose depth position you actually want to use (what this point actually is should depend a bit on how you’ve done you’re bill boarding but probably either the base or the centre). You will be storing it as a 3d vector but it’s actually a 4d vector with the w component = 1. So call that V. Then you need the modelview and projection matrices times together to make the model-view-projection matrix. Call that mvp. Now once you’ve done that it is actually quite simple.

V’ = V * mvp
Depth = v’.z / v’.w

And if you are only using an orthographic projection then you don’t need the perspective division (although it wouldn’t hurt to do it). Then, as you said, you can pass that up to your fragment shader and set all the sprites depth components to that.

#5

Thanks for the help. This is what I was planning on doing, but I wasn’t sure if there was a better way.

I had tried this as well but it made the characters just look like they were squished. It seems that as long as no X or Z rotations happen once the scene starts the illusion works and it looks like the guy is standing properly. At least to me, I’ll see what others think.

#6

i think you got the answer…

In my 3D engine i have two stages of drawing. I do my 3D models first then i switch to 2D elements to do my HUD. effectively you can do the same thing here.

Note: doing this i do remove the depth testing. This would work well if your 2d figure was always going to be the same distance from the camera looking at him. otherwise you may still want to treat your char. as a 2D element in the 3d world.

#7

Thanks for the advice. I don’t think that will work for me since I am trying to allow the character to move behind 3D objects. Think of it as being like in the game Paper Mario where the world is 3D but the people are 2D.

The above approach where I set the z buffer directly from a uniform isn’t going to work. Doing this gives every fragment the same value in the z buffer, which effectively does nothing at all since the quads were rotated to be flat with the camera anyway. I’ve tried changing the point where the buffer is calculated from to be slightly ahead of where the sprite is drawn but haven’t been able to find satisfactory results. I’ll post in this thread again if I am able to solve this problem.

#8

Not sure if this will help but basically what I did for rendering billboarded sprites in a 3d world.

Disable the depth mask and render in back to front order from the camera. This requires sorting the sprites by distance from camera which is actually very easy and fast even for thousands of sprites.

Although, looking at the picture it looks like the sprite is clipping into the object. Not sure but it looks like you need to render the sprite without depth testing of sorts so it will always render it in front of anything. What order are you rendering the world? Is is world then sprite or sprite then world?

#9

The trouble is that I don’t want the sprite to always show up if he is behind something. The issue becomes worse if the camera is rotated downward. I made a short video to show what I am talking about.

As you can see, it looks like he should be in front of the cubes, but sometimes he isn’t. I think what I need to do is somehow linearly move the depth from the bottom of the sprite to the top, based somehow on the X rotation of the viewing angle. I’ll keep working away on it and update this thread if I have any problems.

On the plus side, I do have the camera rotations working for the character, which is fun.

#10

Are you using the same shader for the character and the other geometry? I meant to use different shaders. If that isn’t it could you elaborate a bit on why it isn’t working.

#11

I am using two different shaders for sprites and general geometry. The reason I cannot simply pass a uniform float to the fragment shader is because this will not have the desired effect. As you can see in the video above, when the camera is rotated about the x-axis the guy start to intersect with the cubes when it stands too close in front of it. The more rotation happens, the worse the problem is. Since sprites are rotated to face the camera directly, they already have constant z coordinates in clip space. Another way to think about this is that the sprites themselves are not rotated at all, only the rest of the world is. So the cube, which is suppose to be facing “up” is actually tilted forward. This means that the values stored to the depth buffer from the cube are increasing as you move up along the face. In contrast, the sprites depth buffers values are constant, so at some point the cube will be in front of the cube. Passing one uniform constant or another won’t change the fact the the problem is arising from the cubes slant, not the sprites.

I hope that clears up the issue. I myself was also confused because I was thinking that the sprite was being rotated (which it is in world space, but not in camera space). The only solution I can think of right now is to linearly interpolate between two desired depth buffer values (based on the tangent of the angle of rotation about the horizontal), one at the base and another at the top. Luckily the fragment shader can do this for me so I don’t think it will be too difficult to implement. If I get something working I’ll update this thread.

#12

Oh I see now. I thought you were only getting the sprite to face the camera in terms of yaw but you’re actually going for the full nine yards. In that case I suppose it’s actually the opposite you want to go for. I would have thought that the way I thought you were doing it would look better but your video certainly looked good. Actually I’m pretty sure I’ve heard of games going half way and having the pitch of the sprite half billboarded and half normal.

Back on topic: I think you’re proposed solution (which will work as far as I can tell) would be easier if you did your billboarding in the vertex shader. That way you can save on the uniforms and make use of the varying variables which interpolate pretty quickly.

#13

Ehh more complications. I migrated all my sprite code over to a geometry shader which simplified things quite a bit but I’m, still unable to actually get the z buffers correct. At first I was getting very weird results but I realized that this was likely due the fact that I was linearly interpolating depth values which is a nonlinear space. So now I’m trying the following, but it seems like I’m still not quite getting what I want:

  1. Find the cosine of the angle of rotation between the sprites Y-axis and world Y-axis. I pass in the camera’s rotation matirx and do the following calculation:

dot(Ry, y)

Since the sprite is rotated to always face perfectly forward, y= (0, 1, 0) and this dot product is actually mostly zeros so the result is just R[1][1].

  1. Get the sine instead. That’s easy: sin(t) = sqrt(1 - cos^2(t)).

  2. Set the desired depth to be the normal depth at the sprite bottom, and + sin(t) at the top (since the hypotenuse is just the sprite’s height and right now they are all height 1). Then just pass this over to the fragment shader who will happily interpolate. Inside the fragment shader I redo the perspective divide so as to not mess up the non-linearity. (I forgot to do this at first and was confused for a good while).

Unfortunately I’m still not getting results I want. Now the sprites top sticks out from under things it shouldn’t. I know why this is but I’m not sure what the solution ought to be.

#14

First of all, I’ve never used geometry shaders (keeping compatibility with OpenGL 2.0) so I’m not entirely sure how they work. Buti have some problems with what you’ve said. So you’re taking the cosine of the rotation matrix up vector and the sprites up vector but you seem to be doing that after billboarding it. So those vectors are just going to be opposite right? But then that should just be -1. I can’t see how that would ever be 0. Also I’ve written down a few diagrams and I can’t see how the trigonometry would work. Not the identity stuff, the rest of it.

Since I can’t understand what you’re doing, I will say how I would do it in a vertex shader. After a hastily drawn diagram, I’ve decided the best way to do it would be to construct a rotation matrix client side than pass that to the vertex shader, multiply the vertex to get the position but pass the in-multiplied z value (and as you said the w value as well) as a varying value and set that to the depth value.

#15

Thanks so much for helping me talk through all this. Here is a diagram that I think will explain what I’m trying to do:

I wouldn’t worry about the geometry shader. Basically all it does is take a vertex and emit a quad to draw the sprite on. The main reason this is simpler is because it doesn’t need to care about the camera’s rotation component since it just undoes it anyway, so I don’t have to pass that along the pipeline. Aside from a simple pass-through shader I think it’s one of the simpler ones to implement.

#16

I assumed that was how the geometry shader worked - just wasn’t sure. In fact I considered suggesting doing the billboarding there in the first place but since I wasn’t sure …

It seems we’re doing it two different ways. My way is just to use the depth value of the vertex before it was billboarded. Your way is to project the vertex along the camera’s forward vector until the base. I would say my way was simpler computationally, conceptually and practically. But technically due to the perspective projection, it is slightly inaccurate (more so the greater the rotation), which I’m only realizing now thanks to you. So your way is more accurate (if this can be such a thing) but I think it needs the tiniest bit of refinement.

Imagine if the camera was rotated the other way than it is in your diagram, then you would actually have to subtract the change in depth rather than add it right? A couple of weeks ago, it would have taken me several hours to solve this problem, but luckily for you I’ve recently had to solve it to implement a touch screen rotation system (where one finger works as a pivot point and the other drags the view around). The answer is to multiply the value by the sign of the cross product of the two vectors. I’m pretty sure GLSL has a sign() method. If not then it will be more difficult.

On to why it isn’t working. I’m afraid the only thing I can think of is that the vectors aren’t normalized, but I’m sure they would be. It might help if you did some more testing to see if there is any pattern to it not working. A tip on that front - I once modified my shader to set the fragment colour to a sort of greyscale of the depth value. (vec4(depth, depth, depth, 1); ). And rendered that on one half of the screen and the regular image on the other. I found it helps.

One other thing to note:
|A X B| = |A|.|B|.sin(theta)

Where || is magnitude, X is cross product and . is regular scalar multiply. Sorry to patronize but I’ve had some horrific misunderstandings before when it comes to mathematical notation. My point being that this is probably easier than messing around with trigonometric identities. Especially if you’re going to take my suggestion above.

Edit: Sorry to take so long. Typed out the reply, forgot to press post, made some tea, ended up talking for about an hour, came back, realized I am an idiot (funny how I keep forgetting that), wrote this apology.

#17

Thanks for the tip about outputting depth in the fragment shader. That will definitely help. After looking at my own diagram again I believe I am actually doing the computation wrong. You are correct, I want the depth value from before the sprite was billboarded. This actually makes the most sense because the desired depth value should still come from a point the sprites’s height away from it’s base. The value I was using was too large which is why his head was sticking out of the box when he was standing behind it. Here is an image which I think shows what you are saying:

So yes, the vectors need to be normalized. Fortunately this is easy to fix, as long as you remember to unlinearize in the fragment shader (is “unlinearize” a word???). I have a habit of writing up replies in this thread before trying to program them so it could be that I’m jumping the gun once again. Hopefully not!

Also, you may be interested in the way that the billboarding happens. Since all the modelview matrix for a billboard does is translate and “unrotate” you can simplify the calculation quite a bit. Here is basically what the geometry shader does. I’m using block matrix notation here so if it’s a little confusing let me know:

From there I just generate 4 verticies at (-0.5, 0, 0), (0.5, 0, 0), (-0.5, 1, 0), and (0.5, 1, 0) and multiply those by the matrix A.
This puts the sprite exactly where I want it (at location ‘s’ in world coordinates) with no rotation!

P.S. I don’t feel patronized but you should know that I am working on a PhD in mathematics so I do understand the notation pretty well. :slight_smile: Still, you never know who you’re talking to on the internet so there’s no reason you would know that. ;D As always, it’s nice to have a person to bounce ideas off. (It says that this thread has been read over 600 times at this point… anyone else want to chime in?)

#18

See that’s odd because I’m tending towards siding with your original method now. But yes, your new diagram does show what I’m saying - as long as “just right” would form an arc with the top of the sprites top with the centre at the spites base. However, as you yourself said, I think the proof will be in the pudding. ie If it works. And don’t worry - I think everyone does that.

That is an interesting method. Thanks. I’ve never seen this “block matrix notation” before but I think I understand what it is.

[quote]I am working on a PhD in mathematics so I do understand the notation pretty well
[/quote]
Now I feel very stupid for telling you about the cross product. But good to know. I myself am working on an mathematics A-Level (among other things). An A-Level is a pre-degree type qualification if you’re not from the UK.

#19

I’ve got it working by using the technique discussed above. It still isn’t perfect, but it’s good enough. There is still a bit of undesirable behavior when sprites are too close to 3D objects or if the camera is at some extreme angle, but overall it works well enough for my purposes. As another poster said above, I can keep this from happening by implementing collision detection.

Thanks for all the help. I’m going to clean up the code and then I’ll post it in case anyone else wanted to see it.

#20

As promised, here is the code for the proper billboard shader:


////// VERTEX SHADER //////

#version 330

in vec3 position;
in vec2 texCoord;
in vec2 size;

out vec2 texCoord0;
out vec2 size0;

void main() {
	gl_Position = vec4(position, 1);
	size0 = size;
	texCoord0 = texCoord;
}

////// GEOMETRY SHADER //////

#version 330

layout (points) in;
layout (triangle_strip, max_vertices = 4) out;

uniform mat4 projectionMatrix;
uniform mat4 cameraMatrix;
uniform float scale = 6.5;

in vec2 texCoord0[];
in vec2 size0[];

out vec2 texCoord1;
out float depth1;

void main() {
	
	mat4 modelMatrix = mat4(1);
	modelMatrix[3] = gl_in[0].gl_Position;
	modelMatrix[0][0] = scale * size0[0].x;
	modelMatrix[1][1] = scale * size0[0].y;
	mat4 nonBillboardMatrix = projectionMatrix * cameraMatrix * modelMatrix;

	modelMatrix[3] = cameraMatrix * gl_in[0].gl_Position;
	mat4 billboardMatrix = projectionMatrix * modelMatrix;	

	// Emmit the verts

    gl_Position = billboardMatrix * vec4(-0.5, 0.0, 0.0, 1.0);
    texCoord1 = vec2(texCoord0[0].x, texCoord0[0].y + size0[0].y);
    depth1 = ((gl_DepthRange.diff * nonBillboardMatrix[3][2]  / nonBillboardMatrix[3][3]) 
    			+ gl_DepthRange.near + gl_DepthRange.far) / 2.0;
    EmitVertex();
    
    gl_Position = billboardMatrix * vec4(0.5, 0.0, 0.0, 1.0);
    texCoord1 = texCoord0[0] + size0[0];
    EmitVertex();

    gl_Position = billboardMatrix * vec4(-0.5, 1, 0.0, 1.0);
    texCoord1 = texCoord0[0];
    depth1 = ((gl_DepthRange.diff * (nonBillboardMatrix[1][2] + nonBillboardMatrix[3][2]) 
    	/ (nonBillboardMatrix[1][3] + nonBillboardMatrix[3][3])) 
    	+ gl_DepthRange.near + gl_DepthRange.far) / 2.0;
    EmitVertex();

    gl_Position = billboardMatrix * vec4(0.5, 1, 0.0, 1.0);
    texCoord1 = vec2(texCoord0[0].x + size0[0].x, texCoord0[0].y);
    EmitVertex();

    EndPrimitive();
}

////// FRAGMENT SHADER //////

#version 330

in vec2 texCoord1;
in float depth1;

uniform sampler2D sampler;

void main() {
	vec4 color = texture(sampler, texCoord1);

	gl_FragColor = color;
	gl_FragDepth = depth1;
	if(color.a != 1)
		discard;
}

As I said above, this still doesn’t work perfectly but it’s good enough for my needs and much better than what I had before!