[solved] [lwjgl] [glsl] Blending & depth buffer problems

Newbie & Debugging Questions
#1

SOLVED EDIT:

My suspicions about textures not being properly mapped to fragments were correct! Using shaders provided by this (http://lwjgl.org/wiki/index.php?title=GLSL_Tutorial:_Texturing) page and then discarding fragments with alpha worked!

Vertex shader:


void main()
{
	gl_Position = ftransform();
	gl_TexCoord[0] = gl_MultiTexCoord0;
}

Fragment shader:


uniform sampler2D tex;

void main()
{
	gl_FragColor = texture2D(tex, gl_TexCoord[0].st);
	if(gl_FragColor.a < 1.0)
	{
		discard;
	}
}

ORIGINAL PROBLEM:

Hello everyone!

Yesterday I came across an issue that I first thought would be an easy work-around. When OpenGL uses blending it mixes fragments already written to the colour buffer, and rewrites fragments written to the depth buffer. Unfortunately this causes really annoying artefacts where there is plain alpha in a texture, and upon research I found out a few solutions. The first thing is to use blending like I already am and simply order rendering of transparent objects from back to front, but this seems over-complex for a problem so mundane. The second option, is to use a fragment shader that simply discards the fragment all together if it contains alpha. As I have read, the second option seems to be very simple and eliminates the need to order back-to-front as well as the need to use blending.

However…

It’s not working. I used the shader provided by this (http://www.opengl.org/wiki/Transparency_Sorting) page and many variants constructed from other tutorials, even some blatant copy & pastes. Still, discard doesn’t seem to work at all. As I understand, discard should literally cut out holes in the quads where there’s alpha on the textures, but it doesn’t. I even tried to discard any fragment that has a green value greater than 0, the “discard” command does nothing. I’m really tearing my hair out over this because not only would it solve my problem with blending, it would remove alpha fragments from the depth buffer and make shadow mapping possible.

Here is the problem:

And here’s what it should look like (accept with depth testing):

My questions are: is there ultimately a better approach to transparency sorting that will allow for shadow mapping in the future? Am I misinterpreting the use of the “discard” command? If the latter, how can I properly use it to achieve transparency sorting with out the obvious front-to-back procedure?

Thanks for reading, and sorry for the long post! I just don’t understand what to do, and tutorials don’t actually come out and explain the steps to solving this.

#2

Share your shader with us.

#3 

in vec2 coord;
out vec4 color;

uniform sampler2D tex;

void main()
{
	vec4 texel = texture(tex, coord);
	if(texel.a < 1)
	{
		discard;
	}
	color = texel;
}

This is the same one as on the OpenGL wiki, but every other variant I’ve tried fails just like this one.

#4

Are you sure the shader is bound?

Why don’t you write to gl_FragColor ? is that deprecated already?

#5 

@@if(texel.a < 1)

This is the problem. The alpha ranges between 0 and 1. And you are discarding all the colors if it has transparency.

#6

That’s what he wants: discard all fragments, to see whether discard actually works.

#7

My guess is that the fragment shader is invalid, fails to compile/link, or is not bound.

#8

The fragment shader works, if I change it to the following everything becomes red.


void main()
{
	gl_FragColor = vec4(1, 0, 0, 1);
}

EDIT: Using glGetShaderi(GL_COMPILE_STATUS) returns a clean compile too :frowning:

#9

Well, work from something that works, and slowly work towards the end result.


void main()
{
@@   if(0.0 < 1.0) discard;
   gl_FragColor = vec4(1, 0, 0, 1);
}

Is the scene empty now?

#10

Yes, does it make a difference if I write the number one as 1 or 1.0?

#11

Depends on the driver, vendor and GLSL version.

#12

Yeah, there are no errors.

If I apply textures in my Java program do fragments take on the texel’s colour, or do I have to re-map textures in the shader? Because maybe it’s determining colours before the textures have been applied, as if they were on a solid white plane?

#13

In case this is an issue with the shader loading it’s self, here is the class I made for handling shaders:


package core;

import static org.lwjgl.opengl.GL20.*;
import static org.lwjgl.opengl.GL11.*;
import java.io.BufferedReader;
import java.io.FileReader;

public class Shader
{
	private int programHandle;
	private int vertHandle;
	private int fragHandle;
	
	public Shader(String res)
	{
		programHandle = glCreateProgram();
		vertHandle = glCreateShader(GL_VERTEX_SHADER);
		fragHandle = glCreateShader(GL_FRAGMENT_SHADER);
		
		StringBuilder vertSource = new StringBuilder();
		StringBuilder fragSource = new StringBuilder();
		try
		{
			BufferedReader vertReader = new BufferedReader(new FileReader("src/res/" + res + ".vs"));
			BufferedReader fragReader = new BufferedReader(new FileReader("src/res/" + res + ".fs"));
			String line;
			while((line = vertReader.readLine()) != null)
			{
				vertSource.append(line).append('\n');
			}
			while((line = fragReader.readLine()) != null)
			{
				fragSource.append(line).append('\n');
			}
			vertReader.close();
			fragReader.close();
		}
		catch(Exception e)
		{
			System.err.println(e);
		}
		
		glShaderSource(vertHandle, vertSource);
		glShaderSource(fragHandle, fragSource);
		glCompileShader(vertHandle);
		glCompileShader(fragHandle);
		if(glGetShaderi(vertHandle, GL_COMPILE_STATUS) == GL_FALSE)
		{
			System.err.println("Vertex shader failed to compile");
		}
		if(glGetShaderi(fragHandle, GL_COMPILE_STATUS) == GL_FALSE)
		{
			System.err.println("Fragment shader failed to compile");
		}
		glAttachShader(programHandle, vertHandle);
		glAttachShader(programHandle, fragHandle);
		glLinkProgram(programHandle);
		glValidateProgram(programHandle);
	}
	
	public void bind()
	{
		glUseProgram(programHandle);
	}
	
	public void release()
	{
		glUseProgram(0);
	}
	
	public void destroy()
	{
		glDeleteProgram(programHandle);
		glDeleteShader(vertHandle);
		glDeleteShader(fragHandle);
	}
}

EDIT: It’s getting late here, I’m going in a few minutes, I’ll check back tomorrow, and thanks for all the quick replies! I’ll have to ask my programming teacher about this, it appears as though “discard” should be working properly, so I’m unsure as to what I should do next.

#14

Why are you guys talking about shaders?
Isnt this problem clearly caused by drawing order (depth buffer does not know about discards)?

#15

The shader is a possible elegant solution to the problem that would also allow for shadow mapping that includes the alpha in textures. Also, if I could avoid the heavy process of manually sorting with a fast program that runs on the GPU it’d save a lot time and processing power.

#16

Discard prevent depth writes.