antialiased texels for nearest-filter type textures.

taking a random texture like …

… zooming in alot …

… setting nearest filter …

… is good for a start but pretty useless when texels do not align perfectly with screen pixels, which is never happening

… things get ugly (zoomed to 33% resolution) :

a straight foward fix to that is using lots of msaa-samples and running sample-shading. very expensive thing to do, not fast, uses much memory, way too many samples evaluated but still … as a reference, this is nice ! 8x-msaa-sample-shading :

so here’s a shader i use for textures and screen-quads with distorted texture-coords to smooth out texels - fast, but not blurring like a linear-filter would do. no msaa used, result looks like :


there is no big difference to the 8x-msaa reference, actually it looks a bit smoother.

fragment shader :

#version 400

uniform sampler2D tex;

in      vec2      st0;
out     vec4      frag0;

vec4 textureLinearAA( const in sampler2D tex, const in vec2 st )
{
  float miplevel = textureQueryLod(tex,st).y;
  
  // minification (miplevel > 1.0) is best sampled by a standard linear sampler.
  // switching to linear at > 0.0 would be faster but fading generates 
  // sharper pixels without many artifacts.
  if(miplevel > 0.999) return texture2D(tex,st);
  
  const float aa = 0.8; // 0.6 matches close to 8x-sample-shading. 0.0 = no AA
  
  ivec2 tex_dim     = textureSize(tex,0);  // texture-dimension could be provided by uniforms for better speed
  vec2  inv_tex_dim = 1.0 / tex_dim;       // ^
  
  vec2  fw   = fwidth(st);
- vec2  fwtd = max(fw.x,fw.y)*tex_dim;
- float edge = min(min(fwtd.x, fwtd.y)*aa, 0.5);
+ vec2 edge = min( fw * tex_dim, vec2(0.5) );
  
  vec2  dd  = st.xy * tex_dim + 0.5;
  vec2  p   = vec2(floor(dd)*inv_tex_dim);
  vec2  f   = smoothstep(0.5 - edge,0.5 + edge, fract(dd)); 
  // vec2  f   = fract(dd);
  // fract(dd) instead of smoothstep(fract(dd)) would look like a default linear-filter.
  
  // might put that outside as well.
  const ivec2 gatherOffsets[4] = ivec2[4]
  (
    ivec2(0,0), ivec2(1,0),
    ivec2(0,1), ivec2(1,1)
  );
  
  vec4 _r   = textureGatherOffsets(tex, p, gatherOffsets, 0);
  vec4 _g   = textureGatherOffsets(tex, p, gatherOffsets, 1);
  vec4 _b   = textureGatherOffsets(tex, p, gatherOffsets, 2);
  vec4 _a   = textureGatherOffsets(tex, p, gatherOffsets, 3);
  
  vec4 _rgba = mix( mix( vec4(_r.x,_g.x,_b.x,_a.x),
                         vec4(_r.y,_g.y,_b.y,_a.y), f.x ),
                    mix( vec4(_r.z,_g.z,_b.z,_a.z),
                         vec4(_r.w,_g.w,_b.w,_a.w), f.x ), f.y );

  // magnification
  if(miplevel < 0.0001) return _rgba;
  
  // mix of antialiased baselevel and next linear-filtered miplevel.
  return mix( _rgba, texture2D ( tex, st ), miplevel);
}

void main(void)
{
  frag0 = textureLinearAA(tex,st0);
}

pretty much a drop-in replacement for [icode]texture(tex, st)[/icode]. changing it from RGBA to RGB (and saving some gpu-power) should be straight forward. if you have any suggestions or improvements - i’d love to know :slight_smile:

there is just one thing to take care of, since this is a magnification filter, it uses a default texture(tex, st) lookup when scaling down. so even if the effect it nearest-filter-like you should set the samplers to linear-filtering.

that way things stay smooth when scaling down and crisp when scaling up.

o/

How does it look with moving content?

same as it looks with msaa and sample-shading.

try comparing with

GL11.glEnable(ARBSampleShading.GL_SAMPLE_SHADING_ARB);
ARBSampleShading.glMinSampleShadingARB(1.0f);

and

GL11.glDisable(ARBSampleShading.GL_SAMPLE_SHADING_ARB);

ofc

there are other issues which show with sample-shading enabled tho’. derivates along triangle edges for instance. nothing we can do about.

fixed a bug which showed up after using textures with extrem aspect ratios.

see line 23,24 (delete) and 25 (insert).

o/