I am currently using 6 sample FSAA to smooth out the drawing. This largely works well in more solid shapes like filled elipse or rectangles, but for thin shapes like 1 pixel wide rectangles the output is still very jagged. All drawing is 2D using an ortho projection. When examined closely a 2 pixel line will alternate between 2 solid pixels and one solid and 2 gray pixels. The problem is that the result looks jagged ebcause the alternating is in large bands rather than being continiously variable. It looks more like non-AA drawing smeared, rather than real AA drawing. Using Java2D the gray levels would be continuously variable between black and clear to approximate the amount of coverage a theoretical line would cover that pixel. Is there any way to get this more accurate view of AA using OpenGL for edges? On 1 pixel lines I have seen the whole line disappear rather than going to gray. The result is not much better than non-AA drawing.
Check out this link:
Antialiasing Points and Lines:
http://www.opengl.org/resources/tutorials/advanced/advanced97/notes/node59.html
I have read those sections in the programmer’s guide. I am using scaled 2D drawing. I render a quad say 1000x10. As the scale gets smaller the size of the primitive becomes 500x5 then 100x1. At this point he drawing is almost not being antialiased. I get primitives appearing and disappearing depending on whether they fall on whole pixels or not. I have GL_POLYGON_SMOOTH enabled, and 6 point multi-sampling enabled. For quads that are nearly vertical or horizontal the result is pretty jagged.
Try disabling the polygon smooth thingie. I’m not sure they’re that good supported. Just go with the fullscene AA.
Alternative you could do your own AA by rendering to a larger buffer and scaling down to screen. Perhaps that would yield a better result in your case? (At the expense of more vidmem ofcourse, but the normal AA mode also burns vidmem so that’s a mood point I guess :))
There’s a chapter in the new GPU Gems 2 book:
Fast Prefiltered Lines
Eric Chan and Frédo Durand (Massachusetts Institute of Technology)
It talks about techniques for filled polygons too.