Since hardware acceleration of Java2d supports bitmask transparency shouldn’t it be possible to use it in OpenGL too?
I can’t seem to find anything indicating this.
(I understand there are 2 kinds of “bitmask” transparency - one is assigning a special color as transparent, the other is using a 1 bit black/white texture as bitmask. The last option would be preferrable)
tis called alpha testing 
GL11.glEnable(GL11.GL_ALPHA_TEST);
GL11.glAlphaFunc( GL11.GL_GREATER, 0);
what that basically does is pass fragments through the test that have a transperancy of > 0, upon which it then goes through the stencil test (if enabled), then the depth test (if enabled) before being rasterised.
DP
Then I would have to use RGBA images to make it work I think?
Is there any way I can use a 1 bit black/white image to mask out a 24 bit RGB image, or define one of the colors in a 24 bit image as translucent? I would prefer being able to use masking, since this is for isometric tiles, and 50% of the rectangle containing a tile is transparent - that way I could save 50% space if I just used the same mask texture for all tiles.
I looked at polygon stipples, but they are only 32X32.
I would suggest using RGBA, but define the internal format as RGB5_A1, that gives you 5 bits for each red, green and blue components and 1 bit for alpha resulting in a combined 16 bit image…
If you really really dont want RGBA, then your going to have to use multitexturing with one channel having RGBA8 and one having Alpha4; your still going to have to deal with the overhead of seperate textures and mulitexturing.
Look here for the internal format listing.
DP
You could look at the NeHe tutorial that does just what you are trying to do. It also includes the source in LWJGL.