3d world mouse coordinates?

kappa · January 3, 2006, 7:35pm

i have a small problem that i’m stuck on, i’m trying to get the 3d coordinates of the place i click on in a 3d world with the mouse, i have tried a few methods but haven’t been able to get any good results

currently i do



public void get3dCoord(int mouseX, int mouseY) {
     GLU.gluUnProject( mouseX, mouseY, getZDepthAtOrigin(), getModelviewMatrixA(), getProjectionMatrixA(), getViewportA(), result);
}


public float getZDepthAtOrigin()
    {
        float[] resultf = new float[3];
        project( 320, 240, 0, resultf);
        return ((int)(resultf[2] * 1000F)) / 1000f;  // truncate to 4 decimals
    }

i use gluLookAt to move the camera around the 3d world

GLU.gluLookAt(camX, camY, camZ,  // camera pos
				320, 240, 0, // camera target
				tiltX, tiltY, tiltZ); // camera tilt

with the code above i’ve been able to get ok results from the mouse when camX is at 320, camY is at 240 and camZ is changed to zooms in and out but when i change camX, or camY, the point where the mouse should be on the 3d world map starts going off, what is the correct way to do this?

to add to the problem i also have a 3d height map, i was wondering if there is a good way to get the 3d point on where the pointer clicks as done in games such as c&c: generals, tribal trouble:), dawn of war etc

thanks

Riven · January 3, 2006, 8:50pm

Read the pixel at (320,240) in the depth-buffer. IIRC you ahd to read it as int, and scale it back using your near/far clip-planes. The int is unsigned, so you might want to cast it to a long:

float range = far-near;
float normalized = ((long)depth & 0xFFFFFFFFL) / ( (float) 0xFFFFFFFFL);
float real = normalized * range + near;

…

I didn’t test it, bit it might point you in the right direction.

kappa · January 3, 2006, 9:05pm

do you mean something like

 /**
     * Return the Z depth of the single pixel at the given screen position.
     */
    public static float getZDepth(int x, int y)
    {
        ByteBuffer buf = BufferUtils.createByteBuffer(64*64*4);
        GL11.glReadPixels(x, y, 1, 1, GL11.GL_DEPTH_COMPONENT, GL11.GL_FLOAT, buf);
        return buf.getFloat();
    }

Riven · January 3, 2006, 9:08pm

For some reason I used unsigned ints… But if float works for you, go ahead

Edit: why the massive 64644 buffer? You need like 4 bytes.

kappa · January 3, 2006, 9:12pm

sorry about the above was just copying and pasting code from different places

i’m still have problems with the code currently i have

public void getCoords(int mouseX, int mouseY) {
GLU.gluUnProject( mouseX, mouseY, getZDepth(320, 240), 
                                                                getModelviewMatrixA(), getProjectionMatrixA(), 
                                                                getViewportA(), result);	
}


/**
* Return the Z depth of the single pixel at the given screen position.
*/
public static float getZDepth(int x, int y)
{
ByteBuffer buf = BufferUtils.createByteBuffer(4);
GL11.glReadPixels(x, y, 1, 1, GL11.GL_DEPTH_COMPONENT, GL11.GL_FLOAT, buf);
return buf.getFloat();
}

with the code above now the mouse point on the 3d world is way off, any idea’s

Riven · January 3, 2006, 9:24pm

You have to rescale your depth anyway, because the value you retreive by reading the buffer, is 0.0 -> infinity. So use the near/far values as in my previous example.

Riven · January 3, 2006, 9:25pm

I’ll try to get something working on my own system. Will report soon.

Riven · January 3, 2006, 10:21pm

         int x = 400;
         int y = 300;

         ShortBuffer dBuf = BufferUtils.createShortBuffer(1);
         Matrix4 mat = new Matrix4();
         float[] array = new float[16];
         float[][] proj = new float[4][4];
         float[][] view = new float[4][4];
         float[] obj = new float[3];

         dBuf.clear();
         glReadPixels(x, y, 1, 1, GL_DEPTH_COMPONENT, GL_UNSIGNED_SHORT, dBuf);
         float z = (float) (dBuf.get() & 0xFFFF) / 0xFFFF;

         MatrixUtil.download(mat, GL_PROJECTION_MATRIX);
         mat.store(array);
         for (int k = 0; k < 4; k++)
            for (int p = 0; p < 4; p++)
               proj[k][p] = array[p * 4 + k];

         MatrixUtil.download(mat, GL_MODELVIEW_MATRIX);
         mat.store(array);
         for (int k = 0; k < 4; k++)
            for (int p = 0; p < 4; p++)
               view[k][p] = array[p * 4 + k];

         boolean result = GLU.gluUnProject(x, y, z, view, proj, Overlay.getRectangle(), obj);

         if (result)
         {
            Vec3 obj3 = new Vec3(obj);
            Vec3 cam3 = new Vec3(Camera.POSITION);
            float distance = VecMath.distance3D(obj3, cam3);
            System.out.println("distance=" + distance);
         }

I know it relies on my own libraries, but I think it all speaks for itself. Give me a call when you get stuck.

kappa · January 3, 2006, 10:38pm

thx for your help, i’ll give it a try

kappa · January 3, 2006, 11:33pm

ok just tried it and it sort of works w00t (thx), but however there is a small problem, if i draw something now on the 3d world it comes nicely now under the mouse pointer, however the values i get back for the world coordinates are extreamly large

 X -5011.8174 Y 8933.184 Z -17616.508
 X -9222.333 Y 14470.639 Z -15844.684
 X -9224.201 Y 14470.437 Z -15841.066
 X -9226.223 Y 14471.448 Z -15841.066
 X -9230.611 Y 14475.588 Z -15843.427

as i have a flat grid at z = 0, where a unit moves around on it towards mouse clicks, since the new values are very large it goes off the screen any ideas?

thx

Riven · January 3, 2006, 11:47pm

I transpose my matrices under the hood when reading the matrices in opengl.

OpenGL uses transposed matrices for some awkward reason.

Give it a try.

The code works perfectly for me, so I think this is the only difference between our code.

kappa · January 4, 2006, 1:30pm

i’m not sure what you mean when you say transpose matrices, i use the following methods to get the matrices


public void getCoords(int x, int y) {
     GLU.gluUnProject( x, y, getZDepth(x, y), getModelviewMatrixA(), getProjectionMatrixA(), getViewportA(), result);
}



/**
* Return the Viewport data as array of 4 floats
*/
public static int[] getViewportA()
{
    IntBuffer ib = getViewport();
    int[] ia = new int[4];
    ia[0] = ib.get(0);
    ia[1] = ib.get(1);
    ia[2] = ib.get(2);
    ia[3] = ib.get(3);
    return ia;
}
    
public static IntBuffer getViewport()
{
    IntBuffer bufferViewport = BufferUtils.createIntBuffer(16);
    bufferViewport.clear();
    GL11.glGetInteger(GL11.GL_VIEWPORT, bufferViewport);
    return bufferViewport;
}
	
/**
* Return the projection matrix as a 4x4 float array
*/
public static float[][] getProjectionMatrixA()
{
    FloatBuffer b = getProjectionMatrix();
    return getMatrixAsArray(b);
}
    
public static FloatBuffer getProjectionMatrix()
{
    FloatBuffer bufferProjectionMatrix = BufferUtils.createFloatBuffer(16);
    	
    bufferProjectionMatrix.clear();
    GL11.glGetFloat(GL11.GL_PROJECTION_MATRIX, bufferProjectionMatrix);
    return bufferProjectionMatrix;
}
		
/**
* Return the modelview matrix as a 4x4 float array
*/
public static float[][] getModelviewMatrixA()
{
    FloatBuffer b = getModelviewMatrix();
    return getMatrixAsArray(b);
}
	
public static FloatBuffer getModelviewMatrix()
{
    FloatBuffer bufferModelviewMatrix = BufferUtils.createFloatBuffer(16);
    	
    bufferModelviewMatrix.clear();
    GL11.glGetFloat(GL11.GL_MODELVIEW_MATRIX, bufferModelviewMatrix);
    return bufferModelviewMatrix;
}
    
public static float[][] getMatrixAsArray(FloatBuffer fb) {
    float[][] fa = new float[4][4];
    fa[0][0] = fb.get();
    fa[0][1] = fb.get();
    fa[0][2] = fb.get();
    fa[0][3] = fb.get();
    fa[1][0] = fb.get();
    fa[1][1] = fb.get();
    fa[1][2] = fb.get();
    fa[1][3] = fb.get();
    fa[2][0] = fb.get();
    fa[2][1] = fb.get();
    fa[2][2] = fb.get();
    fa[2][3] = fb.get();
    fa[3][0] = fb.get();
    fa[3][1] = fb.get();
    fa[3][2] = fb.get();
    fa[3][3] = fb.get();
    return fa;
}

Riven · January 4, 2006, 3:51pm

This is how to get the transposed matrices (flipping row/column)

public static float[][] getMatrixAsTransposedArray(FloatBuffer fb) {
    float[][] fa = new float[4][4];
    fa[0][0] = fb.get();
    fa[1][0] = fb.get();
    fa[2][0] = fb.get();
    fa[3][0] = fb.get();
    fa[0][1] = fb.get();
    fa[1][1] = fb.get();
    fa[2][1] = fb.get();
    fa[3][1] = fb.get();
    fa[0][2] = fb.get();
    fa[1][2] = fb.get();
    fa[2][2] = fb.get();
    fa[3][2] = fb.get();
    fa[0][3] = fb.get();
    fa[1][3] = fb.get();
    fa[2][3] = fb.get();
    fa[3][3] = fb.get();
    return fa;
}

kappa · January 4, 2006, 6:29pm

thx for the code update but transposing the matrices doesn’t seem to be working i must be doing something wrong, now the values i get are extreamly small


 X 0.0019316514 Y -4.2342854E-4 Z -5.4228705E-5
 X 0.0019316514 Y -4.2342854E-4 Z -5.4228705E-5
 X 0.0019316514 Y -4.2342854E-4 Z -5.4228705E-5
 X 0.0019316514 Y -4.2342854E-4 Z -5.4228705E-5
 X 0.0019316514 Y -4.2342854E-4 Z -5.4228705E-5
 X 0.0019316514 Y -4.2342854E-4 Z -5.4228705E-5
 X 0.0019316514 Y -4.2342854E-4 Z -5.4228705E-5
 X 0.0019316514 Y -4.2342854E-4 Z -5.4228705E-5

kappa · January 4, 2006, 7:06pm

ok i’ve just managed to get so sortta good results, i have switched back to the non-transposed matricies, i was using grid lines before as a plane and i think the rays from readpixel where going right through it, when i switched to a quad plane it seems to work nicely, i have also now turned on the heightmap and it seems to work nicely with it, however this only works when the mouse is actually on the plane and not in 3d space.

Riven · January 4, 2006, 7:34pm

read-pixels reads from the depth-buffer, and when you are drawing lines, instead of filled shapes, there isn’t anything written to the depth-buffer inside the shape that the lines make up.

Also be sure that you read the projection/modelview-matrices at the right time. If you are rendering the 2D overlay, and read the projection-matrix from there, you can’t get the correct coords of 3d-space because you are working with the wrong matrix.

I think that is obvious, but from the problem you have, that might be the problem.

Riven · January 5, 2006, 10:33am

Is the above helpful to you?

I’d like to help you with this one.

kappa · January 5, 2006, 6:38pm

ok thx for the help i’ve got it working now, was indeed the fact that i was using lines, switching to quads has solved the problem, i’ll let you know if i have any other issues.

thx