Alright, so I am thoroughly stuck. 
I’ve been trying to use Camera.unproject(Vector3) for picking tiles off a tile map (mouse cursor coords -> world coords), but I am either doing something horribly wrong or don’t understand what unproject() actually does.
Here are some snippets from my test class:
I setup a camera like this:
cam = new PerspectiveCamera(30, Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
cam.near = 0.1f;
cam.far = 200f;
cam.position.set(10, 10, 10);
cam.rotate(90, 1, 0, 0); //now the X-Y plane is the 'floor' of the world
cam.lookAt(0, 0, 0); //set direction
cam.position.set(20, 20, 8); //move it back a bit
cam.update();
(And yes, I’m doing isometric tiles with a PerspectiveCamera; I like the look)
Then in the render() method:
/*I'm leaving out the cam.translate() calls here, but it translates the x and y from WASD input (flyover)*/
cam.update();
cursor.set(mouse); //cursor is a Vector3, as is mouse, the last screenX and screenY from mouseMoved(), with z = 0 (near plane unprojection)
cam.unproject(cursor);
renderer.setProjectionMatrix(cam.combined);
renderer.begin(ShapeType.FilledRectangle);
renderer.setColor(Color.BLUE);
renderer.filledRect(cursor.x - .5f, cursor.y - .5f, 1, 1); //draw 1x1 quad centered on cursor on the z-plane ('floor')
renderer.end();
For some reason the unprojected point is always underneath the camera, out of view. Adding
cursor.x -= cam.position.z;
cursor.y -= cam.position.z;
Fixes the underneath-the-camera issue, but it seems really hacky, and I feel like it should be unnecessary.
But still the problem remains: the quad is never anywhere near the mouse cursor, exempt when at the center of the screen. In fact, this current code barely moves the quad.
I’ve tried all sorts of scaling and such, like cursor.mul(100) but it either just explodes or is still very off.
Now, I know this camera positioning works fine, I can render and look at isometric tiles no problem, so it’s not the camera’s position missing the quad, just that the unproject is acting very strange.
The best result I have gotten by fiddling with the cursor vector’s scaling before unprojection is this: (Red dot is where my cursor was)
http://img28.imageshack.us/img28/9543/bx9.png
http://img845.imageshack.us/img845/9789/wzb.png
From this code (again in the render method):
cursor.set(mouse.x - Gdx.graphics.getWidth() / 2, mouse.y - Gdx.graphics.getHeight() / 2, mouse.z);
cursor.y *= 2;
cam.unproject(cursor.mul((float) (cam.position.z * Math.sqrt(300)))); // (sqrt(300) is the distance from the camera to the point on the floor it's looking at)
cursor.x -= cam.position.z;
cursor.y -= cam.position.z;
cursor.z = 0;
So, what am I doing wrong and/or is there a better way to do this? (I need in-world coordinates that correspond to where the mouse is)
If my question is unclear in any way, just tell me, I’ll try to clear it up.
Thanks in advance, I’m really having trouble with this.