So basically, you’re trying to detect ray-triangle intersection (intersection of a ray and a triangle).
There are several websites with articles and tutorials about detecting intersection (ray-triangle, ray-plane, ray-sphere, ray-point, etc.)
Here’s one that looks promising: http://www.lighthouse3d.com/tutorials/maths/ray-triangle-intersection/
You could try the 3D Math Primer for Graphics and Game Development (it’s a book, not a website). It has an entire chapter about detecting intersection. Also, make sure your method of transforming screen coordinates to normalized device coordinates actually works the way it should.
If you don’t want to implement ray-? intersection tests yourself, you could use a physics engine (jBullet, as I’m sure you already know, is the most popular, if not the only, physics engine for Java. It’s a port of the ever-popular Bullet physics engine, which was written in C++. Bullet Physics is tied in first place with Nvidia’s PhysX on the popularity rankings, but I’m pretty sure Bullet is easier to use than PhysX).
I think Bullet has a rayTest function to detect ray-? intersection, not too sure. Bullet can be a royal pain in the neck (it’s not very easy to use), but…hey, no complaints! It’s awesome!
Good luck and have fun!