This is called “unprojecting.” What you generally have to do here is to take the transformation which brings you from view-space to clip-space, and then just do the opposite transformation. That “transformation” is usually a 4x4 matrix.
And in OpenGL this transformation is the “projection” matrix (without any view or model transformations).
And in matrix-speak the “opposite” transformation of a matrix is the inverse of that matrix. So do a matrix inversion of that projection matrix.
After that, you can multiply any clip-space position/vector by that matrix and end up with the homogeneous view-space position.
Also you don’t need any sine/cosine functions to do that.
Now the next problem is how to get a vector in clip-space from your mouse coordinates.
First thing, we take your mouse coordinates, which are in window coordinates and transform them to normalized device coordinates (NDC) by normalizing them between (-1, -1) and (+1, +1), so that the center pixel coordinate of your window yields (0, 0).
Then I’d build a NDC/clip-space coordinate (NDC.x, NDC.y, 0, 1), where NDC.x and NDC.y are your previously computed mouse coordinates in NDC space and the ‘Z’ coordinate is 0 which means the near frustum plane.
You can then matrix-vector multiply that vector by the previously computed inverted projection matrix, which will yield a position in homoegenous view-space.
Then, do a perspective divide (dividing your vector by its own ‘w’ component) to get from homogeneous coordinates to real 3D coordinates.
Next thing is to make sure that the vector is exactly 10 units of length away from your camera. An easy solution is to normalize that vector and multiply by 10.
Since we are in view/camera space, the vector is still pointing into the same direction when viewed from the camera.
Now if desired, we just need to do another transformation from view-space to world-space. This can be done using the inverse view matrix in OpenGL.
There you have your world-space coordinate.
Note that this is a general algorithm that works with any projection (orthographic or perspective) and with any camera/view position/rotation etc…
If your setup is simpler and you restrict your camera to certain properties, then a simpler solution might be possible.
Here is a nice overview of all the spaces involved in OpenGL: http://www.songho.ca/opengl/gl_transform.html (view-space is called “eye coordinates” there).