The method of Ray Casting I will be going over is the UnProject method, it works by getting the coordinates of your mouse in 3D space at the nearest and farthest plane on the Z axis. I think everyone will be glad to know that there is very little math involved for just casting the ray. Lets get started.
For the sake of time, I have created two convenience methods to return FloatBuffers and IntBuffers which will be used in the program.
private FloatBuffer compileBuffer(float[] floats) {
FloatBuffer buffer = BufferUtils.createFloatBuffer(floats.length);
buffer.put(floats);
buffer.flip();
return buffer;
}
private IntBuffer compileBuffer(int[] ints) {
IntBuffer buffer = BufferUtils.createIntBuffer(ints.length);
buffer.put(ints);
buffer.flip();
return buffer;
}
Ok, now to the fun part.
First off, we have to make 2 float arrays, I will call them matModelView and matProjView. Make the size of the array 16 (this is the most amount of information that can be returned from an openGL get method).
float[] matModelView = new float[16], matProjView = new float[16];
These float arrays will represent the Model View Matrix and the Projection Matrix.
Now we create an Integer array of size 16 to represent the view port, I will call mine view.
int[] view = new int[16];
At this point, we have to grab our mouse coordinates, if you have your mouse grabbed, your coordinates will just be the middle of the Display.
float mouseX = width / 2;
float mouseY = height / 2;
If your mouse is not grabbed, you will have to get the mouse x and y:
float mouseX = Mouse.getX();
float mouseY = Mouse.getY();
You may have to mess around with inverting the mouseY coordinate, I don’t know for sure because I grab my mouse.
Now we can make 2 Vector3f variables to later hold the coordinates for the ray.
Vector3f start = new Vector3f();
Vector3f end = new Vector3f();
Time to start using those convenience methods 
FloatBuffer modelBuffer = compileBuffer(matModelView);
FloatBuffer projBuffer = compileBuffer(matProjView);
FloatBuffer startBuffer = compileBuffer(new float[]{start.x, start.y, start.z, 1});
FloatBuffer endBuffer = compileBuffer(new float[]{end.x, end.y, end.z, 1});
IntBuffer viewBuffer = compileBuffer(view);
This just makes FloatBuffers with the size of the inputted through the parameters. I guess you didn’t really have to make those variables in the beggining, I just think it looks neater.
The reason that the start and end buffers have a size 4 float array is because that is the minimum allowed by openGL. The 3rd index (4th number) doesn’t really matter and can be anything.
Now we can grab our values from openGL.
glGetFloat(GL_MODELVIEW_MATRIX, modelBuffer);
glGetFloat(GL_PROJECTION_MATRIX, projBuffer);
glGetInteger(GL_VIEWPORT, viewBuffer);
This, very simply, just gets the value of the modelview, projection view, and view port values and stores them into the corresponding objects.
Here is where we actually call the gluUnProjcect method, we will call it twice to get the near and far coordinates.
gluUnProject(mouseX, mouseY, 0.0f, modelBuffer, projBuffer, viewBuffer, startBuffer);
gluUnProject(mouseX, mouseY, 1.0f, modelBuffer, projBuffer, viewBuffer, endBuffer);
As you can see, we input the 2D mouse coords, the requested Z distance, and the 3 values we just grabbed. The last parameter is the object in which to store the output of gluUnProject.
That is basically it, other than assigning the start and end Vectors using the startBuffer and endBuffer objects:
start = new Vector3f(startBuffer.get(0), startBuffer.get(1), startBuffer.get(2));
end = new Vector3f(endBuffer.get(0), endBuffer.get(1), endBuffer.get(2));
See, very simple. If you did it right, you should get a method that looks like this (except for the mouse coords, if your mouse is not grabbed) :
float[] matModelView = new float[16], matProjView = new float[16];
int[] view = new int[16];
float mouseX = width / 2;
float mouseY = height / 2;
Vector3f start = new Vector3f();
Vector3f end = new Vector3f();
FloatBuffer modelBuffer = compileBuffer(matModelView);
FloatBuffer projBuffer = compileBuffer(matProjView);
FloatBuffer startBuffer = compileBuffer(new float[]{start.x, start.y, start.z, 1});
FloatBuffer endBuffer = compileBuffer(new float[]{end.x, end.y, end.z, 1});
IntBuffer viewBuffer = compileBuffer(view);
glGetFloat(GL_MODELVIEW_MATRIX, modelBuffer);
glGetFloat(GL_PROJECTION_MATRIX, projBuffer);
glGetInteger(GL_VIEWPORT, viewBuffer);
gluUnProject(mouseX, mouseY, 0.0f, modelBuffer, projBuffer, viewBuffer, startBuffer);
gluUnProject(mouseX, mouseY, 1.0f, modelBuffer, projBuffer, viewBuffer, endBuffer);
start = new Vector3f(startBuffer.get(0), startBuffer.get(1), startBuffer.get(2));
end = new Vector3f(endBuffer.get(0), endBuffer.get(1), endBuffer.get(2));
I hoped I could help someone. My next tutorial will be Ray Picking, so actually getting the intersections and what not.