The rotation calculations are incorrect. A single sin() or cos() is too simple. (also you don’t need a tan())
If you want to rotate around the X axis (via vangle) and around the Y axis (via hangle), you should start with building the rotation matrices:
Then you can do two matrix*vector multiplications with the “standard forward vector”, which is the direction the bullet should shoot in with hangle/vangle = 0.0.
If you formulate that out, you’ll get the resulting formula for the this.motionX/Y/Z variables.
EDIT: Example code how to derive the rotation formula:
double xangle = toRadians(45); // <- example angle around x axis
double yangle = toRadians(45); // <- example angle around y axis
// standard "forward" vector (where to shoot at when xangle/yangle = 0.0
// assuming standard OpenGL convention with z=-1 pointing into screen)
double x = +0;
double y = +0;
double z = -1;
// apply rotation around x with assuming general (x, y, z) vector
// use https://en.wikipedia.org/wiki/Rotation_matrix#Basic_rotations
double x2 = x * 1 + y * 0 + z * 0;
double y2 = x * 0 + y * cos(xangle) + z * -sin(xangle);
double z2 = x * 0 + y * sin(xangle) + z * cos(xangle);
// apply rotation around y with assuming general (x, y, z) vector
// use https://en.wikipedia.org/wiki/Rotation_matrix#Basic_rotations
double x3 = x2 * cos(yangle) + y2 * 0 + z2 * sin(yangle);
double y3 = x2 * 0 + y2 * 1 + z2 * 0;
double z3 = x2 * -sin(yangle) + y2 * 0 + z2 * cos(yangle);
// we can simplify (x3, y3, z3) with assuming (0, 0, -1) vector:
double x4 = -cos(xangle) * sin(yangle);
double y4 = sin(xangle);
double z4 = -cos(xangle) * cos(yangle);
this.motionX = x4 * speed;
this.motionY = y4 * speed;
this.motionZ = z4 * speed;
You probably have to reverse the order of rotations and rotate around Y first, and then around X.
You should get the idea: Apply rotations around the “horizontal” and the “vertical” axes (whatever that might be for you) by means of matrix*vector multiplications with the base rotation matrices.
The correct solution, which works for you, depends on:
what transformations you do additionally
what your “standard forward vector” is (i.e. where should your player look at when hangle and vangle are both zero, this also depends on your model which you want to rotate)
how you set up your coordinate system (what is X, Y and Z, respectively, in your world)
Like I mentioned above, you can also try reversing the order of rotations. First rotate the vector around Y and then around X, which is (assuming (0, 0, -1) as “forward”):
double x = -sin(yangle);
double y = cos(yangle) * sin(xangle);
double z = -cos(yangle) * cos(xangle);