In current version of JOML, class Vector3f
public float angleCos(Vector3f v) {
double length1 = Math.sqrt(x * x + y * y + z * z);
double length2 = Math.sqrt(v.x * v.x + v.y * v.y + v.z * v.z);
double dot = x * v.x + y * v.y + z * v.z;
return (float) (dot / (length1 * length2));
}
public float angle(Vector3f v) {
float cos = angleCos(v);
// This is because sometimes cos goes above 1 or below -1 because of lost precision
cos = Math.min(cos, 1);
cos = Math.max(cos, -1);
return (float) Math.acos(cos);
}
And Vector4f looks the same as the 3D (just extended). The problem is 2D regardless of the number of dimensions, so you want to try to work it as a 2D problem. The version is slower and has more error than is needed.
This is the exact sub-problem is the starting point of formulating slerp and the analysis of quaternion lerp. The only difference between 2D problems in 2D vs. more than 2 is you have to find the plane. (and put it back in place if needed)