[LWJGL] Camera Rotation not working right

[The_Lion_King]

Working on my Java version of my 3D engine. Recently added Camera rotation utilizing Quaterions. It rotates but not correctly. It looks close to correct but it seems like models sometimes end up rotating along their zAxis.

private int yTotal = 0;
	private int xTotal = 0;
	private Vector3f up = new Vector3f(0,1,0);
	private Vector3f xAxis = new Vector3f(1,0,0);
	private Quat totalRot = Quat.IDENTITY;
	public void rotate(float x, float y)
	{
		xTotal += x;
		yTotal += y;
		if(x != 0)
		{
			Quat deltaRot = Quat.angleAxis(up, x);
			TotalRot = Quat.multiply(totalRot, deltaRot);
			xAxis = Quat.multiply(deltaRot, xAxis);
		}
		if(y != 0)
		{
			Quat deltaRot = Quat.angleAxis(xAxis, y);
			totalRot = Quat.multiply(totalRot, deltaRot);
		}
	}
	
	public void draw()
	{
		...
		Matrix4f cameraRotation = Quat.toMatrix(totalRot);
		passAsUniform(cameraRotation, cameraRotationLocation);
		...
	}

GLSL Vertex Shader:

#version 330 core

uniform mat4 RotationMatrix;
uniform mat4 ViewRotationMatrix;
uniform mat4 TranslationMatrix;
uniform mat4 ViewMatrix;

uniform mat4 PerspectiveMatrix;

layout (location = 0) in vec4 position;
layout (location = 1) in vec4 vertexColor;
layout (location = 2) in vec2 texturePos;
layout (location = 3) in vec3 normal;

out vec4 fragColor;


void main()
{
	fragColor = vertexColor;
	gl_Position =  PerspectiveMatrix * ViewRotationMatrix * ViewMatrix * TranslationMatrix * RotationMatrix * position;
}

[edit] I use my own Quat class. The multiply function multiplys the first one by the second. Ex. (Pseudo code) multiply(Quat q1, Quat q2) { return q1 * q2; }