[engine] How to represent object transforms

I am developing a very simple game engine, and I am going through the different requirements for the physics/collision, and the lighting and such. And I have come to the conclusion that every game entity requires a 4x4 object matrix for graphics. Although, some systems like physics, and collision require a 3 component vector to a point in space, representing a translation. My problem is, should I handle each entity of having a Matrix4 for it’s object-space transform, or should I have vec3s for translation, rotation, and scale, then calculate a matrix for them each frame? Or is this bad practice/CPU intensive?

Store them in decomposed states. This use less memory(16floats vs 8/10(vec3 pos, quaternion orientation, vec1/3 scale)
This way you can also avoid problems related to interpolated rotations, inverse transforms etc. Composing matrix from these hardly use any cpu cycles.