Does one change vertex positions in 3D space or only use matrices for that?

Hello dear JGO,

When you have a model in 3D space, you load it by giving it an origin and the vertices are positioned relative to that origin.
Now, when you do physics calculations/rendering etc, do you ever change the positions of the vertices or do you just use the matrices for that purpose?

I would guess, that for performance reasons, you only load the model at lets say x:0|y:0|z:0 and Scale/Rotate/Transform for rendering and never change the vertex position itself again?!
But for physics calculations, how do you do that? If for example a model is deformed, how would you store this data?

Thank you very much in advance!