NB: if you’re doing something SUPER intensive, but SUPER low-complexity (e.g. particle effects), then don’t bother with containers, go straight to arrays/buffers. For everything else…
Only once your game is running so slowly that it’s becoming frustrating to debug and test it.
You can tolerate a lot of low performance before you get to that point. Live with it. Focus on the rest of the game. Do the optimization only when it becomes necessary.
Not “last”, but “only as soon as when you need it, and no earlier”. e.g. when I notice the minimum compile/test/debug cycle being slowed down by more than 30 seconds just because of low performance … I optimize.
Worst: selecting the right subset of items from the container
Second worst: iterating over the container
For the former, you can create custom containers - or … just use smaller containers, and pre-split your data into chunks that you frequently access as subsets.
For both items, arrays (or, better, Buffers) of flyweight impersonated objects are much faster. But don’t bother until/unless you actually need it.