24bit vs 32bit ColorModel performance

So is a 24 bit color model (for use in a MemoryImageSource) slower than a 32 bit color model?

I’ve read around and people say that 32 bit is faster because the processors are able to handle 32 bit numbers faster than 24 bit ones, and also I read that not using the 8 bits at the end of the 24 bits (because modern registers are 32 bit) is a waste of memory.