While the problems with templates and huge includes are undeniable, I've found in my experience that for a decently size C++ program the linking stage alone can take way more than the 3-4 seconds the GP is talking about.

C and C++ are definitely some hard beasts to compile, even without boost.