I have a fast machine, never noticed a difference in shell start up time between with and without and given how much more pleasant it is with I'll take the hit.