They're uncomfortable, mess up your hair, make you extra sweaty, and you need to put them somewhere.

None of these are good enough reasons to not wear a helmet while you're riding a bike, but any one of them would be a good enough reason to not ride your bike at all. That's the point OP is trying to make by saying you can skip your helmet if you want.