I've done this for a few years, but the one use case it's worse is when you are in a slow/stopped lane and want/need to switch to a lane that's moving faster, and you need to see far back in that adjacent lane to see approaching cars. In those cases I'll actually move my mirror in, then readjust it out later.