The problem I have with refactoring todos is that after a time it's no longer clear to me why the refactoring was needed (unless it's obvious, but in that case I probably already did it).
To be clear, I usually add TODOs only for observable features and enhancements, rarely for internal issues like refactoring. My point is that I usually refactor only when I'm already in the code for other reasons, as a reward for doing the immediate observable changes. This helps focus my refactoring efforts only on code that is already changing for the customer's benefit.
