My intuition is that anything with a really eccentric orbit (10s to 10s of thousands of AU) has more of a chance of passing through many gravitational fields that change its orbit.
I suppose that makes sense... if something is in an orbit that's flattened down until it's only slightly off from being a straight line in and out, then any very slight perturbations could really affect how close it comes to the sun on the next pass.
The sun moves too a little bit, due to the pull of the planets. Maybe it's enough to make a difference?