How strong that effect is, depends on how you implement the algorithm.
You don't have to assume a person has read all ancestors, just because he votes on a comment. — If you assume s/he has read only the closest 3 ancestors (which I think is more reasonable), the problem you describe is largely gone.
If you do, however, assume all ancestors have been read, I think there would be a tendency that the most popular comments cycled through the topmost positions (up, down, up, down, between position 1, 2, 3 perhaps). But things would not be sorted by time.
You don't have to assume a person has read all ancestors, just because he votes on a comment. — If you assume s/he has read only the closest 3 ancestors (which I think is more reasonable), the problem you describe is largely gone.
If you do, however, assume all ancestors have been read, I think there would be a tendency that the most popular comments cycled through the topmost positions (up, down, up, down, between position 1, 2, 3 perhaps). But things would not be sorted by time.