The problem with your argument is that indistinguishable doesn't mean the equal. I'm not saying there is no difference between adjacent colors. I'm saying that the difference is indistinguishable. A lot of indistinguishable differences add to a distinguishable difference.
Think about it with numbers: if your sequence A, B, C... are each zero units apart, then yes, A = B = C = ...
In actuality, though, they are not zero units apart. They are a small, but positive distance apart. Those tiny differences add up so the distance from A to C is twice as far as the distance from A to B.
Let's do an example, using just colors with no red or green in their RGB representation so we just have to give the B component, which is a decimal from from 0 to 255. Let's call 0 "black".
Your suggestion was that if a color is close enough to another to be indistinguishable it the two should have the same name.
1 is close enough to 0, so gets the name "black" too.
How about 2? That's farther from 0 "black". Is it distinguishable? If not, it too is "black", and we can go on to compare 3 and 0, and beyond.
At some point we get our lowest blue-only RGB color that is distinguishable from 0 "black". Suppose that is 4. So we have 0, 1, 2, and 3 are all called "black", and since 4 is distinguishable from 0 it needs a new name, say "very light blue".
But now we have 3 "black" only 1 away from 4 "very light blue". Those are probably indistinguishable, so are supposed to have the same name. Oops.
Having more than one name doesn’t change the story, and neither does some colors having no name at all. For each pair of adjacent colors which is indistinguishable to most people, and the two colors must either have the same name (the two sets of names are equal) or different names (the two sets of names have no intersection).
Think about it with numbers: if your sequence A, B, C... are each zero units apart, then yes, A = B = C = ...
In actuality, though, they are not zero units apart. They are a small, but positive distance apart. Those tiny differences add up so the distance from A to C is twice as far as the distance from A to B.