This is ignoring difference between individual distancing from such person and cancel culture campaigns, whose characteristic pattern is transitivity (boycotting third persons just because they do not participate themselves in boycott). Amplifying effect of such transitivity creates a social coercion in the form that Mill warned against in chapter 1. That is why he mentioned "though not to parade the avoidance" - to not induce society-wide coordination leading to social coercion.
Mill distinguishes between natural penalties (which are just results of others distancing from the person) and artificial punishment:
"It makes a vast difference both in our feelings and in our conduct towards him, whether he displeases us in things in which we think we have a right to control him, or in things in which we know that we have not. If he displeases us, we may express our distaste, and we may stand aloof from a person as well as from a thing that displeases us; but we shall not therefore feel called on to make his life uncomfortable."
I think it is clear that actions like urging an employer to fire such person are more on level "make his life uncomfortable" than "we may express our distaste and we distance".
The "displeases us" you quoted is for "the part of a person's life which concerns only himself". For example, "No person ought to be punished simply for being drunk."
For actions "which concerns others" Mill writes:
"The evil consequences of his acts do not then fall on himself, but on others; and society, as the protector of all its members, must retaliate on him; must inflict pain on him for the express purpose of punishment, and must take care that it be sufficiently severe. In the one case, he is an offender at our bar, and we are called on not only to sit in judgment on him, but, in one shape or another, to execute our own sentence: in the other case, it is not our part to inflict any suffering on him, except what may incidentally follow from our using the same liberty in the regulation of our own affairs, which we allow to him in his."
Yes, but in the first chapter, Mill is clear that even public speech falls into sphere of liberty, the part of person's life "that affect only himself":
"But there is a sphere of action in which society, as distinguished from the individual, has, if any, only an indirect interest; comprehending all that portion of a person’s life and conduct which affects only himself, or if it also affects others, only with their free, voluntary, and undeceived consent and participation. When I say only himself, I mean directly, and in the first instance: for whatever affects himself, may affect others through himself; and the objection which may be grounded on this contingency, will receive consideration in the sequel. This, then, is the appropriate region of human liberty."
"It comprises, first, the inward domain of consciousness; demanding liberty of conscience, in the most comprehensive sense; liberty of thought and feeling; absolute freedom of opinion and sentiment on all subjects, practical or speculative, scientific, moral, or theological."
"The liberty of expressing and publishing opinions may seem to fall under a different principle, since it belongs to that part of the conduct of an individual which concerns other people; but, being almost of as much importance as the liberty of thought itself, and resting in great part on the same reasons, is practically inseparable from it."
(Also note that your original citation was also related to judgements based on actions that concern only the judged person: "Though doing no wrong to anyone, a person may so act as to compel us to judge him, and feel to him, as a fool, or as a being of an inferior order")
Mill distinguishes between natural penalties (which are just results of others distancing from the person) and artificial punishment:
"It makes a vast difference both in our feelings and in our conduct towards him, whether he displeases us in things in which we think we have a right to control him, or in things in which we know that we have not. If he displeases us, we may express our distaste, and we may stand aloof from a person as well as from a thing that displeases us; but we shall not therefore feel called on to make his life uncomfortable."
I think it is clear that actions like urging an employer to fire such person are more on level "make his life uncomfortable" than "we may express our distaste and we distance".