Good. Seriously. It's good that you're not a racist scumbag.
But the actual quality of you as a human being wasn't the issue. The issue is that, if you're going to block speech by scumbags, there is someone who is going to consider you a scumbag. If you allow speech from scumbags to get blocked, then you need the people controlling the blocking to have the right ideas of who is a scumbag. Will the right people be the ones making the decisions? Will they always be the right people - not just this year and next, but always?
I'm not willing to condone that power, because it can be used against me, not just against the people I want it to be used against.
By defending their freedom of speech, I am defending my freedom of speech. I may not care about theirs, but I care about mine.
And for you to not be worried about this being used against you... you just said that these people are in power. That means that they at least have the capability of using it against you in the near future.