Coming from someone who used to be seen as a white man and is now seen as a white woman, it matters. Noticing that it mattered for me made me take notice when it happens to others. Spoiler: it happens a lot. I shut it down real fast when it happens in front of me, but I used to be part of the problem as well. It's (usually) not malice; people genuinely don't realize that "cultural fit" is usually code for "like me". That ability to be blind to things like that is essentially the nature of what people call "privilege". Doesn't make you a bad person, it just means you should probably listen to other perspectives before you interject your own.
"Affirmative action" isn't a thing. It never really was. All it ever meant was ensuring that opportunities were offered to minorities to interview, but those opportunities still paid out at a below-average rate relative to straight / white / male people. Your company won't share diversity numbers outside of HR, but if you have a friend you can usually get them off the record. They don't share them because the numbers are bad.
"Affirmative action" isn't a thing. It never really was. All it ever meant was ensuring that opportunities were offered to minorities to interview, but those opportunities still paid out at a below-average rate relative to straight / white / male people. Your company won't share diversity numbers outside of HR, but if you have a friend you can usually get them off the record. They don't share them because the numbers are bad.