This is my take on it as well. You can only know what it’s like to be yourself and communicating the qualia of being you is impossible to do with words. You can only ever know what it’s like to be you, so aphantasia cannot be confirmed or denied.
People that think they have aphantasia like talking about it because we like applying labels to ourselves and it makes them feel distinct. I’m sure this is not a popular opinion, but given the inability of knowing what it’s like to be someone else, it’s the only rational belief about aphantasia.
The myth that everyone, or even anyone, can imagine anything, and is doing so regularly, has no basis, yet is the only cited basis of the diagnosis.