The point is to stabilize and calm the body-mind-spirit complex, a harmony which is called yoga.
Taking drugs, even alchohol, stresses the body and mind, and may distort and cloud the spirit. This work against yoga / equanimity / clarity.
There are traditions that uses drugs ritually, so is part of those traditions. Though, it is often a more perilous and dedicated path. Not everyone needs to live life as a sage or munk.
So doing sadhana (practices) will work over time. It's not a competition.
Because they destabilise the mind, which is particularly dangerous when pursuing powerful spiritual practices where balance and stability of body, mind, emotions and energy are basic requirements.
An imperfect analogy is taking drugs and then driving fast on a windy mountain road.
Why not?