As a hearing person who has communicated with deaf people over relay service, let me tell you it sucks just as much being on the other end of a relay call. It seems to have all the disadvantages of email and phone rolled into one, with just enough added latency over a straight phone call to be annoying.
I wish I had an alternative to suggest, but whether anybody likes it or not, every so often, hearing-impaired people are going to need to communicate with people over the phone, and those of us who can hear need to just suck it up.
While VRS is probably the closest to real-time voice communication, it's similar to speaking through a synchronous translator for another language. I've done synchro work and it's not the best way. No wonder so many political gestures get miscommunicated at conferences!
I'd say SMS and chat are far better (or email), as you are actually communicating with the person directly.
I wish I had an alternative to suggest, but whether anybody likes it or not, every so often, hearing-impaired people are going to need to communicate with people over the phone, and those of us who can hear need to just suck it up.