Disregarding the (slow) advancements of E-Mail encryption (DANE), and the fact that's desireable that no one company has your data. The big point is that it's fair to assume that everybody has access to your mails while in transit, but using gmail Google has access to all your mails at rest.
"Ditching GMail changes nothing about your privacy situation."
Actually, there is quite a bit to be gained by running your own mail server - especially in relation to other people on your mail server.
For instance, everyone at rsync.net logs into (al)pine over SSH. So yes, if you email rsync.net and we converse, that is like the postcard - every hop it goes through can see it.
But no piece of internal rsync.net email has ever traversed a network of any kind. Internal rsync.net emails are just local copy operations from one mailspool to another.
The same could be true of your company ... or your family.
Ditching GMail changes nothing about your privacy situation.
An E-Mail is a postcard. Sending a postcard via DHL instead of USPS doesn't make it any more confidential, even if you happen to be the owner of DHL.