"[EBS is] fine as a basic file store but should never be used for databases."
Doesn't Heroku use EBS for all of their postgres databases?
It may be the case that postgres works better on EBS than mongo does. Postgres has a traditional write-ahead log that minimizes (and spreads out) block writes and hides latencies. Mongo does not.
WAL only helps so much, since you also need seeks for reads. Cassandra has a WAL + log-structured storage + no read-before-update design, so it basically eliminates seeks on writes entirely, and EBS is still ass for workloads that don't fit in cache. Which, if you're bothering to use C*, is almost all of them.
Doesn't Heroku use EBS for all of their postgres databases?
It may be the case that postgres works better on EBS than mongo does. Postgres has a traditional write-ahead log that minimizes (and spreads out) block writes and hides latencies. Mongo does not.