The set of people willing to use a mouse with their phone is probably a strict superset of those who want to run X, so I wouldn't think of it as problematic.
Also, good on the OP. But X on Android? It's a jfb Hell.
"nobody"? Asus Transformers are a very popular option for Android tablets, in part because they have keyboards and touchpads and a full sized USB port for easily plugging in mice.