Interestingly, I don't feel faster with the keyboard. I do, however, feel more comfortable when I stick to one mode (keyboard-only or mouse-only). It's switching that feels awkward. I also don't use Xmonad out of pure Geekiness (though I started that way). I use it because I find it more comfortable. I work at the same time with OSX, and while moving windows with the mouse is probably just as fast, it annoys me.
Now the reason why the keyboard is slower than the mouse may be sub-optimal short-cuts. For example, when editing, one very often needs to change lines, or otherwise navigate the text. A sensible short-cut to support that could be alt + jkli (assuming a Qwerty layout). Then you don't have to leave the home row at all. But if you have to use the actual arrows, it's slower, and that may make the mouse the better choice.
Now the reason why the keyboard is slower than the mouse may be sub-optimal short-cuts. For example, when editing, one very often needs to change lines, or otherwise navigate the text. A sensible short-cut to support that could be alt + jkli (assuming a Qwerty layout). Then you don't have to leave the home row at all. But if you have to use the actual arrows, it's slower, and that may make the mouse the better choice.