When I use Vim I barely even use hkjl except to fine-tune larger movement commands. I'll search for a symbol, or move word-by-word, or use f/F/t/T to move to a different character on a line, which gets me in the ballpark.

Even if hjkl is faster, I would be optimising for a case that just doesn't show up that much.