If your SSH private key is on the Yubikey then you will not lose your private keys. Even in the case of U2F, the attacker will not figure out your U2F private keys or even all the places you are registered.
His point, which is correct, is that you'll persistently lose access to your server anyways, because a backdoored SSH client is almost as bad as a compromised key. I use a Y4 for SSH, but it's good to be clear-eyed about the limitations.
