> I'm not even sure you can load your openssh server key into ssh-agent, can you?

No need, servers only need to do signature verifications during authentications, thus they only need users/clients public keys which must be listed as authorized_keys.

Edit: I maybe didn't fully grasp you question, if you were referring to ssh host keys, in this case to my knowledge you're right they can not be used with ssh-add.