I believe you could use SSH config files to restrict which keys that each host would be able to see.

Something along the lines of:

    Host a.example
      IdentityFile ~/.ssh/id_a
      IdentitiesOnly yes