To be useful, that key needs to be present in countless border checkpoint or even police devices, so it's inherently very hard to keep secret.
That's a known trade-off, and I believe some countries accordingly restrict access to their own national authorities (which usually already have access to that data via other means, since they're issuing the document).
You wouldn't need the key to be copyable though - it can be an online check - ie. passport sends challenge, terminal sends challenge to government-hosted HSM, HSM sends response.
Good point, but that's in the end a typical availability/security tradeoff, and I could imagine that at least some verifying authorities would rather err on the side of the former.
:-)