The main downside I see is performing schema changes. A very common pattern I use with protobufs in distributed systems is: 1.) add a field to the schema 2.) update and deploy the grpc servers to start populating that new field. The old grpc clients will continue to work ignoring the new field 3.) update and deploy the grpc clients to start using the new field

With packed structs you don’t get the reverse compatibility, so schemas and endpoints have to be immutable which is not the end of the world but it can make changes more difficult.