"Don't architect distributed things that require passwordless ssh keys to run. If you need to distribute files to a bunch of machines, they should be pulled from an rsync or http server, rather than via scp or remoting invoking rsync over ssh."
Is this statement directed at me ?
It seems to me that if you need some combination of deploying files and executing commands, the chances that you're going to implement it more securely than SSH is pretty low.
It's true that compromising that one machine with the SSH key is going to compromise your whole cluster, but at least you had to compromise a machine rather than hijack some network service.
ssh is a network service just like any other, one which can potentially be compromised, via any means (exposed key, social engineering, remote exploit, etc). Why expose things like passwordless keys and make it easier to be compromised? One machine that exposes inbound rsync to transfer files is more secure than a cluster of machines that allow passwordless key inbound ssh. ssh is also harder (more error prone) to secure for limiting commands that can be run or only allowing file transfers. When you configure rsyncd, you already know it's only supposed to allow file transfers, so it's straightforward to audit. ssh has many more capabilities thus making auditing more difficult. The security of the files themselves are already not an issue because they are distributed to the entire cluster, so transferring them over ssh doesn't gain any security advantage.
As for it being directed at you, this has occurred more than once in our environment, I suspect because something/someone set a precedent early on. If this is a serious problem depends on the environment. These are the things that keep systems people up at night. I mean, you don't actually expect me to say "put a passphrase on your ssh key" and "it's okay to have passwordless ssh keys scattered around the cluster" do you?
Is this statement directed at me ? It seems to me that if you need some combination of deploying files and executing commands, the chances that you're going to implement it more securely than SSH is pretty low.
It's true that compromising that one machine with the SSH key is going to compromise your whole cluster, but at least you had to compromise a machine rather than hijack some network service.