You can use a hash of the site appended with a .pass wide pepper as the name of the directory storing credentials for a particular site, then use a wrapper script that hashes its input before passing it to pass.
This is all a lot of effort, if I went down that road I might as well skip "pass" and handle the passwords myself. What I like about pass is that there isn't much setup.
Full disk encryption also doesn't prevent a running application from seeing the directory structure. But I guess this is not a very realistic attack vector.
Also full disk encryption.