On Mac, the defaults aren't privileged. (Not sure why. That's just how it is.)
The --browse flag is optional and unrelated to ports/privileges. Otherwise, that sounds about right. Caddy defaults to port 80 unless you give it a domain name, then it defaults to port 443 and redirects HTTP on port 80 to HTTPS.
The simplest cmdline I found to run it as a local server on a non-priveledged port with listings is:
Is there a simpler way I'm missing?