Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Looks like you need to set the user agent for this to work (I went with "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (K HTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" and it worked)


curl is for examples on HN. In practice, I never use curl and I never send a user-agent header. If use -H"User-Agent" (no space after the H) it should work fine. Example below. I cannot even think of a single website that requires a user-agent header in order to retrieve a page, but if you send one it will likely be used for something. Fingerprinting, changing the "UI" and blocking are a few uses that come to mind.

    curl -H"User-Agent" https://www.wsj.com/amp/articles/if-you-sell-a-house-these-days-the-buyer-might-be-a-pension-fund-11617544801 \
    |grep -Eo "(<p.*<\/p>)|(<h[12].*</h[12]>)|(<a href=.[^{].*</a>)" \
    |tr -cd '[ -~]' > 1.htm
    firefox ./1.htm




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: