Linux Shell Scripting Cookbook - Third Edition

Setting a user agent string with cURL

Some web pages that check the user agent won't work if there is no user agent specified. For example, some old websites require Internet Explorer (IE). If a different browser is used, they display a message that the site must be viewed with IE. This is because the website checks for a user agent. You can set the user agent with curl.

The --user-agent or -A option sets the user agent:

$ curl URL --user-agent "Mozilla/5.0"

Additional headers can be passed with cURL. Use -H "Header" to pass additional headers:

$ curl -H "Host: www.knopper.net" -H "Accept-language: en" URL

There are many different user agent strings across multiple browsers and crawlers on the Web. You can find a list of some of them ...