Security with Go

Changing the user agent of a request

A common technique to block scrapers and crawlers is to block certain user agents. Some services will blacklist certain user agents that contain keywords such as curl and python. You can get around most of these by simply changing your user agent to firefox.

To set the user agent, you must first create the HTTP request object. The header must be set before making the actual request. This means that you can't use the shortcut convenience functions such as http.Get(). We have to create the client and then create a request, and then use the client to client.Do() the request.

This example creates an HTTP request with http.NewRequest(), and then modifies the request headers to override the User-Agent header. ...