O'Reilly logo

Linux Shell Scripting Cookbook by Sarath Lakshman

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Finding broken links in a website

I have seen people manually checking each and every page on a site to search for broken links. It is possible only for websites having very few pages. When the number of pages become large, it will become impossible. It becomes really easy if we can automate finding broken links. We can find the broken links by using HTTP manipulation tools. Let's see how to do it.

Getting ready

In order to identify the links and find the broken ones from the links, we can use lynx and curl. It has an option -traversal, which will recursively visit pages in the website and build the list of all hyperlinks in the website. We can use cURL to verify whether each of the links are broken or not.

How to do it...

Let's write a Bash script ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required