Cybersecurity Ops with bash

Chapter 1. Command-Line Primer

A computer’s command-line interface gives you an intimate connection with its operating system (OS). Within the operating system lives an astounding amount of functionality that has been honed and perfected over decades of use and development. Sadly, the ability to interact with the OS by using the command line is quickly becoming a lost art. It has been replaced instead by graphical user interfaces (GUIs), which often increase ease of use at the expense of speed and flexibility, and distance the user from the underlying capabilities.

The ability to effectively use the command line is a critical skill for security practitioners and administrators. Many tools of the trade such as Metasploit, Nmap, and Snort require command-line proficiency simply to use them. During penetration testing, your only option may be to use a command-line interface when interacting with a target system, particularly in the early stages of an intrusion.

In order to build a solid foundation, we will begin with an overview of the command line and its components; then we will look at how it can be applied to enhance your cybersecurity capabilities.

The Command Line Defined

Throughout this book, the term command line is used to refer to all of the various non-GUI executables installed with an operating system, along with, and especially, the built-ins, keywords, and scripting capabilities available from the shell—its command-line interface.

To effectively utilize the command line, you need two things: an understanding of the features and options of the existing commands, and a way to sequence commands together by using a scripting language.

In this book, we introduce more than 40 commands that span both the Linux and Windows operating systems, as well as a variety of shell built-ins and keywords. Most of the commands introduced originate from the Linux environment, but as you will see, there are multiple methods for running them on Windows platforms.

Why bash?

For scripting purposes, we choose the bash shell and command language. The bash shell has been around for decades, is available in nearly every version of Linux, and has even permeated the Windows operating system. That makes bash an ideal technology for security operations because the techniques and scripts are cross-platform. The pervasiveness of bash also gives offensive operators and penetration testers a particular advantage, because in many cases there is no additional supporting infrastructure or interpreters to install on a target system.

Command-Line Illustrations

This book makes heavy use of the command line through numerous examples. A single-line command illustration will appear as follows:

ls -l

If the single-line command illustration also displays output, it will appear as follows:

$ ls -l

-rw-rw-r-- 1 dave dave  15 Jun 29 13:49 hashfilea.txt
-rwxrw-r-- 1 dave dave 627 Jun 29 13:50 hashsearch.sh

Note the use of the $ character in the illustration that includes output. The leading $ character is not part of the command, but is meant to represent the simple prompt of the shell command line. It is shown to help you differentiate between the command (as you would type it) and its output to the terminal. The blank line separating the command from its output in these examples will not appear when you run the command. Again, this is to separate the command from the output of the command.

Windows command examples are run using Git Bash, not the Windows command prompt unless explicitly stated.

Running Linux and bash on Windows

The bash shell and the commands we discuss are installed by default on virtually all distributions of Linux. The same is not true for the Windows environment. Thankfully, there are a variety of methods for running Linux commands and bash scripts on Windows systems. The four options we cover here are Git Bash, Cygwin, the Windows Subsystem for Linux, and the Windows Command Prompt and PowerShell.

Git Bash

You can run many standard Linux commands and the bash shell in the Windows environment if you have installed Git, which includes a port of bash. Git Bash is the method of choice for the examples presented in this book because of its popularity, and its ability to run standard Linux and bash commands as well as call many native Windows commands.

You can download Git from the Git website. Once it’s installed, you can run bash by right-clicking on the desktop or in a folder and selecting Git Bash Here.

Cygwin

Cygwin is a full-featured Linux emulator that also includes the ability to install a variety of packages. It is similar to Git Bash in that it allows calling many native Windows commands in addition to the standard Linux commands. Cygwin can be downloaded from the project website.

Windows Subsystem for Linux

Windows 10 includes a native method to run Linux (and hence bash) if the Windows Subsystem for Linux (WSL) is installed. To install WSL, follow these steps:

1. Click the Windows 10 search box.

2. Search for Control Panel.

3. Click Programs and Features.

4. Click “Turn Windows features on or off.”

5. Select the “Windows Subsystem for Linux” checkbox.

6. Restart the system.

7. Open the Windows Store.

8. Search for Ubuntu and install it.

9. After Ubuntu is installed, open the Windows Command Prompt and type ubuntu.

Note that when using a WSL Linux distribution in this manner, you can run bash scripts and mount the Windows filesystem, but you cannot make system calls to native Windows commands as you can with Git Bash and Cygwin.

Tip

Once you have installed WSL, you can choose to install versions of Linux other than Ubuntu, such as Kali, by visiting the Windows Store.

Windows Command Prompt and PowerShell

Once you have installed the Windows Subsystem for Linux, you have the ability to run Linux commands and bash scripts directly from the Windows Command Prompt and PowerShell as well by using the bash -c command.

For example, you can run the Linux pwd command from the Windows Command Prompt against your current working directory:

C:\Users\Paul\Desktop>bash -c "pwd"

/mnt/c/Users/Paul/Desktop

If you have multiple Linux distributions installed as part of WSL, you can use the distribution name in place of bash when invoking a command:

C:\Users\Paul\Desktop>ubuntu -c "pwd"

/mnt/c/Users/Paul/Desktop

You can also use this method to execute packages installed within your WSL Linux distribution that have a command-line interface, such as Nmap.

This seemingly minor addition gives you the ability to leverage the entire arsenal of Linux commands, packages, and bash capabilities from within the Windows Command Prompt, and from batch and PowerShell scripts.

Command-Line Basics

The command line is a generic term that refers to the means by which commands were given to an interactive computer system before the invention of GUIs. On Linux systems, it is the input to the bash (or other) shell. One of the basic operations of bash is to execute a command—that is, to run another program. When several words appear on the command line, bash assumes that the first word is the name of the program to run and the remaining words are the arguments to the command. For example, to have bash run the command called mkdir and to pass it two arguments -p and /tmp/scratch/garble, you would type this:

mkdir -p /tmp/scratch/garble

By convention, programs generally have their options located first, and have them begin with a leading -, as is the case here with the -p option. This particular command is being told to create a directory called /tmp/scratch/garble. The -p option indicates the user’s selection of a particular behavior—namely, that no errors will be reported and any intervening directories will be created (or attempted) as needed (e.g., if only /tmp exists, then mkdir will first create /tmp/scratch before attempting to create /tmp/scratch/garble).

Commands, Arguments, Built-ins, and Keywords

The commands that you can run are either files, built-ins, or keywords.

Files are executable programs. They may be files that are the result of a compile process and now consist of machine instructions. An example of this is the ls program. You can find that file in most Linux filesystems at /bin/ls.

Another type of file is a script, a human-readable text file, in one of several languages that your system may support by means of an interpreter (program) for that language. Examples of these scripting languages are bash, Python, and Perl, just to name a few. We’ll create some scripts (written in bash) in the chapters ahead.

Built-ins are part of the shell. They look like executables, but there is no file in the filesystem that is loaded and executed to do what they do. Instead, the work is done as part of the shell. The pwd command is an example of a built-in. It is faster and more efficient to use a built-in. Similarly, you, the user, can define functions within the shell that will be used much like built-in commands.

There are other words that look like commands but are really just part of the language of the shell. The if is an example. It is often used as the first word on a command line, but it isn’t a file; it’s a keyword. It has a syntax associated with it that may be more complex than the typical command -options arguments format of the command line. We describe many of these keywords in brief in the next chapter.

You can use the type command to identify whether a word is a keyword, a built-in, a command, or none of those. The -t option keeps the output to a single word:

$ type -t if

keyword

$ type -t pwd

builtin

$ type -t ls

file

You can use the compgen command to determine what commands, built-ins, and keywords are available to you. Use the -c option to list commands, -b for built-ins, and -k for keywords:

$ compgen -k

if
then
else
elif
.
.
.

If this distinction seems confusing at this point, don’t worry about it. You often don’t need to know the difference, but you should be aware that using built-ins and keywords are so much more efficient than commands (executables in external files), especially when invoked repeatedly in a loop.

Standard Input/Output/Error

A running program is called, in operating systems jargon, a process. Every process in the Unix/Linux/POSIX (and thus Windows) environment has three distinct input/output file descriptors. These three are called standard input (or stdin, for short), standard output (stdout), and standard error (stderr).

As you might guess by its name, stdin is the default source for input to a program—by default, the characters coming from the keyboard. When your script reads from stdin, it is reading characters typed on the keyboard or (as you shall see shortly) it can be changed to read from a file. Stdout is the default place for sending output from a program. By default, the output appears in the window that is running your shell or shell script. Standard error can also be sent output from a program, but it is (or should be) where error messages are written. It’s up to the person writing the program to direct any output to either stdout or stderr. So be conscientious when writing your scripts to send any error messages not to stdout but to stderr.

Redirection and Piping

One of the great innovations of the shell was that it gave you a mechanism whereby you could take a running program and change where it got its input and/or change where it sent its output without modifying the program itself. If you have a program called handywork that reads its input from stdin and writes its results to stdout, you can change its behavior as simply as this:

handywork < data.in  > results.out

This will run handywork but will have the input come not from the keyboard but instead from the data file called data.in (assuming such a file exists and has input in the format we want). Similarly, the output is being sent not to the screen but into a file called results.out (which will be created if it doesn’t exist and overwritten if it does). This technique is called redirection because we are redirecting input to come from a different place and redirecting output to go somewhere other than the screen.

What about stderr? The syntax is similar. We have to distinguish between stdout and stderr when redirecting data coming out of the program, and we make this distinction through the use of the file descriptor numbers. Stdin is file descriptor 0, stdout is file descriptor 1, and stderr is file descriptor 2, so we can redirect error messages this way:

handywork 2> err.msgs

This redirects only stderr and sends any such error message output to a file we call err.msgs (for obvious reasons).

Of course, we can do all three on the same line:

handywork < data.in  > results.out  2> err.msgs

Sometimes we want the error messages combined with the normal output (as it does by default when both are written to the screen). We can do this with the following syntax:

handywork < data.in  > results.out 2>&1

This says to send stderr (2) to the same location as file descriptor 1 (&1). Note that without the ampersand, the error messages would just be sent to a file named 1. This combining of stdout and stderr is so common that there is a useful shorthand notation:

handywork < data.in  &> results.out

If you want to discard standard output, you can redirect it to a special file called /dev/null as follows:

handywork < data.in > /dev/null

To view output on the command line and simultaneously redirect that same output to a file, use the tee command. The following displays the output of handywork to the screen and also saves it to results.out:

handywork < data.in | tee results.out

Use the -a option on the tee command to append to its output file rather than overwrite it. The | character is known as a pipe. It allows you to take the output from one command or script and provide it as input into another command. In this example, the output of handywork is piped into the tee command for further processing.

A file will be created or truncated (i.e., content discarded) when output is redirected using the single greater-than sign. If you want to preserve the file’s existing content, you can, instead, append to the file by using a double greater-than sign, like this:

handywork < data.in  >> results.out

This executes handywork, and then any output from stdout will be appended to the file results.out rather than overwriting its existing content.

Similarly, this command line:

handywork < data.in  &>> results.out

executes handywork and then appends both stdout and stderr to the file results.out rather than overwriting its existing content.

Running Commands in the Background

Throughout this book, we will be going beyond one-line commands and will be building complex scripts. Some of these scripts can take a significant amount of time to execute, so much so that you may not want to spend time waiting for them to complete. Instead, you can run any command or script in the background by using the & operator. The script will continue to run, but you can continue to use the shell to issue other commands and/or run other scripts. For example, to run ping in the background and redirect standard output to a file, use this command:

ping 192.168.10.56 > ping.log &

You will likely want to redirect both standard output and/or standard error to a file when sending tasks to the background, or the task will continue to print to the screen and interrupt other activities you are performing:

ping 192.168.10.56 &> ping.log &

Warning

Be careful not to confuse & (which is used to send a task to the background) and &> (which is used to perform a combined redirect of standard output and standard error).

You can use the jobs command to list any tasks currently running in the background:

$ jobs

[1]+  Running                 ping 192.168.10.56 > ping.log &

Use the fg command and the corresponding job number to bring the task back into the foreground:

$ fg 1

ping 192.168.10.56 > ping.log

If your task is currently executing in the foreground, you can use Ctrl-Z to suspend the process and then bg to continue the process in the background. From there, you can use jobs and fg as described previously.

From Command Line to Script

A shell script is just a file that contains the same commands that you could type on the command line. Put one or more commands into a file and you have a shell script. If you called your file myscript, you can run that script by typing bash myscript or you can give it execute permission (e.g., chmod 755 myscript) and then you can invoke it directly to run the script: ./myscript. We often include the following line as the first line of the script, which tells the operating system which scripting language we are using:

#!/bin/bash -

Of course, this assumes that bash is located in the /bin directory. If your script needs to be more portable, you could use this approach instead:

#!/usr/bin/env bash

It uses the env command to look up the location of bash and is considered the standard way to address the portability problem. It makes the assumption, however, that the env command is to be found in /usr/bin.

Summary

The command line is analogous to a physical multitool. If you need to drive a screw into a piece of wood, the best choice is a specialized tool such as a hand or power screwdriver. However, if you are stranded in the woods with limited resources, there is nothing better than a multitool. You can use it to drive a screw into a piece of wood, cut a length of rope, and even open a bottle. The same is true for the command line: its value is not in how well it can perform one particular task, but in its versatility and availability.

In recent years, the bash shell and Linux commands have become ubiquitous. By using Git Bash or Cygwin, you can easily access these capabilities from the Windows environment. For even more capability, you can install the Windows Subsystem for Linux, which gives you the ability to run full versions of Linux operating systems and access the capabilities directly from the Windows Command Prompt and PowerShell.

In the next chapter, we discuss the power of scripting, which comes from being able to run commands repeatedly, make decisions, and loop over a variety of inputs.

Workshop

1. Write a command that executes ifconfig and redirects standard output to a file named ipaddress.txt.

2. Write a command that executes ifconfig and redirects standard output and appends it to a file named ipaddress.txt.

3. Write a command that copies all of the files in the directory /etc/a to the directory /etc/b and redirects standard error to the file copyerror.log.

4. Write a command that performs a directory listing (ls) on the root file directory and pipes the output into the more command.

5. Write a command that executes mytask.sh and sends it to the background.

6. Given the following job list, write the command that brings the Amazon ping task to the foreground:

   [1]   Running                 ping www.google.com > /dev/null &
   [2]-  Running                 ping www.amazon.com > /dev/null &
   [3]+  Running                 ping www.oreilly.com > /dev/null &

Visit the Cypersecurity Ops website for additional resources and the answers to these questions.