Mastering UNIX® Shell Scripting: Bash, Bourne, and Korn Shell Scripting for Programmers, System Administrators, and UNIX Gurus, Second Edition

CHAPTER6

Automated FTP Stuff

In many shops the business relies on nightly, or even hourly, file transfers of data that is to be processed. Due to the importance of this data, the data movement must be automated. The extent of automation in the FTP (and SFTP) world is threefold. We want the ability to move outbound files to another site, move inbound files from a remote location to our local machine, and check a remote site on a regular basis for files that are ready to download. In this chapter we are going to create some shell scripts to handle each of these scenarios.

Most businesses that rely on this type of data movement also require some pre-FTP and post-FTP processing to ready the system for the files before the transfer takes place and to verify the data integrity or file permissions after the transfer. For this pre and post processing we need to build into the shell script the ability to either hard-code the pre- and post-processing events or point to a file that performs these tasks. Now we are up to five pieces of code that we need to create.

Before you settle on a technique to move the data, be sure to also look at remote copy (rcp), Secure Copy (scp), and rsync, which are covered in Chapter 7, “Using rsync to Efficiently Replicate Data.” If you need to move a large amount of data, rsync is your best option because of its built-in compression and the ability to update changes only, for certain file types. A database file cannot be updated with changes only, of course, ...