book

Developing Bioinformatics Computer Skills

by Per Jambeck, Cynthia Gibas

April 2001

Intermediate to advanced

448 pages

12h 41m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Copyright
Preface
Audience for This BookStructure of This BookOur Approach to BioinformaticsURLs Referenced in This BookConventions Used in This BookComments and QuestionsAcknowledgments
I. Introduction
1. Biology in the Computer Age
1.1. How Is Computing Changing Biology?1.1.1. The Eye of the Fly1.1.2. Labels in Gene Sequences1.1.3. Comparing eyeless and aniridia with BLAST1.2. Isn't Bioinformatics Just About Building Databases?1.2.1. The First Information Age in Biology1.3. What Does Informatics Mean to Biologists?1.4. What Challenges Does Biology Offer Computer Scientists?1.5. What Skills Should a Bioinformatician Have?1.6. Why Should Biologists Use Computers?1.6.1. A New Approach to Data Collection1.7. How Can I Configure a PC to Do Bioinformatics Research?1.7.1. Why Use Unix or Linux?1.8. What Information and Software Are Available?1.8.1. Why Do I Need to Install a Program from the Web?1.9. Can I Learn a Programming Language Without Classes?1.10. How Can I Use Web Information?1.11. How Do I Understand Sequence Alignment Data?1.12. How Do I Write a Program to Align Two Biological Sequences?1.13. How Do I Predict Protein Structure from Sequence?1.14. What Questions Can Bioinformatics Answer?
2. Computational Approaches to Biological Questions
2.1. Molecular Biology's Central Dogma2.1.1. Replication of DNA2.1.2. Genomes and Genes2.1.3. Transcription of DNA2.1.4. Translation of mRNA2.1.5. Molecular Evolution2.2. What Biologists Model2.2.1. Accessing 3D Molecules Through a 1D Representation2.2.2. Abstractions for Modeling Protein Structure2.2.3. Mathematical Modeling of Biochemical Systems2.3. Why Biologists Model2.4. Computational Methods Covered in This Book2.5. A Computational Biology Experiment2.5.1. Identifying the Problem2.5.2. Separating the Problem into Simpler Components2.5.3. Evaluating Your Needs2.5.4. Selecting the Appropriate Data Set2.5.5. Identifying the Criteria for Success2.5.6. Performing and Documenting a Computational Experiment2.5.6.1. Documentation issues in computational biology2.5.6.2. Electronic notebooks
II. The Bioinformatics Workstation
3. Setting Up Your Workstation
3.1. Working on a Unix System3.1.1. What Does an Operating System Do?3.1.2. Why Use Unix?3.1.3. Different Flavors of Unix3.1.3.1. Linux3.1.3.1.1. Will Linux run on your computer?3.1.3.2. Other common flavors3.1.4. Graphical Interfaces for Unix3.2. Setting Up a Linux Workstation3.2.1. Installing Linux3.2.1.1. System requirements3.2.1.2. Partitioning your disk3.2.1.3. Selecting major package groupings3.2.1.4. Other useful packages to add3.3. How to Get Software Working3.3.1. Unix tar Archives3.3.2. Binary Distributions3.3.3. RPM Archives3.3.3.1. GnoRPM3.3.4. Source Distributions3.3.5. Perl Scripts3.3.6. Putting It in Your Path3.3.7. Sharing Software Among Multiple Users3.4. What Software Is Needed?
4. Files and Directories in Unix
4.1. Filesystem Basics4.1.1. Moving Around the Directory Hierarchy4.1.2. Paths to Files and Directories4.1.3. Using a Process-Based File Hierarchy4.1.4. Establishing File-Naming Conventions for Your Work4.1.5. Structuring a Project: An Example4.2. Commands for Working with Directories and Files4.2.1. Moving Around the Filesystem4.2.1.1. You are here: pwd4.2.1.2. Changing directories with cd4.2.2. Finding Files and Directories4.2.2.1. Listing files with ls4.2.2.2. Interpreting ls output4.2.2.3. Finding files with find4.2.2.4. Finding an executable file with which4.2.2.5. Finding an executable file with whereis4.2.3. Manipulating Files and Directories4.2.3.1. Copying files and directories with cp4.2.3.2. Moving and renaming files and directories with mv4.2.3.3. Creating new links to files and directories with ln4.2.3.4. Creating and removing directories with mkdir and rmdir4.2.3.5. Removing files with rm4.3. Working in a Multiuser Environment4.3.1. Users and Groups4.3.2. User Directories4.3.3. File Permissions and Statistics4.3.3.1. Viewing file attributes with stat4.3.3.2. Changing file ownership and permissions with chmod4.3.3.3. Changing file and directory ownership with chown and chgrp4.3.4. System Administration4.3.5. Conventions for Organizing Files4.3.6. Locating Files in System Directories
5. Working on a Unix System
5.1. The Unix Shell5.1.1. What Flavors of Shell Are There?5.2. Issuing Commands on a Unix System5.2.1. The Command-Line Format5.2.2. Unix Information Commands5.2.3. Standard Input and Output5.2.4. Redirection of Command Input and Output5.2.5. Operators5.2.6. Wildcard Characters5.2.7. Running X Commands5.3. Viewing and Editing Files5.3.1. Viewing and Combining Files with cat5.3.2. more: A Step in the Right Direction5.3.3. less: The Gold Standard5.3.4. Editing Files with vi and vim5.3.5. The GNU Emacs Editor5.3.6. Viewing Binary Files with strings5.3.7. od and Binary Data5.4. Transformations and Filters5.4.1. Extracting the Beginning of a File with head5.4.2. Extracting the End of a File with tail5.4.3. Splitting Files with split and csplit5.4.4. Separating File Components with cut5.4.5. Combining Files with paste5.4.6. Merging Datafiles with join5.4.7. Sorting Files with sort5.4.7.1. Specifying sort keys5.5. File Statistics and Comparisons5.5.1. Comparing Files with cmp and diff5.5.2. Counting Words with wc5.6. The Language of Regular Expressions5.6.1. Searching for Patterns with grep5.7. Unix Shell Scripts5.8. Communicating with Other Computers5.8.1. The Web5.8.2. IP Addresses and Hostnames5.8.3. telnet5.8.4. ftp5.8.5. Displaying from a Remote Terminal5.8.6. Communication and File Sharing5.8.7. Media Compatibility5.8.8. Accessing Devices as Unix Filesystems5.8.9. Accessing Devices as DOS Disks5.9. Playing Nicely with Others in a Shared Environment5.9.1. Processes and Process Management5.9.1.1. Checking the load average5.9.1.2. Listing processes with ps5.9.1.3. top5.9.1.4. Signaling processes with kill5.9.1.5. Setting process priorities with nice and renice5.9.2. Scheduling Recurring Activities with cron5.9.2.1. Submitting jobs to cron using crontab5.9.2.2. Using cron to schedule a recurrent database search5.9.2.3. Scheduling processes with batch and at5.9.3. Monitoring Space Usage and File Sizes5.9.3.1. Checking disk usage with du5.9.3.2. Checking for free disk space with df5.9.3.3. Checking your compliance with system quotas with quota5.9.4. Creating Archives of Your Data5.9.4.1. tar: Hold the feathers5.9.4.2. compress5.9.4.3. gzip
III. Tools for Bioinformatics

6. Biological Research on the Web
6.1. Using Search Engines6.1.1. Boolean Searching6.1.2. Search Engine Algorithms6.2. Finding Scientific Articles6.2.1. Using PubMed Effectively6.3. The Public Biological Databases6.3.1. Data Annotation and Data Formats6.3.2. 3D Molecular Structure Data6.3.3. DNA, RNA, and Protein Sequence Data6.3.4. Genomic Data6.3.5. Biochemical Pathway Data6.3.6. Gene Expression Data6.4. Searching Biological Databases6.4.1. GenBank6.4.1.1. Saving search results6.4.1.2. Saving large result sets6.4.2. PDB6.5. Depositing Data into the Public Databases6.5.1. GenBank Deposition6.5.2. PDB Deposition6.6. Finding Software6.7. Judging the Quality of Information6.7.1. Authority6.7.2. Transparency6.7.3. Timeliness
7. Sequence Analysis, Pairwise Alignment, and Database Searching
7.1. Chemical Composition of Biomolecules7.2. Composition of DNA and RNA7.3. Watson and Crick Solve the Structure of DNA7.4. Development of DNA Sequencing Methods7.4.1. The Chemical Composition of Proteins7.4.2. Mechanisms of Molecular Evolution7.5. Genefinders and Feature Detection in DNA7.5.1. Predicting Gene Locations7.5.2. Feature Detection7.6. DNA Translation7.7. Pairwise Sequence Comparison7.7.1. Scoring Matrices7.7.2. Gap Penalties7.7.3. Dynamic Programming7.7.4. Global Alignment7.7.4.1. Using ALIGN to produce a global sequence alignment7.7.5. Local Alignment7.7.5.1. Tools for local alignment7.8. Sequence Queries Against Biological Databases7.8.1. Local Alignment-Based Searching Using BLAST7.8.1.1. The BLAST algorithm7.8.1.2. NCBI BLAST and WU-BLAST7.8.1.3. What do the various BLAST programs do?7.8.1.4. Building a local database with formatdb7.8.1.5. Evaluating BLAST results7.8.2. Local Alignment Using FASTA7.8.2.1. The FASTA algorithm7.8.2.2. The FASTA programs7.9. Multifunctional Tools for Sequence Analysis7.9.1. NCBI SEALS7.9.2. The Biology Workbench7.9.3. DoubleTwist
8. Multiple Sequence Alignments, Trees, and Profiles
8.1. The Morphological to the Molecular8.2. Multiple Sequence Alignment8.2.1. Progressive Strategies for Multiple Alignment8.2.2. Multiple Alignment with ClustalW8.2.3. Viewing and Editing Alignments with Jalview8.2.4. Sequence Logos8.3. Phylogenetic Analysis8.3.1. Phylogenetic Trees Based on Pairwise Distances8.3.2. Phylogenetic Trees Based on Neighbor Joining8.3.3. Phylogenetic Trees Based on Maximum Parsimony8.3.4. Phylogenetic Trees Based on Maximum Likelihood Estimation8.3.5. Software for Phylogenetic Analysis8.3.5.1. PHYLIP8.3.5.1.1. The PHYLIP input format8.3.5.2. Generating input for PHYLIP with ClustalX8.4. Profiles and Motifs8.4.1. Motif Databases8.4.1.1. Blocks8.4.1.2. PROSITE8.4.1.3. Pfam8.4.1.4. PRINTS8.4.1.5. COG8.4.1.6. Accessing multiple databases8.4.2. Constructing and Using Your Own Profiles8.4.2.1. Finding new motifs with MEME8.4.2.2. Searching for motifs with MAST and MetaMEME8.4.2.3. Motif discovery with other programs8.4.2.4. HMMer8.4.3. Incorporating Motif Information into Pairwise Alignment
9. Visualizing Protein Structures and Computing Structural Properties
9.1. A Word About Protein Structure Data9.2. The Chemistry of Proteins9.2.1. From 1D to 3D9.2.2. Interatomic Forces and Protein Structure9.2.2.1. Covalent interactions9.2.2.2. Hydrogen bonds9.2.2.3. Hydrophobic and hydrophilic interactions9.2.2.4. Charge-charge, charge-dipole, and dipole-dipole interactions9.2.2.5. Van der Waals forces9.2.2.6. Repulsive forces9.2.2.7. Relative strength of interatomic forces9.3. Web-Based Protein Structure Tools9.4. Structure Visualization9.4.1. Molecular Structure Viewers for Your Web Browser9.4.1.1. RasMol9.4.1.2. Cn3D9.4.1.3. SWISS-PDBViewer9.4.2. Standalone Modeling Packages9.4.2.1. MolMol9.4.2.2. MidasPlus9.4.2.3. VMD9.4.3. Creating High-Quality Graphics with MolScript9.4.4. Active Site Visualization with LIGPLOT9.4.5. dimplot9.5. Structure Classification9.5.1. Secondary Structure from Coordinates9.5.1.1. STRIDE9.5.2. Topology Cartoons9.5.2.1. TOPS9.5.3. Classification Databases9.5.3.1. SCOP9.5.3.2. CATH9.5.3.3. Unique protein structure data sets9.6. Structural Alignment9.6.1. Comparing Two Protein Structures9.6.1.1. ProFit9.6.2. DALI Domain Dictionary9.6.3. CE and CL9.6.4. VAST9.7. Structure Analysis9.7.1. Analyzing Structure Quality9.7.1.1. PROCHECK9.7.1.2. WHAT IF/ WHAT CHECK9.7.2. Intramolecular Interactions9.7.2.1. Computing contacts with HBPLUS9.8. Solvent Accessibility and Interactions9.8.1. Computing Solvent Accessibility with naccess9.8.2. Solvent Accessibility with Alpha Shapes9.9. Computing Physicochemical Properties9.9.1. Macromolecular Electrostatics9.9.2. Visualization of Molecular Surfaces with Mapped Properties9.9.2.1. GRASP/GRASS9.10. Structure Optimization9.10.1. Informatics Plays a Role in Optimization9.10.2. Rotamer Libraries9.10.3. PDFs9.11. Protein Resource Databases9.11.1. GeneCensus9.11.2. PRESAGE9.11.3. BIND9.12. Putting It All Together
10. Predicting Protein Structure and Function from Sequence
10.1. Determining the Structures of Proteins10.1.1. Solving Protein Structures by X-ray Crystallography10.1.2. Solving Structures by NMR Spectroscopy10.2. Predicting the Structures of Proteins10.2.1. CASP: The Search for the Holy Grail10.3. From 3D to 1D10.4. Feature Detection in Protein Sequences10.5. Secondary Structure Prediction10.5.1. Alignment-Based and Hybrid Methods10.5.2. Single Sequence Prediction Methods10.5.3. Measuring Prediction Accuracy10.5.4. Putting Predictions to Use10.5.5. Predicting Transmembrane Helices10.5.6. Threading10.6. Predicting 3D Structure10.6.1. Homology Modeling10.6.1.1. Modeller10.6.1.2. How Modeller builds a model10.6.1.3. ModBase: a database of automatically generated models10.6.1.4. The SWISS-MODEL server10.6.2. Tools for Ab-Initio Prediction10.7. Putting It All Together: A Protein Modeling Project10.7.1. Finding Homologous Structures10.7.2. Looking for Distant Homologies10.7.3. Predicting Secondary Structure from Sequence10.7.4. Using Threading Methods to Find Potential Folds10.7.5. Using Profile Methods to Align Distantly Related Sequences10.7.6. Building a Homology Model10.8. Summary
11. Tools for Genomics and Proteomics
11.1. From Sequencing Genes to Sequencing Genomes11.1.1. Analysis of Raw Sequence Data: Basecalling11.1.2. Sequencing an Entire Genome11.1.2.1. The shotgun approach11.1.2.2. The clone contig approach11.1.2.3. LIMS: Tracking all those minisequences11.2. Sequence Assembly11.3. Accessing Genome Informationon the Web11.3.1. NCBI Genome Resources11.3.2. TIGR Genome Resources11.3.3. EnsEMBL11.3.4. Other Sequencing Centers11.3.5. Organism-Specific Resources11.4. Annotating and Analyzing Whole Genome Sequences11.4.1. Genome Annotation11.4.1.1. MAGPIE11.4.2. Genome Comparison11.4.2.1. PipMaker11.4.2.2. MUMmer11.5. Functional Genomics: New Data Analysis Challenges11.5.1. Sequence-Based Approaches for Analyzing Gene Expression11.5.2. DNA Microarrays: Emerging Technologiesin Functional Genomics11.5.3. Bioinformatics Challenges in Microarray Design and Analysis11.5.3.1. Planning array experiments11.5.3.2. Analyzing scanned microarray images with CrazyQuant11.5.3.3. Visualizing high-dimensional data11.5.3.4. Clustering expression profiles11.5.3.5. A note on commercial software for expression analysis11.6. Proteomics11.6.1. Experimental Approaches in Proteomics11.6.2. Informatics Challenges in 2D-PAGE Analysis11.6.3. Tools for Proteomics Analysis11.6.4. Generalizing the Array Approach11.7. Biochemical Pathway Databases11.7.1. Illustration of a Complex Metabolic Pathway11.7.2. EC Nomenclature11.7.3. WIT and KEGG11.7.4. PathDB11.8. Modeling Kinetics and Physiology11.8.1. Modeling Kinetics with Gepasi11.8.2. XPP11.8.3. Using the Virtual Cell Portal11.9. Summary
IV. Databases and Visualization
12. Automating Data Analysis with Perl
12.1. Why Perl?12.1.1. Where Do I Get Perl?12.2. Perl Basics12.2.1. Hello World12.2.2. A Bioinformatics Example12.2.3. Variables12.2.3.1. Scalars12.2.3.2. Arrays12.2.3.3. Hashes12.2.4. Loops12.2.5. Subroutines12.3. Pattern Matching and Regular Expressions12.4. Parsing BLAST Output Using Perl12.5. Applying Perl to Bioinformatics12.5.1. Bioperl12.5.2. CGI.pm12.5.3. LWP12.5.4. PDL12.5.5. DBI12.5.6. GD
13. Building Biological Databases
13.1. Types of Databases13.1.1. Flat File Databases13.1.1.1. Flat file databases in biology13.1.2. Relational Databases13.1.2.1. How tables are organized13.1.2.2. The database schema13.1.3. Object-Oriented Databases13.2. Database Software13.2.1. Sequence Retrieval System13.2.2. Oracle13.2.3. PostgreSQL13.2.4. Open Source Object DBMS13.2.5. MySQL13.3. Introduction to SQL13.3.1. SQL Datatypes13.3.2. SQL Commands13.3.2.1. Adding a new table to a database13.3.2.2. Changing an existing table13.3.2.3. Adding data to an existing table13.3.2.4. Altering existing data in a table13.3.3. Accessing Your Database with the SQLSELECT Command13.3.3.1. Choosing fields to select13.3.3.2. Using a WHERE clause to specify selection conditions13.3.3.3. Joining output from multiple tables13.4. Installing the MySQL DBMS13.4.1. Setting Up MySQL13.4.1.1. Using the mysql client program13.4.1.2. Using the mysqladmin client program to set up MySQL13.4.1.3. Restarting the MySQL server13.4.2. Securing Your MySQL Server13.4.3. Setting Up the Data Directory13.4.4. Creating a New Database13.5. Database Design13.5.1. On Entities and Attributes13.5.2. Creating a Database from Your Data Model13.5.3. Creating Relationships Between Tables13.6. Developing Web-Based Software That Interacts with Databases13.6.1. CGI13.6.2. XML13.6.2.1. XML applications13.6.3. PHP13.6.3.1. Accessing MySQL databases with PHP13.6.3.2. Collecting information from a form with PHP
14. Visualization and Data Mining
14.1. Preparing Your Data14.2. Viewing Graphics14.2.1. xzgv14.2.2. Ghostview and gv14.2.3. The GIMP14.3. Sequence Data Visualization14.3.1. Making Publication-Quality Alignmentswith TEXshade14.3.2. Viewing Sequence Distances Geometrically14.4. Networks and Pathway Visualization14.5. Working with Numerical Data14.5.1. gnuplot and xgfe14.5.2. Grace: The Pocketknife of Data Visualization14.5.3. Multidimensional Analysis: XGobi and XGvis14.5.4. Programming for Data Analysis14.5.4.1. R and S-plus14.5.4.2. Online resources for R14.5.4.3. Matlab and Octave14.6. Visualization: Summary14.7. Data Mining and Biological Information14.7.1. Problems in Data Mining and Machine Learning14.7.1.1. Supervised and unsupervised learning14.7.2. A Collection of Data Mining Techniques14.7.2.1. Decision trees14.7.2.2. Neural networks14.7.2.3. Genetic algorithms14.7.2.4. Support vector machines
Bibliography
UnixSysAdminPerlGeneral ReferenceBioinformatics ReferenceMolecular Biology/Biology ReferenceProtein Structure and BiophysicsGenomicsBiotechnologyDatabasesVisualizationData Mining
Colophon

Content preview from Developing Bioinformatics Computer Skills

Chapter 4. Files and Directories in Unix

Now that you've set up your workstation, let's spend some time talking about how to get around in a Unix system. In this chapter, we introduce basic Unix concepts, including the structure of the filesystem, file ownership, and commands for moving around the filesystem and creating files.^[*] Another important focus of this chapter, however, is the approach you should take to organizing your research data so that it can be accessed efficiently by you and by others.

Filesystem Basics

All computer filesystems, whether on Unix systems or desktop PCs, are basically the same. Files are named locations on the computer's storage device. Each filename is a pointer to a discrete object with a beginning and end, whether it's a program that can be executed or simply a set of data that can be read by a program. Directories or folders are containers in which files can be grouped. Computer filesystems are organized hierarchically, with a root directory that branches into subdirectories and subdirectories of subdirectories.

This hierarchical system can help organize and share information, if used properly. Like the taxonomy of species developed by the early biologists, your file hierarchy should organize information from the general level to the specific. Each time the filesystem splits into subdirectories, it should be because there are meaningful divisions to be created within a larger class of files.

Why should you organize your computer files in a systematic, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Algorithmic and Artificial Intelligence Methods for Protein Bioinformatics

Publisher Resources

ISBN: 1565926641Catalog Page Errata

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Developing Bioinformatics Computer Skills

by Per Jambeck, Cynthia Gibas

Chapter 4. Files and Directories in Unix

Filesystem Basics

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.