book

Introduction to Programming for Researchers: Learning Programming Fundamentals Through Dataset Processing in Bash and Python

Name: Introduction to Programming for Researchers: Learning Programming Fundamentals Through Dataset Processing in Bash and Python
Author: James R. Derry
ISBN: 9798868816154

by James R. Derry

January 2026

Intermediate to advanced

464 pages

13h 42m

English

Apress

Read now

Unlock full access

Introduction to Programming for Researchers
Introduction
Acknowledgements
Contents
About the Authors
About the Technical Reviewers
1. Introduction
1.1 Modern Computers and Their History1.1.1 Today, Most Personal Computers Are Used Primarily As Communication Devices1.1.2 A Brief History of the Computer Age1.2 Our Modern Idea of Computers: A Theory of Computation
2. Digital Computation
2.1 Fundamentals of Computation I: Transistors, Logic Gates, and Moore’s Law2.1.1 Transistors and Moore’s Law2.1.2 The Transistor Is the Fundamental Physical Unit of Computation2.1.3 Transistors Are Organized into Logic Gates2.1.4 Moore’s Law2.1.5 The Transistor Budget2.2 Fundamentals of Computation II: Bits, Boolean Logic, and the Digital Age of George Boole and Claude Shannon2.2.1 George Boole2.2.2 Claude Shannon2.3 Fundamentals of Computation III: Data, Instructions, and Pointers2.4 Code and Data: Cycles of Fetch, Decode, and Execute, Over and Over
3. Operating Systems
3.1 Operating Systems and Linux3.1.1 A Brief History of Operating Systems, with an Emphasis on UNIX3.2 The UNIX/Linux Filesystem3.3 The Memory Manager and Process Scheduler3.4 Working with Datafiles in Linux3.5 An Introduction to the Process Scheduler
4. Introduction to Bash
4.1 The Bash Shell4.2 Changing the Bash Prompt4.3 Navigating Bash History and the LINUX Filesystem4.4 Files in LINUX4.4.1 Clobbering a File4.4.2 Changing File PermissionsUsing the chmod CommandWith Base-8 Numbers (Octals)4.4.3 Setting the Session noclobber Flag4.5 Character Encoding and Text File Formats4.6 The UNIX Philosophy: An Introduction4.7 The Bash Interpreter4.7.1 Variables4.7.2 The Bash EnvironmentWriting Bash Scripts Using Bash Environment Variables4.7.3 The Bash Interpreter4.7.4 The Symbol Table and Variables4.8 Some Bash Tools for Working with Datafiles4.9 The General-Purpose Bash Commands time and watch4.10 The Bash Pipeline4.11 Some Advice for Writing Bash Scripts and Pipelines

5. Bash: Combining Commands and Variables to Make Pipelines and Scripts to Process Data
5.1 Introduction to vim and the Bash script: How to Make an Executable Script That Runs on the Command Line and Takes Arguments5.1.1 Linting Our Executable Script5.1.2 Review5.2 tr Command: Translate or Delete Characters5.2.1 A Bash Pipeline SpellcheckerCan We Do Better?5.2.2 A DOS to Linux Files Converter5.3 Extracting and Analyzing Data from Datafiles Using Bash Pipelines5.4 gawk: How Many Named Stars Are There?5.5 Getting Resultsets from Bash Queries5.5.1 Advanced Subject: Format Printing5.6 How to Make an Executable Script That Prompts Users for Input5.7 Datetime in Bash5.8 Introduction to Regular Expressions Using grep -E5.9 Words You Can Make on a Calculator5.10 Regular Expressions, sed, and tr: Reformatting Records in a Dataset5.11 Finding Approximate Matches with agrep5.12 Write Once, Run Everywhere: Embedding Our Executable Scripts in Pipelines and Invoking Them in Other Scripts
6. Algorithms and Coding
6.1 An Introduction to Algorithms6.1.1 Recipes as Algorithms6.1.2 Definition of an Algorithm from The Art of Computer Programming6.1.3 Control Flow6.1.4 Euclid GCD Algorithm6.1.5 The Little Hummer Card Trick6.1.6 The Bubblesort Algorithm6.1.7 Notes on the Mystical History of Algorithms6.2 An Introduction to Programming Style6.2.1 Richard Hamming on Programming Style, As Told by Brian Kernighan
7. Floating-Point Numbers
7.1 Floating-Point Numbers7.2 Working with Floats: Rules of Thumb7.3 Arbitrary Precision Math with mpmath7.4 Improving the Accuracy of Floating-Point Calculations with Herbie
8. Introduction to Python
8.1 Python Primer8.1.1 Python Comes with Built-Ins: Built-In Functions, Built-In Datatypes and Collections, and Built-In Modules8.1.2 Python Comes with a Built-In Error-Reporting Module Called the Traceback8.1.3 The IPython Interactive Shell I8.1.4 In Python, Everything Is an Object
9. Using Python As a Calculator
9.1 Datetime I: Duration of the COVID-19 Pandemic9.2 Datetime II: Solving Date Problems9.3 Heat Loss9.4 Find the GC Content Percentage of a Nucleotide String9.5 Stoichiometry with SymPy9.6 Ideal Gas Law9.7 Work Performed by Expanding Gas9.8 Acceleration of Sun on Earth9.9 Sound Level9.10 SymPy on Jupyter Notebooks9.10.1 The Jupyter Notebook9.11 Falling Bodies9.12 Spherical Trigonometry in Navigation9.13 Climate Data9.14 Digital Signals9.15 Image-Driven Data Analysis: Flood Mitigation
10. Programming
10.1 Our First Program: Of Functions, Modules, Garbage Filters, Tests, and Docstrings10.1.1 The Least Necessary to Write a Working Function10.1.2 The Least Necessary to Write a Useful Function10.1.3 Documenting Our Function for Coders and Users10.1.4 Saving Our Function to a Module10.1.5 Autoreloading Edited Module Content to an IPython Session10.1.6 Writing Garbage Filters: Handling Bad Input10.1.7 Automating Testing Our Code Using Unit Tests10.1.8 Linting Our Code with Pylint10.2 Introduction to Programming I: Implementing Algorithms10.2.1 The IPython Interactive Shell II10.2.2 Back to Python and Coding10.3 Testing If Symbol in String Is Nucleotide10.4 Euclid GCD10.5 Converting Decimal Fractions into Binary10.5.1 The More You Know: Brahmagupta (598–668ce)10.6 Introduction to Programming II: Unit Testing10.6.1 Unit Tests and Our unittest_template.py File10.7 Finding the Reverse Complement of a Nucleotide String10.7.1 A Brief Introduction to the Python Dictionary10.7.2 Developing Code by Test-Driven Development (TDD)10.8 Counting Symbols in a String10.8.1 Error-Trapping10.9 IPython Magics10.10 Introduction to Programming V: Good Programming Practices10.11 Programs = Algorithms + Data Structures10.12 Interlude: Ilayda Develops Her Stoichiometry Code
11. Functions
11.1 Subroutines: The Genesis of Functions in Programming Languages11.2 Functions11.3 Modules11.4 Documenting Your Functions, Making Them Robust with Error-Trapping and Exception Handling, and Proofing Them11.5 Positional vs. Named Arguments11.6 Multiple Return Values from a Python Function11.7 Lambdas11.8 Matters of Style When Writing Functions11.9 A Calculus Primer: Numeric Integration and Differentiation Using Python11.9.1 Numerical Integration11.9.2 The More You Know11.9.3 Back to Numerical Integration11.9.4 Using Python and Numpy for Numerical Integration11.9.5 Numerical Differentiation11.9.6 Numerical Calculus with SymPy
12. Software Design
12.1 Writing Programs: Top-Down Design Methodology12.2 Writing Programs: Converting a Top-Down Design into a Program of Subroutines12.3 Writing Your Code Base As a Set of Files12.4 Writing Programs: A Practical Perspective
13. Working with Datasets
13.1 Accessing the Tabular Contents of Datafiles in Python Using a List of Lists (LoL)13.2 Nested Collections13.3 Finding the Minimum and Maximum Values in an Unsorted Collection13.4 Parsers13.4.1 Revisiting fasta Parsers13.5 Too Big to Handle: Preprocessing Large Datasets, Extracting Only Needed Dimensions13.6 One Record per Text File
14. Programming Efficiency
14.1 The Analysis of Algorithms14.2 O(n): Finding a Value in an Unsorted Collection of Index/Value Pairs14.3 O(nm): Nested Loops and Their Time Complexity14.3.1 Illustrating Executing Nested Loops with Nested Dolls14.3.2 The Output from Running Our Example Nested Loop14.3.3 Can We Do Better? Algebra to the Rescue!14.4 O(ln2 n): Binary Trees14.5 Functional Equivalence and Profiling Code14.5.1 Dictionary As Lookup Table vs. Conditional Testing14.6 Finding Min and Max Values in an Unsorted Collection II14.7 Multiple Passes Through Dataset vs. Single Pass14.7.1 An Outline of the Problem and a Solution14.7.2 Extending the Solution
15. Other Subjects
15.1 An Introduction to Graph Theory15.1.1 Saving Our Python Collections by Pickling Them15.2 Writing Python Scripts That Write Scripts15.3 Interactive Scripts That Prompt Users for Input15.4 The Python Half-Open Interval, Range Objects, and Slicing15.4.1 The Python Half-Open Interval15.4.2 The Range Object Revisited15.5 Finding Intervals with Overlap15.6 Finding Interval Overlap in Genomic Sequences15.7 Slicing Lists and Strings15.8 From Nucleotide String to Amino Acid Strings15.9 Comprehension15.9.1 Slicing LoL, Extracting Columns with Comprehension15.10 The Sieve of Eratosthenes15.11 Transposing a Matrix15.11.1 Transposing Tabular Datasets in Python15.12 Stacks and Queues15.12.1 Stacks15.12.2 Queues15.12.3 Algorithm: The Josephus Problem15.13 Recursion15.13.1 A Few Recursive Functions
16. SciPy
16.1 matplotlib: Graphics with SciPy16.2 NetworkX: Working with Graphs16.3 NumPy: Foundational Library of SciPy16.3.1 The ndarray16.3.2 Linear Algebra Has Three Objects: Scalar, Vector, and Matrix16.3.3 Single-Instruction Multiple Data Registers in CPUs16.3.4 Universal Functions (ufuncs) and Vectorized Operations16.3.5 Row and Column Vectors in Memory and Data Processing16.4 Linear Algebra16.4.1 Datasets As Matrices I16.4.2 Making a Rotatable 3D Graph from Data in a Dataset16.4.3 Datasets As Matrices II: Partitioning a Matrix16.5 Pandas: Working with Labeled Datasets in Pandas16.6 Pandas: Using Masks to Query Recordsets16.7 Pandas: Getting Statistics on Datasets16.7.1 Using Seaborn16.8 Pandas: Processing Datasets Programmatically16.9 SymPy: Symbolic Python
17. Odds and Ends
17.1 Writing Programs: Writing, Rewriting, and Matters of Style17.2 Python Sets17.3 Datetime in Datasets17.3.1 Filling In Missing Datetime Entries17.4 Introduction to Parallel Programming
18. Writing a Large Project
18.1 Putting It All Together: Solving Triangles18.2 Design Considerations18.3 Input and Output18.4 Organization of the Code18.5 Test-Driven Development (TDD) and Unit Testing18.6 Linting Our Code18.7 Pencil to Paper: Our Top-Down Design18.8 Our First Unit Test18.9 Our First Draft of solve_triangle.py18.10 Running Our First Unit Test18.11 Running solve_triangle Function the First Time18.12 A Note on Structured Design18.13 Adding Design Comments18.14 Writing Our First Draft18.15 Testing Our Code18.16 The Garbage Filter: Writing Unit Tests and Coding18.17 Finishing solve_triangle(), v118.18 Improving solve_triangle(), v118.19 TI-59 Calculator: Triangle Solution, Master Library ROM Module—A Different Approach18.20 Losing the rnd Bool from the Argument List18.21 Rethinking the Main Section of solve_triangle()18.22 solve_triangle(), v218.23 What’s Left to Do?
Suggested Reading
Chapter 1Chapter 2Chapter 3Chapter 6Chapter 7
References
Chapter 1Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 9Chapter 10Chapter 11Chapter 12Chapter 14Chapter 15Chapter 16Chapter 18
Index

Content preview from Introduction to Programming for Researchers: Learning Programming Fundamentals Through Dataset Processing in Bash and Python

J. R. DerryIntroduction to Programming for Researchershttps://doi.org/10.1007/979-8-8688-1615-4_13

13. Working with Datasets

James R. Derry¹

(1)

Austin, TX, USA

Of all aspects of research that can or does benefit from automation, the one common to all STEM research is the processing of datasets, especially large datasets. Our focus will be on the most common type of datasets, tabular datasets, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Python Made Easy: A Beginner’s Guide to Coding, Data Structures, and Practical Applications

Publisher Resources

ISBN: 9798868816154Purchase Link Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Introduction to Programming for Researchers: Learning Programming Fundamentals Through Dataset Processing in Bash and Python

by James R. Derry

13. Working with Datasets

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.