book

Effective Computation in Physics

by Anthony Scopatz, Kathryn D. Huff

June 2015

Intermediate to advanced

550 pages

13h 27m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What Is This Book?Who This Book Is ForWho This Book Is Not ForCase Study on How to Use This Book: Radioactive Decay ConstantsAccessing Data and LibrariesCreating a Simple ProgramAutomating Data CollectionAnalyzing and Plotting the DataKeeping Track of ChangesTesting the CodeDocumenting the CodePublishingWhat to Do While Reading This BookConventions Used in This BookUsing Code ExamplesInstallation and SetupStep 1: Download and Install Miniconda (or Anaconda)Step 2: Install the PackagesSafari® Books OnlineHow to Contact UsAcknowledgments
Navigating the ShellThe Shell Is a Programming LanguagePaths and pwdHome Directory (~)Listing the Contents (ls)Changing Directories (cd)File Inspection (head and tail)Manipulating Files and DirectoriesCreating Files (nano, emacs, vi, cat, >, and touch)Copying and Renaming Files (cp and mv)Making Directories (mkdir)Deleting Files and Directories (rm)Flags and WildcardsGetting HelpReading the Manual (man)Finding the Right Hammer (apropos)Combining Utilities with Redirection and Pipes (>, >>, and |)Permissions and SharingSeeing Permissions (ls -l)Setting Ownership (chown)Setting Permissions (chmod)Creating Links (ln)Connecting to Other Computers (ssh and scp)The EnvironmentSaving Environment Variables (.bashrc)Running Programs (PATH)Nicknaming Commands (alias)Scripting with BashCommand Line Wrap-up
Running PythonCommentsVariablesSpecial VariablesBoolean ValuesNone Is Not Zero!NotImplemented Is Not None!OperatorsStringsString IndexingString ConcatenationString LiteralsString MethodsModulesImporting ModulesImporting Variables from a ModuleAliasing ImportsAliasing Variables on ImportPackagesThe Standard Library and the Python EcosystemPython Wrap-up
ListsTuplesSetsDictionariesContainers Wrap-up
Conditionalsif-else Statementsif-elif-else Statementsif-else ExpressionExceptionsRaising ExceptionsLoopswhile Loopsfor LoopsComprehensionsFlow Control and Logic Wrap-up
Functions in PythonKeyword ArgumentsVariable Number of ArgumentsMultiple Return ValuesScopeRecursionLambdasGeneratorsDecoratorsFunction Wrap-up
Object OrientationObjectsClassesClass VariablesInstance VariablesConstructorsMethodsStatic MethodsDuck TypingPolymorphismDecorators and MetaclassesObject Orientation Wrap-up

Preparing DataExperimental DataSimulation DataMetadataLoading DataNumPyPyTablesPandasBlazeCleaning and Munging DataMissing DataAnalysisModel-Driven AnalysisData-Driven AnalysisVisualizationVisualization ToolsGnuplotmatplotlibBokehInkscapeAnalysis and Visualization Wrap-up
Messy MagnetismMetacharacters on the Command LineListing Files with Simple PatternsGlobally Finding Filenames with Patterns (find)grep, sed, and awkFinding Patterns in Files (grep)Finding and Replacing Patterns in Files (sed)Finding and Replacing a Complex Patternsed ExtrasManipulating Columns of Data (awk)Python Regular ExpressionsRegular Expressions Wrap-up
ArraysdtypesSlicing and ViewsArithmetic and BroadcastingFancy IndexingMaskingStructured ArraysUniversal FunctionsOther Valuable FunctionsNumPy Wrap-up
Files in PythonAn Aside About Computer ArchitectureBig Ideas in HDF5File ManipulationsHierarchy LayoutChunkingIn-Core and Out-of-Core OperationsIn-CoreOut-of-CoreQueryingCompressionHDF5 UtilitiesStoring Data Wrap-up
Hash TablesResizingCollisionsData FramesSeriesThe Data Frame StructureB-TreesK-D TreesData Structures Wrap-up
Scale and ScalabilityProblem ClassificationExample: N-Body ProblemNo ParallelismThreadsMultiprocessingMPIParallelism Wrap-up
Deploying the Software ItselfpipCondaVirtual MachinesDockerDeploying to the CloudDeploying to SupercomputersDeployment Wrap-up
makeRunning makeMakefilesTargetsSpecial TargetsBuilding and Installing SoftwareConfiguration of the MakefileCompilationInstallationBuilding Software and Pipelines Wrap-up
What Is Version Control?The Lab Notebook of Computational PhysicsVersion Control Tool TypesGetting Started with GitInstalling GitGetting Help (git --help)Control the Behavior of Git (git config)Local Version Control with GitCreating a Local Repository (git init)Staging Files (git add)Checking the Status of Your Local Copy (git status)Saving a Snapshot (git commit)git log: Viewing the HistoryViewing the Differences (git diff)Unstaging or Reverting a File (git reset)Discard Revisions (git revert)Listing, Creating, and Deleting Branches (git branch)Switching Between Branches (git checkout)Merging Branches (git merge)Dealing with ConflictsVersion Conrol Wrap-Up
Repository Hosting (github.com)Creating a Repository on GitHubDeclaring a Remote (git remote)Sending Commits to Remote Repositories (git push)Downloading a Repository (git clone)Fetching the Contents of a Remote (git fetch)Merging the Contents of a Remote (git merge)Pull = Fetch and Merge (git pull)ConflictsResolving ConflictsRemote Version Control Wrap-up
Encountering a BugPrint StatementsInteractive DebuggingDebugging in Python (pdb)Setting the TraceStepping ForwardQuerying VariablesSetting the StateRunning Functions and MethodsContinuing the ExecutionBreakpointsProfilingViewing the Profile with pstatsViewing the Profile GraphicallyLine Profiling with KernprofLintingDebugging Wrap-up
Why Do We Test?When Should We Test?Where Should We Write Tests?What and How to Test?Running TestsEdge CasesCorner CasesUnit TestsIntegration TestsRegression TestsTest GeneratorsTest CoverageTest-Driven DevelopmentTesting Wrap-up
Why Prioritize Documentation?Documentation Is Very ValuableDocumentation Is Easier Than You ThinkTypes of DocumentationTheory ManualsUser and Developer GuidesReadme FilesCommentsSelf-Documenting CodeDocstringsAutomationSphinxDocumentation Wrap-up
Document ProcessingSeparation of Content from FormattingTracking ChangesText EditorsMarkup LanguagesLaTeXBibliographiesPublication Wrap-up
Ticketing SystemsWorkflow OverviewCreating an IssueAssigning an IssueDiscussing an IssueClosing an IssuePull Requests and Code ReviewsSubmitting a Pull RequestReviewing a Pull RequestMerging a Pull RequestCollaboration Wrap-up
What Is Copyrightable?Right of First PublicationWhat Is the Public Domain?Choosing a Software LicenseBerkeley Software Distribution (BSD) LicenseGNU General Public License (GPL)Creative Commons (CC)Other LicensesChanging the LicenseCopyright Is Not EverythingLicensing Wrap-up
Where to Go from Here

Content preview from Effective Computation in Physics

Chapter 14. Building Pipelines and Software

The most effective analysis and software tools are reused and shared often. Physics software, libraries, and analysis pipelines are no exception. Indeed, this book is intentionally titled Effective Computation in Physics. Effective computational physicists streamline repetitive work and liberate themselves from the mundane. They are also able to build upon the work of others by installing and using external libraries. This chapter will cover both strategies, since they are intricately related by an emphasis on automation and by a versatile tool called make.

By the end of this chapter, you should be able to get started with the following tasks:

Automating complex workflows and analysis pipelines
Configuring and installing external libraries
Creating build systems for your own software

By automating the tedious steps, you are much more likely to encourage colleagues to extend your work and to, therefore, positively impact the future of your field. This chapter will help you to assist your peers in building, installing, and reproducing your analysis pipeline by automating the various steps of building that pipeline and your software:

The first step, configuration, detects platform-dependent variables and user-specified options. It then uses those to customize later steps in the process. For example, an analysis pipeline might run a Python program on a data file to create a ...