book

Effective Computation in Physics

by Anthony Scopatz, Kathryn D. Huff

June 2015

Intermediate to advanced

550 pages

13h 27m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What Is This Book?Who This Book Is ForWho This Book Is Not ForCase Study on How to Use This Book: Radioactive Decay ConstantsAccessing Data and LibrariesCreating a Simple ProgramAutomating Data CollectionAnalyzing and Plotting the DataKeeping Track of ChangesTesting the CodeDocumenting the CodePublishingWhat to Do While Reading This BookConventions Used in This BookUsing Code ExamplesInstallation and SetupStep 1: Download and Install Miniconda (or Anaconda)Step 2: Install the PackagesSafari® Books OnlineHow to Contact UsAcknowledgments
Navigating the ShellThe Shell Is a Programming LanguagePaths and pwdHome Directory (~)Listing the Contents (ls)Changing Directories (cd)File Inspection (head and tail)Manipulating Files and DirectoriesCreating Files (nano, emacs, vi, cat, >, and touch)Copying and Renaming Files (cp and mv)Making Directories (mkdir)Deleting Files and Directories (rm)Flags and WildcardsGetting HelpReading the Manual (man)Finding the Right Hammer (apropos)Combining Utilities with Redirection and Pipes (>, >>, and |)Permissions and SharingSeeing Permissions (ls -l)Setting Ownership (chown)Setting Permissions (chmod)Creating Links (ln)Connecting to Other Computers (ssh and scp)The EnvironmentSaving Environment Variables (.bashrc)Running Programs (PATH)Nicknaming Commands (alias)Scripting with BashCommand Line Wrap-up
Running PythonCommentsVariablesSpecial VariablesBoolean ValuesNone Is Not Zero!NotImplemented Is Not None!OperatorsStringsString IndexingString ConcatenationString LiteralsString MethodsModulesImporting ModulesImporting Variables from a ModuleAliasing ImportsAliasing Variables on ImportPackagesThe Standard Library and the Python EcosystemPython Wrap-up
ListsTuplesSetsDictionariesContainers Wrap-up
Conditionalsif-else Statementsif-elif-else Statementsif-else ExpressionExceptionsRaising ExceptionsLoopswhile Loopsfor LoopsComprehensionsFlow Control and Logic Wrap-up
Functions in PythonKeyword ArgumentsVariable Number of ArgumentsMultiple Return ValuesScopeRecursionLambdasGeneratorsDecoratorsFunction Wrap-up
Object OrientationObjectsClassesClass VariablesInstance VariablesConstructorsMethodsStatic MethodsDuck TypingPolymorphismDecorators and MetaclassesObject Orientation Wrap-up

Preparing DataExperimental DataSimulation DataMetadataLoading DataNumPyPyTablesPandasBlazeCleaning and Munging DataMissing DataAnalysisModel-Driven AnalysisData-Driven AnalysisVisualizationVisualization ToolsGnuplotmatplotlibBokehInkscapeAnalysis and Visualization Wrap-up
Messy MagnetismMetacharacters on the Command LineListing Files with Simple PatternsGlobally Finding Filenames with Patterns (find)grep, sed, and awkFinding Patterns in Files (grep)Finding and Replacing Patterns in Files (sed)Finding and Replacing a Complex Patternsed ExtrasManipulating Columns of Data (awk)Python Regular ExpressionsRegular Expressions Wrap-up
ArraysdtypesSlicing and ViewsArithmetic and BroadcastingFancy IndexingMaskingStructured ArraysUniversal FunctionsOther Valuable FunctionsNumPy Wrap-up
Files in PythonAn Aside About Computer ArchitectureBig Ideas in HDF5File ManipulationsHierarchy LayoutChunkingIn-Core and Out-of-Core OperationsIn-CoreOut-of-CoreQueryingCompressionHDF5 UtilitiesStoring Data Wrap-up
Hash TablesResizingCollisionsData FramesSeriesThe Data Frame StructureB-TreesK-D TreesData Structures Wrap-up
Scale and ScalabilityProblem ClassificationExample: N-Body ProblemNo ParallelismThreadsMultiprocessingMPIParallelism Wrap-up
Deploying the Software ItselfpipCondaVirtual MachinesDockerDeploying to the CloudDeploying to SupercomputersDeployment Wrap-up
makeRunning makeMakefilesTargetsSpecial TargetsBuilding and Installing SoftwareConfiguration of the MakefileCompilationInstallationBuilding Software and Pipelines Wrap-up
What Is Version Control?The Lab Notebook of Computational PhysicsVersion Control Tool TypesGetting Started with GitInstalling GitGetting Help (git --help)Control the Behavior of Git (git config)Local Version Control with GitCreating a Local Repository (git init)Staging Files (git add)Checking the Status of Your Local Copy (git status)Saving a Snapshot (git commit)git log: Viewing the HistoryViewing the Differences (git diff)Unstaging or Reverting a File (git reset)Discard Revisions (git revert)Listing, Creating, and Deleting Branches (git branch)Switching Between Branches (git checkout)Merging Branches (git merge)Dealing with ConflictsVersion Conrol Wrap-Up
Repository Hosting (github.com)Creating a Repository on GitHubDeclaring a Remote (git remote)Sending Commits to Remote Repositories (git push)Downloading a Repository (git clone)Fetching the Contents of a Remote (git fetch)Merging the Contents of a Remote (git merge)Pull = Fetch and Merge (git pull)ConflictsResolving ConflictsRemote Version Control Wrap-up
Encountering a BugPrint StatementsInteractive DebuggingDebugging in Python (pdb)Setting the TraceStepping ForwardQuerying VariablesSetting the StateRunning Functions and MethodsContinuing the ExecutionBreakpointsProfilingViewing the Profile with pstatsViewing the Profile GraphicallyLine Profiling with KernprofLintingDebugging Wrap-up
Why Do We Test?When Should We Test?Where Should We Write Tests?What and How to Test?Running TestsEdge CasesCorner CasesUnit TestsIntegration TestsRegression TestsTest GeneratorsTest CoverageTest-Driven DevelopmentTesting Wrap-up
Why Prioritize Documentation?Documentation Is Very ValuableDocumentation Is Easier Than You ThinkTypes of DocumentationTheory ManualsUser and Developer GuidesReadme FilesCommentsSelf-Documenting CodeDocstringsAutomationSphinxDocumentation Wrap-up
Document ProcessingSeparation of Content from FormattingTracking ChangesText EditorsMarkup LanguagesLaTeXBibliographiesPublication Wrap-up
Ticketing SystemsWorkflow OverviewCreating an IssueAssigning an IssueDiscussing an IssueClosing an IssuePull Requests and Code ReviewsSubmitting a Pull RequestReviewing a Pull RequestMerging a Pull RequestCollaboration Wrap-up
What Is Copyrightable?Right of First PublicationWhat Is the Public Domain?Choosing a Software LicenseBerkeley Software Distribution (BSD) LicenseGNU General Public License (GPL)Creative Commons (CC)Other LicensesChanging the LicenseCopyright Is Not EverythingLicensing Wrap-up
Where to Go from Here

Content preview from Effective Computation in Physics

Chapter 15. Local Version Control

In science, reproducibility is paramount. A fundamental principle of science, reproducibility is the requirement that experimental results from independent laboratories should be commensurate. In scientific computation simulations, data munging and analysis pipelines are experimental analogs. To ensure that results are repeatable, it must be possible to unwind code and analysis to previous versions, and to replicate plots. The most essential requirement is that all previous versions of the code, data, and provenance metadata must be robustly and retrievably archived. The best practice in scientific computing is called version control.

Rather than inventing a system of indexed directories holding full versions of your code from each day in the lab, the best practice in software development is to use a version control system that automates archiving and retrieval of text documents such as source code.

This chapter will explain:

What version control is
How to use it for managing files on your computer
And how to use it for managing files in a collaboration

First up, this chapter will discuss what version control is and how it fits into the reproducible workflow of an effective researcher in the physical sciences.