book

Software Engineering at Google

by Titus Winters, Tom Manshreck, Hyrum Wright

March 2020

Intermediate to advanced

602 pages

18h 36m

English

O'Reilly Media, Inc.

Audiobook available

Read now

Unlock full access

Includes

Includes Quizzes

Programming Over TimeGoogle’s PerspectiveWhat This Book Isn’tParting RemarksConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Time and ChangeHyrum’s LawExample: Hash OrderingWhy Not Just Aim for “Nothing Changes”?Scale and EfficiencyPolicies That Don’t ScalePolicies That Scale WellExample: Compiler UpgradeShifting LeftTrade-offs and CostsExample: MarkersInputs to Decision MakingExample: Distributed BuildsExample: Deciding Between Time and ScaleRevisiting Decisions, Making MistakesSoftware Engineering Versus ProgrammingConclusionTL;DRs
Help Me Hide My CodeThe Genius MythHiding Considered HarmfulEarly DetectionThe Bus FactorPace of ProgressIn Short, Don’t HideIt’s All About the TeamThe Three Pillars of Social InteractionWhy Do These Pillars Matter?Humility, Respect, and Trust in PracticeBlameless Post-Mortem CultureBeing GoogleyConclusionTL;DRs
Challenges to LearningPhilosophySetting the Stage: Psychological SafetyMentorshipPsychological Safety in Large GroupsGrowing Your KnowledgeAsk QuestionsUnderstand ContextScaling Your Questions: Ask the CommunityGroup ChatsMailing ListsYAQS: Question-and-Answer PlatformScaling Your Knowledge: You Always Have Something to TeachOffice HoursTech Talks and ClassesDocumentationCodeScaling Your Organization’s KnowledgeCultivating a Knowledge-Sharing CultureEstablishing Canonical Sources of InformationStaying in the LoopReadability: Standardized Mentorship Through Code ReviewWhat Is the Readability Process?Why Have This Process?ConclusionTL;DRs
Bias Is the DefaultUnderstanding the Need for DiversityBuilding Multicultural CapacityMaking Diversity ActionableReject Singular ApproachesChallenge Established ProcessesValues Versus OutcomesStay Curious, Push ForwardConclusionTL;DRs
Managers and Tech Leads (and Both)The Engineering ManagerThe Tech LeadThe Tech Lead ManagerMoving from an Individual Contributor Role to a Leadership RoleThe Only Thing to Fear Is…Well, EverythingServant LeadershipThe Engineering ManagerManager Is a Four-Letter WordToday’s Engineering ManagerAntipatternsAntipattern: Hire PushoversAntipattern: Ignore Low PerformersAntipattern: Ignore Human IssuesAntipattern: Be Everyone’s FriendAntipattern: Compromise the Hiring BarAntipattern: Treat Your Team Like ChildrenPositive PatternsLose the EgoBe a Zen MasterBe a CatalystRemove RoadblocksBe a Teacher and a MentorSet Clear GoalsBe HonestTrack HappinessThe Unexpected QuestionOther Tips and TricksPeople Are Like PlantsIntrinsic Versus Extrinsic MotivationConclusionTL;DRs
Always Be DecidingThe Parable of the AirplaneIdentify the BlindersIdentify the Key Trade-OffsDecide, Then IterateAlways Be LeavingYour Mission: Build a “Self-Driving” TeamDividing the Problem SpaceAlways Be ScalingThe Cycle of SuccessImportant Versus UrgentLearn to Drop BallsProtecting Your EnergyConclusionTL;DRs

Why Should We Measure Engineering Productivity?Triage: Is It Even Worth Measuring?Selecting Meaningful Metrics with Goals and SignalsGoalsSignalsMetricsUsing Data to Validate MetricsTaking Action and Tracking ResultsConclusionTL;DRs
Why Have Rules?Creating the RulesGuiding PrinciplesThe Style GuideChanging the RulesThe ProcessThe Style ArbitersExceptionsGuidanceApplying the RulesError CheckersCode FormattersConclusionTL;DRs
Code Review FlowHow Code Review Works at GoogleCode Review BenefitsCode CorrectnessComprehension of CodeCode ConsistencyPsychological and Cultural BenefitsKnowledge SharingCode Review Best PracticesBe Polite and ProfessionalWrite Small ChangesWrite Good Change DescriptionsKeep Reviewers to a MinimumAutomate Where PossibleTypes of Code ReviewsGreenfield Code ReviewsBehavioral Changes, Improvements, and OptimizationsBug Fixes and RollbacksRefactorings and Large-Scale ChangesConclusionTL;DRs
What Qualifies as Documentation?Why Is Documentation Needed?Documentation Is Like CodeKnow Your AudienceTypes of AudiencesDocumentation TypesReference DocumentationDesign DocsTutorialsConceptual DocumentationLanding PagesDocumentation ReviewsDocumentation PhilosophyWHO, WHAT, WHEN, WHERE, and WHYThe Beginning, Middle, and EndThe Parameters of Good DocumentationDeprecating DocumentsWhen Do You Need Technical Writers?ConclusionTL;DRs
Why Do We Write Tests?The Story of Google Web ServerTesting at the Speed of Modern DevelopmentWrite, Run, ReactBenefits of Testing CodeDesigning a Test SuiteTest SizeTest ScopeThe Beyoncé RuleA Note on Code CoverageTesting at Google ScaleThe Pitfalls of a Large Test SuiteHistory of Testing at GoogleOrientation ClassesTest CertifiedTesting on the ToiletTesting Culture TodayThe Limits of Automated TestingConclusionTL;DRs
The Importance of MaintainabilityPreventing Brittle TestsStrive for Unchanging TestsTest via Public APIsTest State, Not InteractionsWriting Clear TestsMake Your Tests Complete and ConciseTest Behaviors, Not MethodsDon’t Put Logic in TestsWrite Clear Failure MessagesTests and Code Sharing: DAMP, Not DRYShared ValuesShared SetupShared Helpers and ValidationDefining Test InfrastructureConclusionTL;DRs
The Impact of Test Doubles on Software DevelopmentTest Doubles at GoogleBasic ConceptsAn Example Test DoubleSeamsMocking FrameworksTechniques for Using Test DoublesFakingStubbingInteraction TestingReal ImplementationsPrefer Realism Over IsolationHow to Decide When to Use a Real ImplementationFakingWhy Are Fakes Important?When Should Fakes Be Written?The Fidelity of FakesFakes Should Be TestedWhat to Do If a Fake Is Not AvailableStubbingThe Dangers of Overusing StubbingWhen Is Stubbing Appropriate?Interaction TestingPrefer State Testing Over Interaction TestingWhen Is Interaction Testing Appropriate?Best Practices for Interaction TestingConclusionTL;DRs
What Are Larger Tests?FidelityCommon Gaps in Unit TestsWhy Not Have Larger Tests?Larger Tests at GoogleLarger Tests and TimeLarger Tests at Google ScaleStructure of a Large TestThe System Under TestTest DataVerificationTypes of Larger TestsFunctional Testing of One or More Interacting BinariesBrowser and Device TestingPerformance, Load, and Stress testingDeployment Configuration TestingExploratory TestingA/B Diff Regression TestingUATProbers and Canary AnalysisDisaster Recovery and Chaos EngineeringUser EvaluationLarge Tests and the Developer WorkflowAuthoring Large TestsRunning Large TestsOwning Large TestsConclusionTL;DRs
Why Deprecate?Why Is Deprecation So Hard?Deprecation During DesignTypes of DeprecationAdvisory DeprecationCompulsory DeprecationDeprecation WarningsManaging the Deprecation ProcessProcess OwnersMilestonesDeprecation ToolingConclusionTL;DRs
What Is Version Control?Why Is Version Control Important?Centralized VCS Versus Distributed VCSSource of TruthVersion Control Versus Dependency ManagementBranch ManagementWork in Progress Is Akin to a BranchDev BranchesRelease BranchesVersion Control at GoogleOne VersionScenario: Multiple Available VersionsThe “One-Version” Rule(Nearly) No Long-Lived BranchesWhat About Release Branches?MonoreposFuture of Version ControlConclusionTL;DRs
The Code Search UIHow Do Googlers Use Code Search?Where?What?How?Why?Who and When?Why a Separate Web Tool?ScaleZero Setup Global Code ViewSpecializationIntegration with Other Developer ToolsAPI ExposureImpact of Scale on DesignSearch Query LatencyIndex LatencyGoogle’s ImplementationSearch IndexRankingSelected Trade-OffsCompleteness: Repository at HeadCompleteness: All Versus Most-Relevant ResultsCompleteness: Head Versus Branches Versus All History Versus WorkspacesExpressiveness: Token Versus Substring Versus RegexConclusionTL;DRs
Purpose of a Build SystemWhat Happens Without a Build System?But All I Need Is a Compiler!Shell Scripts to the Rescue?Modern Build SystemsIt’s All About DependenciesTask-Based Build SystemsArtifact-Based Build SystemsDistributed BuildsTime, Scale, Trade-OffsDealing with Modules and DependenciesUsing Fine-Grained Modules and the 1:1:1 RuleMinimizing Module VisibilityManaging DependenciesConclusionTL;DRs
Code Review Tooling PrinciplesCode Review FlowNotificationsStage 1: Create a ChangeDiffingAnalysis ResultsTight Tool IntegrationStage 2: Request ReviewStages 3 and 4: Understanding and Commenting on a ChangeCommentingUnderstanding the State of a ChangeStage 5: Change Approvals (Scoring a Change)Stage 6: Commiting a ChangeAfter Commit: Tracking HistoryConclusionTL;DRs
Characteristics of Effective Static AnalysisScalabilityUsabilityKey Lessons in Making Static Analysis WorkFocus on Developer HappinessMake Static Analysis a Part of the Core Developer WorkflowEmpower Users to ContributeTricorder: Google’s Static Analysis PlatformIntegrated ToolsIntegrated Feedback ChannelsSuggested FixesPer-Project CustomizationPresubmitsCompiler IntegrationAnalysis While Editing and Browsing CodeConclusionTL;DRs
Why Is Dependency Management So Difficult?Conflicting Requirements and Diamond DependenciesImporting DependenciesCompatibility PromisesConsiderations When ImportingHow Google Handles Importing DependenciesDependency Management, In TheoryNothing Changes (aka The Static Dependency Model)Semantic VersioningBundled Distribution ModelsLive at HeadThe Limitations of SemVerSemVer Might OverconstrainSemVer Might OverpromiseMotivationsMinimum Version SelectionSo, Does SemVer Work?Dependency Management with Infinite ResourcesExporting DependenciesConclusionTL;DRs
What Is a Large-Scale Change?Who Deals with LSCs?Barriers to Atomic ChangesTechnical LimitationsMerge ConflictsNo Haunted GraveyardsHeterogeneityTestingCode ReviewLSC InfrastructurePolicies and CultureCodebase InsightChange ManagementTestingLanguage SupportThe LSC ProcessAuthorizationChange CreationSharding and SubmittingCleanupConclusionTL;DRs
CI ConceptsFast Feedback LoopsAutomationContinuous TestingCI ChallengesHermetic TestingCI at GoogleCI Case Study: Google TakeoutBut I Can’t Afford CIConclusionTL;DRs
Idioms of Continuous Delivery at GoogleVelocity Is a Team Sport: How to Break Up a Deployment into Manageable PiecesEvaluating Changes in Isolation: Flag-Guarding FeaturesStriving for Agility: Setting Up a Release TrainNo Binary Is PerfectMeet Your Release DeadlineQuality and User-Focus: Ship Only What Gets UsedShifting Left: Making Data-Driven Decisions EarlierChanging Team Culture: Building Discipline into DeploymentConclusionTL;DRs
Taming the Compute EnvironmentAutomation of ToilContainerization and MultitenancySummaryWriting Software for Managed ComputeArchitecting for FailureBatch Versus ServingManaging StateConnecting to a ServiceOne-Off CodeCaaS Over Time and ScaleContainers as an AbstractionOne Service to Rule Them AllSubmitted ConfigurationChoosing a Compute ServiceCentralization Versus CustomizationLevel of Abstraction: ServerlessPublic Versus PrivateConclusionTL;DRs

Content preview from Software Engineering at Google

Afterword

Software engineering at Google has been an extraordinary experiment in how to develop and maintain a large and evolving codebase. I’ve seen engineering teams break ground on this front during my time here, moving Google forward both as a company that touches billions of users and as a leader in the tech industry. This wouldn’t have been possible without the principles outlined in this book, so I’m very excited to see these pages come to life.

If the past 50 years (or the preceding pages here) have proven anything, it’s that software engineering is far from stagnant. In an environment in which technology is steadily changing, the software engineering function holds a particularly important role within a given organization. Today, software engineering principles aren’t simply about how to effectively run an organization; they’re about how to be a more responsible company for users and the world at large.

Solutions to common software engineering problems are not always hidden in plain sight—most require a certain level of resolute agility to identify solutions that will work for current-day problems and also withstand inevitable changes to technical systems. This agility is a common quality of the software engineering teams I’ve had the privilege to work with and learn from since joining Google back in 2008.

The idea of sustainability is also central to software engineering. Over a codebase’s expected lifespan, we must be able to react and adapt to changes, be that in product ...