Software Engineering at Google

Software Engineering at Google

by Hyrum Wright, Tom Manshreck, Titus Winters
Released February 2020
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492082781

Explore a preview version of Software Engineering at Google right now.

O’Reilly members get unlimited access to live online training experiences, plus books, videos, and digital content from 200+ publishers.

Start your free trial

Book Description

How do you manage a living codebase that evolves and responds to changing requirements and demands over the length of its life? Based on their experience at Google, software engineers Titus Winters and Hyrum K. Wright, along with technical writer Tom Manshreck, present a candid and insightful look at how some of the world’s leading practitioners construct and maintain software.

At Google, software engineering represents roughly 80-90% of the work, while only 10-20% of the time involves original programming. Most teaching on this subject concentrates on programming, but little on software engineering. By emphasizing three fundamental principles that software organizations should keep in mind when designing, architecting, and writing code, this book explains:

  • Fundamental differences between software engineering and programming
  • The software engineering lifecycle from code development to testing to deprecation
  • How to effectively manage your codebase and efficiently respond to change
  • Why culture is important, and how processes, practices, and tools come into play
  • Tradeoffs: how an organization makes optimal software decisions by keeping time and scale in mind

Table of Contents

  1. Foreword
  2. Preface
    1. Programming Over Time
    2. Google’s Perspective
    3. What This Book Isn’t
    4. Parting Remarks
    5. Conventions Used in This Book
    6. Using Code Examples
    7. O’Reilly Online Learning
    8. How to Contact Us
    9. Acknowledgments
  3. I. Thesis
  4. 1. What Is Software Engineering?
    1. Time and Change
      1. Hyrum’s Law
      2. Example: Hash Ordering
      3. Why Not Just Aim for “Nothing Changes”?
    2. Scale and Efficiency
      1. Policies that Don’t Scale
      2. Policies That Scale Well
      3. Example: Compiler Upgrade
      4. Shifting Left
    3. Tradeoffs and Costs
      1. Example: Markers
      2. Inputs to Decision Making
      3. Example: Distributed Builds
      4. Example: Deciding Between Time and Scale
      5. Revisiting Decisions, Making Mistakes
    4. Software Engineering versus Programming
    5. Conclusion
    6. TL;DRs
  5. II. Culture
  6. 2. How to Work Well on Teams
    1. Help Me Hide My Code
    2. The Genius Myth
    3. Hiding Considered Harmful
      1. Early Detection
      2. The Bus Factor
      3. Pace of Progress
      4. Engineers and Offices: Controversial Opinions
      5. In Short, Don’t Hide
    4. It’s All About the Team
      1. The Three Pillars of Social Interaction
      2. Why Do These Pillars Matter?
      3. Humility, Respect, and Trust in Practice
      4. Blameless Postmortem Culture
      5. Being Googley
    5. Conclusion
    6. TL;DRs
  7. 3. Knowledge Sharing
    1. Challenges to Learning
    2. Philosophy
    3. Setting the Stage: Psychological Safety
      1. Mentorship
      2. Psychological Safety in Large Groups
    4. Growing Your Knowledge
      1. Ask Questions
      2. Understand Context
    5. Scaling Your Questions: Ask the Community
      1. Group Chats
      2. Mailing Lists
      3. YAQS: Question and Answer Platform
    6. Scaling Your Knowledge: You Always Have Something to Teach
      1. Office Hours
      2. Tech Talks and Classes
      3. Documentation
      4. Code
    7. Scaling Your Organization’s Knowledge
      1. Cultivating a Knowledge-Sharing Culture
      2. Establishing Canonical Sources of Information
      3. Staying in the Loop
    8. Readability: Standardized Mentorship Through Code Review
      1. What Is the Readability Process?
      2. Why Have This Process?
    9. Conclusion
    10. TL;DRs
  8. 4. Engineering for Equity
    1. Bias Is the Default
    2. Google Misses the Mark on Racial Inclusion
    3. Understanding the Need for Diversity
    4. Building Multicultural Capacity
    5. Making Diversity Actionable
    6. Reject Singular Approaches
    7. Challenge Established Processes
    8. Values versus Outcomes
    9. Stay Curious, Push Forward
    10. Conclusion
    11. TL;DRs
  9. 5. How to Lead a Team
    1. Managers and Tech Leads (and Both)
      1. The Engineering Manager
      2. The Tech Lead
      3. The Tech Lead Manager
      4. Case Study: Influencing Without Authority
    2. Moving from an Individual Contributor Role to a Leadership Role
      1. The Only Thing to Fear Is…Well, Everything
      2. Servant Leadership
    3. The Engineering Manager
      1. Manager Is a Four-Letter Word
      2. Today’s Engineering Manager
      3. Failure Is an Option
    4. Antipatterns
      1. Antipattern: Hire Pushovers
      2. Antipattern: Ignore Low Performers
      3. Antipattern: Ignore Human Issues
      4. Antipattern: Be Everyone’s Friend
      5. Antipattern: Compromise the Hiring Bar
      6. Antipattern: Treat Your Team Like Children
    5. Positive Patterns
      1. Lose the Ego
      2. Be a Zen Master
      3. Be a Catalyst
      4. Remove Roadblocks
      5. Be a Teacher and a Mentor
      6. Set Clear Goals
      7. Be Honest
      8. Track Happiness
    6. The Unexpected Question
    7. Other Tips and Tricks
    8. People Are Like Plants
      1. Intrinsic versus Extrinsic Motivation
    9. Conclusion
    10. TL;DRs
  10. 6. Leading at Scale
    1. Always Be Deciding
      1. The Parable of the Airplane
    2. Identify the Blinders
    3. Identify the Key Trade-Offs
    4. Decide, Then Iterate
    5. Case Study: Addressing the “Latency” of Web Search
    6. Always Be Leaving
    7. Your Mission: Build a “Self-Driving” Team
      1. Dividing the problem space
    8. Always Be Scaling
    9. The Cycle of Success
    10. Important versus Urgent
    11. Learn to Drop Balls
    12. Protecting Your Energy
    13. Conclusion
    14. TL;DRs
  11. 7. Measuring Engineering Productivity
    1. Why Should We Measure Engineering Productivity?
    2. Triage: Is It Even Worth Measuring?
    3. Selecting Meaningful Metrics with Goals and Signals
    4. Goals
    5. Signals
    6. Metrics
    7. Using Data to Validate Metrics
    8. Taking Action and Tracking Results
    9. Conclusion
    10. TL;DRs
  12. III. Processes
  13. 8. Style Guides and Rules
    1. Why Have Rules?
    2. Creating the Rules
      1. Guiding Principles
      2. The Style Guide
    3. Changing the Rules
      1. The Process
      2. The Style Arbiters
      3. Exceptions
    4. Guidance
    5. Applying the Rules
      1. Error Checkers
      2. Code Formatters
      3. Case Study: gofmt
    6. Conclusion
    7. TL;DRs
  14. 9. Code Review
    1. Code Review Flow
    2. Code Is a Liability
      1. How Code Review Works at Google
    3. Ownership
    4. Code Review Benefits
      1. Code Correctness
      2. Comprehension of Code
      3. Code Consistency
      4. Psychological and Cultural Benefits
      5. Knowledge Sharing
    5. Code Review Best Practices
      1. Be Polite and Professional
      2. Write Small Changes
      3. Write Good Change Descriptions
      4. Keep Reviewers to a Minimum
      5. Automate Where Possible
    6. Types of Code Reviews
      1. Greenfield Code Reviews
      2. Behavioral Changes, Improvements, and Optimizations
      3. Bug Fixes and Rollbacks
      4. Refactorings and Large-Scale Changes
    7. Conclusion
    8. TL;DRs
  15. 10. Documentation
    1. What Qualifies as Documentation?
    2. Why Is Documentation Needed?
    3. Documentation Is Like Code
      1. Callout: The Google Wiki
    4. Know Your Audience
      1. Types of Audiences
    5. Documentation Types
      1. Reference Documentation
      2. Design Docs
      3. Tutorials
      4. Conceptual Documentation
      5. Landing Pages
    6. Documentation Reviews
      1. Callout: The Developer Guide Library
    7. Documentation Philosophy
      1. WHO, WHAT, WHEN, WHERE, and WHY
      2. The Beginning, Middle, and End
      3. The Parameters of Good Documentation
      4. Deprecating Documents
    8. When Do You Need Technical Writers?
    9. Conclusion
    10. TL;DRs
  16. 11. Testing Overview
    1. Why Do We Write Tests?
      1. The Story of Google Web Server
      2. Testing at the Speed of Modern Development
      3. Write, Run, React
      4. Benefits of Testing Code
    2. Designing a Test Suite
      1. Test Size
      2. Test Scope
      3. The Beyoncé Rule
      4. A Note on Code Coverage
    3. Testing at Google Scale
      1. The Pitfalls of a Large Test Suite
    4. History of Testing at Google
      1. Orientation Classes
      2. Test Certified
      3. Testing on the Toilet
      4. Testing Culture Today
    5. The Limits of Automated Testing
    6. Conclusion
    7. TL;DRs
  17. 12. Unit Testing
    1. The Importance of Maintainability
    2. Preventing Brittle Tests
      1. Strive for Unchanging Tests
      2. Test via Public APIs
      3. Test State, Not Interactions
    3. Writing Clear Tests
      1. Make Your Tests Complete and Concise
      2. Test Behaviors, Not Methods
      3. Don’t Put Logic in Tests
      4. Write Clear Failure Messages
    4. Tests and Code Sharing: DAMP, Not DRY
      1. Shared Values
      2. Shared Setup
      3. Shared Helpers and Validation
      4. Defining Test Infrastructure
    5. Conclusion
    6. TL;DRs
  18. 13. Test Doubles
    1. The Impact of Test Doubles on Software Development
    2. Test Doubles at Google
    3. Basic Concepts
      1. An Example Test Double
      2. Seams
      3. Mocking Frameworks
    4. Techniques for Using Test Doubles
      1. Faking
      2. Stubbing
      3. Interaction Testing
    5. Real Implementations
      1. Prefer Realism Over Isolation
      2. How to Decide When to Use a Real Implementation
    6. Faking
      1. Why Are Fakes Important?
      2. When Should Fakes Be Written?
      3. The Fidelity of Fakes
      4. Fakes Should Be Tested
      5. What to Do If a Fake Is Not Available
    7. Stubbing
      1. The Dangers of Overusing Stubbing
      2. When Is Stubbing Appropriate?
    8. Interaction Testing
      1. Prefer State Testing Over Interaction Testing
      2. When Is Interaction Testing Appropriate?
      3. Best Practices for Interaction Testing
    9. Conclusion
    10. TL;DRs
  19. 14. Larger Testing
    1. What are Larger Tests?
      1. Fidelity
      2. Common Gaps in Unit Tests
      3. Why Not Have Larger Tests?
    2. Larger Tests at Google
      1. Larger Tests and Time
      2. Larger Tests at Google Scale
    3. Structure of a Large Test
      1. The System Under Test
      2. Test Data
      3. Verification
    4. Types of Larger Tests
      1. Functional Testing of One or More Interacting Binaries
      2. Browser and Device Testing
      3. Performance, Load, and Stress testing
      4. Deployment Configuration Testing
      5. Exploratory Testing
      6. A/B Diff Regression Testing
      7. UAT
      8. Probers and Canary Analysis
      9. Disaster Recovery and Chaos Engineering
      10. User Evaluation
    5. Large Tests and the Developer Workflow
      1. Authoring Large Tests
      2. Running Large Tests
      3. Owning Large Tests
    6. Conclusion
    7. TL;DRs
  20. 15. Deprecation
    1. Why Deprecate?
    2. Why Is Deprecation So Hard?
      1. Deprecation During Design
    3. Types of Deprecation
      1. Advisory Deprecation
      2. Compulsory Deprecation
      3. Deprecation Warnings
    4. Managing the Deprecation Process
      1. Process Owners
      2. Milestones
      3. Deprecation Tooling
    5. Conclusion
    6. TL;DRs
  21. IV. Tools
  22. 16. Version Control and Branch Management
    1. What Is Version Control?
      1. Why Is Version Control Important?
      2. Centralized VCS versus Distributed VCS
      3. Source of Truth
      4. Version Control versus Dependency Management
    2. Branch Management
      1. Work in Progress Is Akin to a Branch
      2. Dev Branches
      3. Release Branches
    3. Version Control at Google
      1. One Version
      2. Scenario: Multiple Available Versions
      3. The “One Version” Rule
      4. (Nearly) No Long-Lived Branches
      5. What About Release Branches?
    4. Monorepos
    5. Future of Version Control
    6. Conclusion
    7. TL;DRs
  23. 17. Code Search
    1. The Code Search UI
    2. How Do Googlers Use Code Search?
      1. Where?
      2. What?
      3. How?
      4. Why?
      5. Who and When?
    3. Why a Separate Web Tool?
      1. Scale
      2. Zero Setup Global Code View
      3. Specialization
      4. Integration with Other Developer Tools
      5. API Exposure
    4. Impact of Scale on Design
      1. Search Query Latency
      2. Index Latency
    5. Google’s Implementation
      1. Search Index
      2. Ranking
    6. Selected Trade-Offs
      1. Completeness: Repository at Head
      2. Completeness: All versus Most-Relevant Results
      3. Completeness: Head versus Branches versus All History versus Workspaces
      4. Expressiveness: Token versus Substring versus Regex
    7. Conclusion
    8. TL;DRs
  24. 18. Build Systems and Build Philosophy
    1. Purpose of a Build System
    2. What Happens Without a Build System?
      1. But All I Need Is a Compiler!
      2. Shell Scripts to the Rescue?
    3. Modern Build Systems
      1. It’s All About Dependencies
      2. Task-Based Build Systems
      3. Artifact-Based Build Systems
      4. Distributed Builds
      5. Time, Scale, Trade-Offs
    4. Dealing with Modules and Dependencies
      1. Using Fine-Grained Modules and the 1:1:1 Rule
      2. Minimizing Module Visibility
      3. Managing Dependencies
    5. Conclusion
    6. TL;DRs
  25. 19. Critique: Google’s Code Review Tool
    1. Code Review Tooling Principles
    2. Code Review Flow
      1. Notifications
    3. Stage 1: Creating a Change
      1. Diffing
      2. Analysis Results
      3. Tight Tool Integration
    4. Stage 2: Request Review
    5. Stage 3 and 4: Understanding and Commenting on a Change
      1. Commenting
      2. Understanding the State of a Change
    6. Stage 5: Change Approvals (Scoring a Change)
    7. Stage 6: Commiting a Change
      1. After Commit: Tracking History
    8. Gerrit
    9. Conclusion
    10. TL;DRs
  26. 20. Static Analysis
    1. Characteristics of Effective Static Analysis
      1. Scalability
      2. Usability
    2. Key Lessons in Making Static Analysis Work
      1. Focus on Developer Happiness
      2. Make Static Analysis a Part of the Core Developer Workflow
      3. Empower Users to Contribute
    3. Tricorder: Google’s Static Analysis Platform
      1. Integrated Tools
      2. Integrated Feedback Channels
      3. Suggested Fixes
      4. Per-Project Customization
      5. Presubmits
      6. Compiler Integration
      7. Analysis While Editing and Browsing Code
    4. Conclusion
    5. TL;DRs
  27. 21. Dependency Management
    1. Why Is Dependency Management So Difficult?
      1. Conflicting Requirements and Diamond Dependencies
    2. Importing Dependencies
      1. Compatibility Promises
      2. Considerations When Importing
      3. How Google Handles Importing Dependendencies
    3. Dependency Management, In Theory
      1. Nothing Changes (aka The Static Dependency Model)
      2. Semantic Versioning
      3. Bundled Distribution Models
      4. Live at Head
    4. The Limitations of SemVer
      1. SemVer Might Overconstrain
      2. SemVer Might Overpromise
      3. Motivations
      4. Minimum Version Selection
      5. So, Does SemVer Work?
    5. Dependency Management with Infinite Resources
      1. Exporting Dependencies
    6. Conclusion
      1. TL;DRs
  28. 22. Large-Scale Changes
    1. What Is a Large-Scale Change?
    2. Who Deals with LSCs?
      1. Filling Potholes
    3. Barriers to Atomic Changes
      1. Technical Limitations
      2. Merge Conflicts
      3. No Haunted Graveyards
      4. Heterogeneity
      5. Testing
    4. Testing LSCs
      1. Riding the TAP Train
      2. Code Review
      3. scoped_ptr to std::unique_ptr
    5. LSC Infrastructure
      1. Policies and Culture
      2. Codebase Insight
      3. Change Management
      4. Testing
      5. Language Support
      6. Operation RoseHub
    6. The LSC Process
      1. Authorization
      2. Change Creation
      3. Sharding and Submitting
      4. Cleanup
    7. Conclusion
    8. TL;DRs
  29. 23. Continuous Integration
    1. CI Concepts
      1. Fast Feedback Loops
      2. Automation
      3. Continuous Testing
    2. CI Is Alerting
      1. CI Challenges
      2. Hermetic Testing
    3. CI at Google
      1. TAP: Google’s Global Continuous Build
      2. CI Case Study: Google Takeout
      3. But I Can’t Afford CI
    4. Conclusion
    5. TL;DRs
  30. 24. Continuous Delivery
    1. Idioms of Continuous Delivery at Google
    2. Velocity Is a Team Sport: How to Break Up a Deployment into Manageable Pieces
    3. Evaluating Changes in Isolation: Flag-Guarding Features
    4. Striving for Agility: Setting Up a Release Train
      1. No Binary Is Perfect
      2. Meet Your Release Deadline
    5. Quality and User-Focus: Ship Only What Gets Used
    6. Shifting Left: Making Data-Driven Decisions Earlier
    7. Changing Team Culture: Building Discipline into Deployment
    8. Conclusion
    9. TL;DRs
  31. 25. Compute as a Service
    1. Taming the Compute Environment
      1. Automation of Toil
      2. Containerization and Multitenancy
      3. Summary
    2. Writing Software for Managed Compute
      1. Architecting for Failure
      2. Batch versus Serving
      3. Managing State
      4. Connecting to a Service
      5. One-Off Code
    3. CaaS over Time and Scale
      1. Containers as an abstraction
      2. One Service to Rule Them All
      3. Submitted Configuration
    4. Choosing a Compute Service
      1. Centralization versus Customization
      2. Level of Abstraction: Serverless
      3. Public versus Private
    5. Conclusion
    6. TL;DRs
  32. V. Conclusion
  33. 26. Conclusion

Product Information

  • Title: Software Engineering at Google
  • Author(s): Hyrum Wright, Tom Manshreck, Titus Winters
  • Release date: February 2020
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492082781