book

How We Test Software at Microsoft®

by Alan Page, Ken Johnston, and Bj Rollison

December 2008

Beginner to intermediate

448 pages

13h 3m

English

Microsoft Press

Read now

Unlock full access

Who This Book Is ForWhat This Book Is AboutFind Additional Content OnlineSupport for This Book
The Microsoft Vision, Values and Why We “Love This Company!”Microsoft Is a Big Software Engineering CompanyDeveloping Big and Efficient BusinessesThe Shared Team ModelWorking Small in a Big CompanyEmploying Many Types of EngineersThe Engineering DisciplinesBeing a Global Software Development CompanySummary
What’s in a Name?Testers at Microsoft Have Not Always Been SDETsI Need More Testers and I Need Them Now!Campus RecruitingIndustry RecruitingLearning How to Be a Microsoft SDETThe Engineering Career at MicrosoftCareer Paths in the Test DisciplineThe Test ArchitectThe IC TesterBecoming a Manager is Not a PromotionTest ManagersSummary
Software Engineering at MicrosoftTraditional Software Engineering ModelsWaterfall ModelSpiral ModelAgile MethodologiesOther ModelsMilestonesAgile at MicrosoftFeature CrewsGetting to DoneIterations and MilestonesPutting It All TogetherProcess ImprovementFormal Process Improvement Systems at MicrosoftMicrosoft and ISO 9000Shipping Software from the War RoomWar, What Is It Good For?Defining the Release—MicrospeakMandatory PracticesExpected vs. MandatoryOne-Stop ShoppingSummary: Completing the Meal
Practicing Good Software Design and Test DesignUsing Test PatternsEstimating Test TimeStarting with TestingAsk QuestionsHave a Test StrategyThinking About TestabilityTest Design SpecificationsExample Elements in a Test Design SpecificationTesting the Good and the BadOther Factors to Consider in Test Case DesignBlack Box, White Box, and Gray BoxExploratory Testing at MicrosoftSummary
The Need for Functional TestingEquivalence Class PartitioningDecomposing Variable DataEquivalence Class Partitioning in ActionAnalyzing Parameter SubsetsThe ECP TestsSummary of Equivalence Class PartitioningBoundary Value AnalysisDefining Boundary TestsA New Formula for Boundary Value AnalysisHidden BoundariesSummary of Boundary Value AnalysisCombinatorial AnalysisCombinatorial Testing ApproachesCombinatorial Analysis in PracticeEffectiveness of Combinatorial AnalysisSummary of Combinatorial AnalysisSummary
Block TestingSummary of Block TestingDecision TestingSummary of Decision TestingCondition TestingSummary of Condition TestingBasis Path TestingSummary of Basis Path TestingSummary

Risky BusinessA Complex ProblemCounting Lines of CodeMeasuring Cyclomatic ComplexityHalstead MetricsObject-Oriented MetricsHigh Cyclomatic Complexity Doesn’t Necessarily Mean “Buggy”What to Do with Complexity MetricsSummary
Modeling BasicsTesting with ModelsDesigning a ModelModeling SoftwareBuilding a Finite State ModelAutomating ModelsGraph Theory and MBTModels for API TestingRandom ModelsGrammar ModelsModeling Without TestingBayesian Graphical ModelingPetri NetsModel-Based Testing Tools at MicrosoftSpec ExplorerModeling with Spec ExplorerSpec Explorer for Visual StudioA Language and EngineModeling TipsSummaryRecommended Reading and Tools
The Bug WorkflowBug TrackingA Bug’s LifeAttributes of a Bug Tracking SystemWhy Write a Bug Report?Anatomy of a Bug ReportBug TriageCommon Mistakes in Bug ReportsUsing the DataHow Not to Use the Data: Bugs as Performance MetricsBug BarsTest Case ManagementWhat Is a Test Case?The Value of a Test CaseAnatomy of a Test CaseTest Case MistakesManaging Test CasesCases and Points: Counting Test CasesTracking and Interpreting the Test ResultsSummary
The Value of AutomationTo Automate or Not to Automate, That Is the QuestionUser Interface AutomationWhat’s in a Test?SEARCH at MicrosoftSetupExecutionAnalysisReportingCleanupHelpRun, Automation, Run!Putting It All TogetherLarge-Scale Test AutomationCommon Automation MistakesSummary
Beyond FunctionalityTesting the “ilities”Performance TestingHow Do You Measure Performance?Stress TestingDistributed Stress TestingDistributed Stress ArchitectureThe Stress ClientThe Stress ServerAttributes of Multiclient Stress TestsCompatibility TestingApplication LibrariesApplication VerifierEating Our DogfoodAccessibility TestingAccessibility PersonasTesting for AccessibilityTesting Tools for Microsoft Active AccessibilityUsability TestingUsability LabsSecurity TestingThreat ModelingFuzz TestingSummary
Code ChurnKeeping It Under ControlTracking ChangesWhat Changed?Why Did It Change?A Home for Source ControlBuild ItThe Daily BuildBreaking the BuildStopping the BreaksStatic AnalysisNative Code AnalysisManaged Code AnalysisJust Another ToolTest Code AnalysisTest Code Is Product CodeEven More ToolsTools for Unique ProblemsTools for EveryoneSummary
Testing and QualityTesting Provides InformationQuality PerceptionCustomers to the RescueGames, Too!Windows Error ReportingThe Way We WERCorporate Error ReportingFilling the BucketsEmptying the BucketsTest and WERSmile and Microsoft Smiles with YouSend a Smile ImpactConnecting with CustomersSummary
Two Parts: About Services and Test TechniquesPart 1: About ServicesThe Microsoft Services StrategyShifting to Internet Services as the FocusGrowing from Large Scale to Mega ScalePower Is the Bottleneck to GrowthServices vs. Packaged ProductS+S ExampleMoving from Stand-Alone to Layered ServicesPart 2 Testing Software Plus ServicesWaves of InnovationDesigning the Right S+S and Services Test ApproachClient SupportBuilt on a ServerPlatform Services vs. Top-Level ServicesLoosely Coupled vs. Tightly Coupled ServicesStateless to StatefulStateless ServicesStateful ServicesTime to Market or Features and QualityRelease Frequency and NamingTesting Techniques for S+SFully Automated DeploymentsTest EnvironmentsThe One BoxThe Test ClusterThe Perf and Scale ClusterThe Integrated Services Test EnvironmentThe Deployment Test ClusterTest Environments SummaryIntegration Testing vs. Test Flags and emulationTest Flags and Emulation Provide AgilityTest FlagsEmulationTesting Against ProductionProduction Dogfood, Now with More Big IronProduction Data, so Tempting but Risky TooPerformance Test Metrics for ServicesPage Load Time 1 and Page Load Time 2Page WeightCompressibilityExpiration DateRound Trip AnalysisSeveral Other Critical Thoughts on S+SContinuous Quality Improvement ProgramCommon Bugs I’ve Seen MissedSummary
Automatic Failure AnalysisOvercoming Analysis ParalysisThe Match GameGood Logging PracticesAnatomy of a Log FileIntegrating AFAMachine VirtualizationVirtualization BenefitsIn the OfficeTest Lab SavingsTest Machine Configuration SavingsVirtual Machine Test ScenariosDaily Build TestingNetwork Topology TestingWhen a Failure Occurs During TestingExport and ImportSnapshotsTest Scenarios Not RecommendedCode Reviews and InspectionsTypes of Code ReviewsFormal ReviewsInformal ReviewsChecklistsOther ConsiderationsActivitiesTaking ActionTime Is on My SideMore Review CollateralTwo Faces of ReviewTools, Tools, EverywhereReduce, Reuse, RecycleWhat’s the Problem?Open DevelopmentSummary
The Need for Forward ThinkingThinking Forward by Moving BackwardStriving for a Quality CultureTesting and Quality AssuranceWho Owns Quality?The Cost of QualityA New Role for TestTest LeadershipThe Microsoft Test Leadership TeamTest Leadership Team ChairTest Leadership in ActionThe Test Architect GroupTest ExcellenceSharingThe Engineering Excellence ForumHelpingCommunicatingKeeping an Eye on the FutureMicrosoft Director of Test ExcellenceThe Leadership TriadInnovating for the Future