book

Software Testing and Continuous Quality Improvement, 3rd Edition

Name: Software Testing and Continuous Quality Improvement, 3rd Edition
Author: William E. Lewis
ISBN: 9781351722209

by William E. Lewis

June 2017

Intermediate to advanced

688 pages

16h 8m

English

Auerbach Publications

Read now

Unlock full access

Cover
Half Title
Title Page
Copyright Page
Table of Contents
Acknowledgments
Introduction
About the Author
SECTION 1 SOFTWARE QUALITY IN PERSPECTIVE
1 A Brief History of Software Testing
Historical Software Testing and Development ParallelsExtreme ProgrammingEvolution of Automated Testing ToolsStatic Capture/Replay Tools (without Scripting Language)Static Capture/Replay Tools (with Scripting Language)Variable Capture/Replay Tools

2 Quality Assurance Framework
What Is Quality?Prevention versus DetectionVerification versus ValidationSoftware Quality AssuranceComponents of Quality AssuranceSoftware TestingQuality ControlSoftware Configuration ManagementElements of Software Configuration ManagementSoftware Quality Assurance PlanSteps to Develop and Implement a Software Quality Assurance PlanStep 1: Document the PlanStep 2: Obtain Management AcceptanceStep 3: Obtain Development AcceptanceStep 4: Plan for Implementation of the SQA PlanStep 5: Execute the SQA PlanQuality StandardsSarbanes–OxleyISO9000Capability Maturity Model (CMM)Level 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizedPeople CMMCMMIMalcolm Baldrige National Quality AwardNotes
3 Overview of Testing Techniques
Black-Box Testing (Functional)White-Box Testing (Structural)Gray-Box Testing (Functional and Structural)Manual versus Automated TestingStatic versus Dynamic TestingTaxonomy of Software Testing Techniques
4 Transforming Requirements to Testable Test Cases
IntroductionSoftware Requirements as the Basis of TestingRequirement Quality FactorsUnderstandableNecessaryModifiableNonredundantTerseTestableTraceableWithin ScopeNumerical Method for Evaluating Requirement QualityProcess for Creating Test Cases from Good RequirementsStep 1: Review the RequirementsStep 2: Write a Test PlanStep 3: Identify the Test SuiteStep 4: Name the Test CasesStep 5: Write Test Case Descriptions and ObjectivesStep 6: Create the Test CasesStep 7: Review the Test CasesTransforming Use Cases to Test CasesStep 1: Draw a Use Case DiagramStep 2: Write the Detailed Use Case TextStep 3: Identify Use Case ScenariosStep 4: Generating the Test CasesStep 5: Generating Test DataSummaryWhat to Do When Requirements Are Nonexistent or Poor?Ad Hoc TestingThe Art of Ad Hoc TestingAdvantages and Disadvantages of Ad Hoc TestingExploratory TestingThe Art of Exploratory TestingExploratory Testing ProcessAdvantages and Disadvantages of Exploratory Testing
5 Quality through Continuous Improvement Process
Contribution of Edward DemingRole of Statistical MethodsCause-and-Effect DiagramFlowchartPareto ChartRun ChartHistogramScatter DiagramControl ChartDeming’s 14 Quality PrinciplesPoint 1: Create Constancy of PurposePoint 2: Adopt the New PhilosophyPoint 3: Cease Dependence on Mass InspectionPoint 4: End the Practice of Awarding Business on Price Tag AlonePoint 5: Improve Constantly and Ceaselessly the System of Production and ServicePoint 6: Institute Training and RetrainingPoint 7: Institute LeadershipPoint 8: Drive Out FearPoint 9: Break Down Barriers between Staff AreasPoint 10: Eliminate Slogans, Exhortations, and Targets for the WorkforcePoint 11: Eliminate Numerical GoalsPoint 12: Remove Barriers to Pride of WorkmanshipPoint 13: Institute a Vigorous Program of Education and RetrainingPoint 14: Take Action to Accomplish the TransformationContinuous Improvement through the Plan, Do, Check, Act ProcessGoing around the PDCA Circle
SECTION 2 WATERFALL TESTING REVIEW
6 Overview
Waterfall Development MethodologyContinuous Improvement “Phased” ApproachPsychology of Life-Cycle TestingSoftware Testing as a Continuous Improvement ProcessThe Testing Bible: Software Test PlanMajor Steps in Developing a Test PlanStep 1: Define the Test ObjectivesStep 2: Develop the Test ApproachStep 3: Define the Test EnvironmentStep 4: Develop the Test SpecificationsStep 5: Schedule the TestStep 6: Review and Approve the Test PlanComponents of a Test PlanTechnical Reviews as a Continuous Improvement ProcessMotivation for Technical ReviewsTypes of ReviewsStructured WalkthroughsInspectionsParticipant RolesSteps for an Effective ReviewStep 1: Plan for the Review ProcessStep 2: Schedule the ReviewStep 3: Develop the Review AgendaStep 4: Create a Review Report
7 Static Testing the Requirements
Testing the Requirements with Ambiguity ReviewsTesting the Requirements with Technical ReviewsInspections and WalkthroughsChecklistsMethodology ChecklistRequirements Traceability MatrixBuilding the System/Acceptance Test Plan
8 Static Testing the Logical Design
Data Model, Process Model, and the LinkageTesting the Logical Design with Technical ReviewsRefining the System/Acceptance Test Plan
9 Static Testing the Physical Design
Testing the Physical Design with Technical ReviewsCreating Integration Test CasesMethodology for Integration TestingStep 1: Identify Unit InterfacesStep 2: Reconcile Interfaces for CompletenessStep 3: Create Integration Test ConditionsStep 4: Evaluate the Completeness of Integration Test Conditions
10 Static Testing the Program Unit Design
Testing the Program Unit Design with Technical ReviewsSequenceSelectionIterationCreating Unit Test Cases
11 Static Testing and Dynamic Testing the Code
Testing Coding with Technical ReviewsExecuting the Test PlanUnit TestingIntegration TestingSystem TestingAcceptance TestingDefect Recording
SECTION 3 SPIRAL (AGILE) SOFTWARE TESTING METHODOLOGY: PLAN, DO, CHECK, ACT
12 Development Methodology Overview
Limitations of Life-Cycle DevelopmentThe Client/Server ChallengePsychology of Client/Server Spiral TestingThe New School of ThoughtTester/Developer PerceptionsProject Goal: Integrate QA and DevelopmentIterative/Spiral Development MethodologyRole of JADsRole of PrototypingMethodology for Developing PrototypesStep 1: Develop the PrototypeStep 2: Demonstrate Prototypes to ManagementStep 3: Demonstrate Prototype to UsersStep 4: Revise and Finalize SpecificationsStep 5: Develop the Production SystemContinuous Improvement “Spiral” Testing Approach
13 Information Gathering (Plan)
Step 1: Prepare for the InterviewTask 1: Identify the ParticipantsTask 2: Define the AgendaStep 2: Conduct the InterviewTask 1: Understand the ProjectTask 2: Understand the Project ObjectivesTask 3: Understand the Project StatusTask 4: Understand the Project PlansTask 5: Understand the Project Development MethodologyTask 6: Identify the High-Level Business RequirementsTask 7: Perform Risk AnalysisComputer Risk AnalysisMethod 1: Judgment and InstinctMethod 2: Dollar EstimationMethod 3: Identifying and Weighting Risk AttributesStep 3: Summarize the FindingsTask 1: Summarize the InterviewTask 2: Confirm the Interview Findings
14 Test Planning (Plan)
Step 1: Build a Test PlanTask 1: Prepare an IntroductionTask 2: Define the High-Level Functional Requirements (in Scope)Task 3: Identify Manual/Automated Test TypesTask 4: Identify the Test Exit CriteriaTask 5: Establish Regression Test StrategyTask 6: Define the Test DeliverablesTask 7: Organize the Test TeamTask 8: Establish a Test EnvironmentTask 9: Define the DependenciesTask 10: Create a Test ScheduleTask 11: Select the Test ToolsTask 12: Establish Defect Recording/Tracking ProceduresTask 13: Establish Change Request ProceduresTask 14: Establish Version Control ProceduresTask 15: Define Configuration Build ProceduresTask 16: Define Project Issue Resolution ProceduresTask 17: Establish Reporting ProceduresTask 18: Define Approval ProceduresStep 2: Define the Metric ObjectivesTask 1: Define the MetricsTask 2: Define the Metric PointsStep 3: Review/Approve the PlanTask 1: Schedule/Conduct the ReviewTask 2: Obtain Approvals
15 Test Case Design (Do)
Step 1: Design Function TestsTask 1: Refine the Functional Test RequirementsTask 2: Build a Function/Test MatrixStep 2: Design GUI TestsTen Guidelines for Good GUI DesignTask 1: Identify the Application GUI ComponentsTask 2: Define the GUI TestsStep 3: Define the System/Acceptance TestsTask 1: Identify Potential System TestsTask 2: Design System Fragment TestsTask 3: Identify Potential Acceptance TestsStep 4: Review/Approve DesignTask 1: Schedule/Prepare for ReviewTask 2: Obtain Approvals
16 Test Development (Do)
Step 1: Develop Test ScriptsTask 1: Script the Manual/Automated GUI/Function TestsTask 2: Script the Manual/Automated System Fragment TestsStep 2: Review/Approve Test DevelopmentTask 1: Schedule/Prepare for ReviewTask 2: Obtain Approvals
17 Test Coverage through Traceability
Use Cases and TraceabilitySummary
18 Test Execution/Evaluation (Do/Check)
Step 1: Setup and TestingTask 1: Regression Test the Manual/Automated Spiral FixesTask 2: Execute the Manual/Automated New Spiral TestsTask 3: Document the Spiral Test DefectsStep 2: EvaluationTask 1: Analyze the MetricsStep 3: Publish Interim ReportTask 1: Refine the Test ScheduleTask 2: Identify Requirement Changes
19 Prepare for the Next Spiral (Act)
Step 1: Refine the TestsTask 1: Update the Function/GUI TestsTask 2: Update the System Fragment TestsTask 3: Update the Acceptance TestsStep 2: Reassess the Team, Procedures, and Test EnvironmentTask 1: Evaluate the Test TeamTask 2: Review the Test Control ProceduresTask 3: Update the Test EnvironmentStep 3: Publish Interim Test ReportTask 1: Publish the Metric GraphicsTest Case Execution StatusDefect Gap AnalysisDefect Severity StatusTest Burnout Tracking
20 Conduct the System Test (Act)
Step 1: Complete System Test PlanTask 1: Finalize the System Test TypesTask 2: Finalize System Test ScheduleTask 3: Organize the System Test TeamTask 4: Establish the System Test EnvironmentTask 5: Install the System Test ToolsStep 2: Complete System Test CasesTask 1: Design/Script the Performance TestsMonitoring ApproachProbe ApproachTest DriversTask 2: Design/Script the Security TestsA Security Design StrategyTask 3: Design/Script the Volume TestsTask 4: Design/Script the Stress TestsTask 5: Design/Script the Compatibility TestsTask 6: Design/Script the Conversion TestsTask 7: Design/Script the Usability TestsTask 8: Design/Script the Documentation TestsTask 9: Design/Script the Backup TestsTask 10: Design/Script the Recovery TestsTask 11: Design/Script the Installation TestsTask 12: Design/Script Other System Test TypesStep 3: Review/Approve System TestsTask 1: Schedule/Conduct the ReviewTask 2: Obtain ApprovalsStep 4: Execute the System TestsTask 1: Regression Test the System FixesTask 2: Execute the New System TestsTask 3: Document the System Defects
21 Conduct Acceptance Testing
Step 1: Complete Acceptance Test PlanningTask 1: Finalize the Acceptance Test TypesTask 2: Finalize the Acceptance Test ScheduleTask 3: Organize the Acceptance Test TeamTask 4: Establish the Acceptance Test EnvironmentTask 5: Install Acceptance Test ToolsStep 2: Complete Acceptance Test CasesTask 1: Identify the System-Level Test CasesTask 2: Design/Script Additional Acceptance TestsStep 3: Review/Approve Acceptance Test PlanTask 1: Schedule/Conduct the ReviewTask 2: Obtain ApprovalsStep 4: Execute the Acceptance TestsTask 1: Regression Test the Acceptance FixesTask 2: Execute the New Acceptance TestsTask 3: Document the Acceptance Defects
22 Summarize/Report Test Results
Step 1: Perform Data ReductionTask 1: Ensure All Tests Were Executed/ResolvedTask 2: Consolidate Test Defects by Test NumberTask 3: Post Remaining Defects to a MatrixStep 2: Prepare Final Test ReportTask 1: Prepare the Project OverviewTask 2: Summarize the Test ActivitiesTask 3: Analyze/Create Metric GraphicsDefects by FunctionDefects by TesterDefect Gap AnalysisDefect Severity StatusTest Burnout TrackingRoot Cause AnalysisDefects by How FoundDefects by Who FoundFunctions Tested and Not TestedSystem Testing Defect TypesAcceptance Testing Defect TypesTask 4: Develop Findings/RecommendationsStep 3: Review/Approve the Final Test ReportTask 1: Schedule/Conduct the ReviewTask 2: Obtain ApprovalsTask 3: Publish the Final Test Report
SECTION 4 PROJECT MANAGEMENT METHODOLOGY
23 The Project Management Framework
The Project FrameworkProduct Quality and Project QualityComponents of the Project FrameworkThe Project Framework and Continuous Quality ImprovementThe Project Framework PhasesInitiation PhasePlanning PhaseExecuting, Monitoring, and Controlling PhasesImplement PhaseScoping the Project to Ensure Product QualityProduct Scope and Project ScopeThe Project CharterThe Scope StatementThe Role of the Project Manager in Quality ManagementThe Role of the Test Manager in Quality ManagementAnalyze the RequirementsPerform a Gap AnalysisAvoid Duplication and RepetitionDefine the Test DataValidate the Test EnvironmentAnalyze the Test ResultsDeliver the QualityAdvice for the Test ManagerRequest Help from OthersCommunicate Issues as They AriseAlways Update Your Business KnowledgeLearn the New Testing Technologies and ToolsImprove the ProcessCreate a Knowledge BaseThe Benefits of the Quality Project Management and the Project Framework
24 Project Quality Management
Project Quality Management ProcessesQuality PlanningIdentifying the High-Level Project ActivitiesEstimating the Test Work EffortTest PlanningEffort Estimation: Model ProjectQuality Standards
25 The Defect Management Process
Quality Control and Defect ManagementDefect Discovery and ClassificationDefect PriorityDefect CategoryDefect TrackingDefect ReportingDefect SummaryDefect MeetingsDefect MetricsQuality Standards
26 Integrated Testing and Development
Quality Control and Integrated TestingIntegrated TestingStep 1: Organize the Test TeamStep 2: Identify the Tasks to IntegrateStep 3: Customize Test Steps and TasksStep 4: Select Integration PointsStep 5: Modify the Development MethodologyStep 6: Test Methodology TrainingStep 7: Incorporate Defect RecordingThe Integrated Team
27 Test Management Constraints
Organizational ArchitectureTraits of a Well-Established Quality OrganizationDivision of ResponsibilitiesOrganizational RelationshipsUsing the Project Framework Where No Quality Infrastructure ExistsAd Hoc Testing and the Project FrameworkUsing a Traceability/Validation MatrixReporting the Progress
SECTION 5 EMERGING SPECIALIZED AREAS IN TESTING
28 Test Process and Automation Assessment
Test Process AssessmentProcess Evaluation MethodologyStep 1: Identify the Key ElementsStep 2: Gather and Analyze the InformationStep 3: Analyze Test MaturityThe Requirements Definition MaturityTest Strategy MaturityTest Effort Estimation MaturityTest Design and Execution MaturityRegression Testing MaturityTest Automation MaturityStep 4: Document and Present FindingsTest Automation AssessmentIdentify the Applications to AutomateIdentify the Best Test Automation ToolTest Scripting ApproachTest Execution ApproachTest Script MaintenanceTest Automation FrameworkBasic Features of an Automation FrameworkDefine the Folder StructureModularize Scripts/Test Data to Increase RobustnessReuse Generic Functions and Application-Specific Function LibrariesDevelop Scripting Guidelines and Review ChecklistsDefine Error Handling and Recovery FunctionsDefine the Maintenance ProcessStandard Automation FrameworksData-Driven FrameworkModular FrameworkKeyword-Driven FrameworkHybrid Framework
29 Nonfunctional Testing
Performance TestingLoad TestingStress TestingVolume TestingPerformance MonitoringPerformance Testing ApproachKnowledge Acquisition ProcessTest DevelopmentPerformance DeliverablesSecurity TestingStep 1: Identifying the Scope of Security TestingStep 2: Test Case Generation and ExecutionTypes of Security TestingNetwork ScanningPurposeToolsApproachVulnerability ScanningPurposeToolsApproachPassword CrackingToolsLog ReviewsApproachFile Integrity CheckersPurposeToolsVirus DetectorsToolsApproachPenetration TestingPurposeApproachUsability TestingGoals of Usability TestingApproach and ExecutionGuidelines for Usability TestingAccessibility Testing and Section 508Compliance Testing
30 SOA Testing
Key Steps of SOA Testing
31 Agile Testing
Agile User Stories Contrasted to Formal RequirementsWhat Is a User Story?Agile PlanningTypes of Agile TestingCompliance Testing
32 Testing Center of Excellence
Industry Best ProcessesTesting MetricsOperating ModelTest Automation FrameworkContinuous Competency Development
33 On-Site/Offshore Model
Step 1: AnalysisStep 2: Determine the Economic Trade-OffsStep 3: Determine the Selection CriteriaProject Management and MonitoringOutsourcing MethodologyOn-Site ActivitiesOffshore ActivitiesImplementing the On-Site/Offshore ModelKnowledge TransferDetailed DesignMilestone-Based TransferSteady StateApplication ManagementPrerequisitesRelationship ModelStandardsBenefits of On-Site/Offshore MethodologyOn-Site/Offshore Model ChallengesOut of SightEstablish TransparencySecurity ConsiderationsProject MonitoringManagement OverheadCultural DifferencesSoftware LicensingFuture of the Onshore/Offshore Model
SECTION 6 MODERN SOFTWARE TESTING TOOLS
34 Software Testing Trends
Automated Capture/Replay Testing ToolsTest Case Builder ToolsNecessary and Sufficient ConditionsTest Data Generation StrategiesSampling from ProductionStarting from ScratchSeeding the DataGenerating Data Based on the DatabaseA Cutting-Edge Test Case Generator Based on Requirements
35 Taxonomy of Software Testing Tools
Testing Tool Selection ChecklistCommercial Vendor Tool DescriptionsOpen-Source Freeware Vendor ToolsWhen You Should Consider Test AutomationWhen You Should NOT Consider Test Automation
36 Methodology to Evaluate Automated Testing Tools
Step 1: Define Your Test RequirementsStep 2: Set Tool ObjectivesStep 3a: Conduct Selection Activities for Informal ProcurementTask 1: Develop the Acquisition PlanTask 2: Define Selection CriteriaTask 3: Identify Candidate ToolsTask 4: Conduct the Candidate ReviewTask 5: Score the CandidatesTask 6: Select the ToolStep 3b: Conduct Selection Activities for Formal ProcurementTask 1: Develop the Acquisition PlanTask 2: Create the Technical Requirements DocumentTask 3: Review RequirementsTask 4: Generate the Request for ProposalTask 5: Solicit ProposalsTask 6: Perform the Technical EvaluationTask 7: Select a Tool SourceStep 4: Procure the Testing ToolStep 5: Create the Evaluation PlanStep 6: Create the Tool Manager’s PlanStep 7: Create the Training PlanStep 8: Receive the ToolStep 9: Perform the Acceptance TestStep 10: Conduct OrientationStep 11: Implement ModificationsStep 12: Train Tool UsersStep 13: Use the Tool in the Operating EnvironmentStep 14: Write the Evaluation ReportStep 15: Determine Whether Goals Have Been Met
SECTION 7 APPENDICES
Appendix A: Spiral (Agile) Testing Methodology
Appendix B: Software Quality Assurance Plan
Appendix C: Requirements Specification
Appendix D: Change Request Form
Appendix E: Test Templates
Appendix F: Checklists
Appendix G: Software Testing Techniques
Bibliography
Glossary
Index

Content preview from Software Testing and Continuous Quality Improvement, 3rd Edition

Methodology to Evaluate Automated Testing Tools

This part provides an outline of the steps involved in acquiring, implementing, and using testing tools. The management of any significant project requires that the work be divided into tasks for which completion criteria can be defined. The transition from one task to another occurs in steps; to permit the orderly progress of the activities, the scheduling of these steps must be determined in advance. A general outline for such a schedule is provided by the steps described. The actual time schedule depends on many factors that must be determined for each specific tool use.

Step 1: Define Your Test Requirements

The goals to be accomplished should be identified in a format that permits ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Improving Product Reliability and Software Quality, 2nd Edition

Publisher Resources

ISBN: 9781351722209

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Software Testing and Continuous Quality Improvement, 3rd Edition

by William E. Lewis

Chapter 36

Methodology to Evaluate Automated Testing Tools

Step 1: Define Your Test Requirements

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.