Testing Cloud Services

Testing Cloud Services

by Kees Blokland, Jeroen Mengerink, Martin Pol
Released September 2013
Publisher(s): Rocky Nook
ISBN: 9781492000020

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Everybody is confronted with cloud computing. Whether you are a user, designer, programmer, project manager, or tester, sooner or later the cloud affects your work. If you are involved in selecting or implementing services from the cloud, or in keeping them up and running, this book will prove to be an invaluable resource.

Testing Cloud Services covers an extensive list of risks that arise when implementing cloud computing, including some traditional risks and some completely new ones, and provides strategies for avoiding these risks and solving problems. Every risk is connected to existing, updated, and new test measures. It is necessary to start testing during the selection of cloud services, and continue end-to-end testing even after going live, as continuity risks arise all the time.

With this book in hand, you will save a lot of time and discover an effective approach to testing that can be applied in practice immediately!

Table of contents

  1. Cover Page
  2. Title Page
  3. Copyright Page
  4. Contents
  5. Introduction from the CEO
  6. Preface
  7. Acknowledgements
  8. 1 Introduction
  9. 2 What is Cloud Computing?
    1. 2.1 Essential characteristics
      1. On-demand self-service
      2. Broad network access
      3. Resource pooling
      4. Rapid elasticity
      5. Measured service
    2. 2.2 Service models
      1. IaaS
        1. IaaS examples
      2. PaaS
        1. PaaS examples
      3. SaaS
        1. SaaS examples
    3. 2.3 Implementation models
      1. Private cloud
      2. Community cloud
      3. Public cloud
      4. Hybrid cloud
  10. 3 Role of the Test Manager
    1. 3.1 In general
      1. 3.1.1 Risk analysis
      2. 3.1.2 Information from and agreement with the supplier
        1. Checklist for general supplier conditions and other sources
        2. Additional terms and conditions
      3. 3.1.3 End-to-end testing
        1. E2E testing is more than system integration
        2. The focus of the E2E test
        3. Creating E2E test cases
        4. Executing E2E test cases
        5. Automating the E2E test execution
        6. E2E regression testing
      4. 3.1.4 Advice
    2. 3.2 Tasks during selection, implementation, and production
      1. 3.2.1 Selection
      2. 3.2.2 Implementation
      3. 3.2.3 Production
    3. 3.3 Testing with the help of the cloud
      1. 3.3.1 Test outsourcing to the cloud with TOGA
        1. Initiation
        2. Setup
        3. Transition
        4. Operation
      2. 3.3.2 Crowdsourced testing
      3. 3.3.3 Test environments in the cloud
        1. Making a snapshot of the virtual environment
        2. Making a representative test environment
        3. Executing a portability test for mobile resources
      4. 3.3.4 Generating load
        1. Heavy load
        2. Worldwide load
  11. 4 From Risk to Test
    1. 4.1 Performance risks
    2. 4.2 Security risks
    3. 4.3 Availability and continuity risks
    4. 4.4 Functionality risks
    5. 4.5 Maintainability risks
    6. 4.6 Legislation and regulations risks
    7. 4.7 Suppliers and outsourcing risks
  12. 5 Test Measures
    1. 5.1 Testing during selection
      1. 5.1.1 Include cloud-related aspects
      2. 5.1.2 Determine completeness and controllability of selection criteria
        1. Selection criteria for a service
        2. Selection criteria for a supplier
      3. 5.1.3 Assess services and suppliers
        1. Inspect specifications and terms
        2. Ask for references and certificates
        3. Perform audits and inspections
        4. Proof of concept
        5. Simulate E2E business processes
      4. 5.1.4 Issue selection advice
      5. 5.1.5 Checklist selection criteria
    2. 5.2 Testing performance
      1. 5.2.1 Load test
      2. 5.2.2 Stress test
      3. 5.2.3 Endurance test or volume test
      4. 5.2.4 Testing elasticity and manual scalability
      5. 5.2.5 Setting up test cases
        1. Setting up operational profiles
        2. Step 1: Identify the initiators of operations
        3. Step 2: List operations
        4. Step 3: Review listed operations
        5. Step 4: Determine the frequency of operations
        6. Step 5: Determine the likelihood of each operation occurring
      6. 5.2.6 Test cases aimed at specific bottlenecks
      7. 5.2.7 Including cloud aspects in test cases
        1. The world
        2. Customer’s resources
      8. 5.2.8 Test cases for the stress test
      9. 5.2.9 Test cases for endurance/volume test
      10. 5.2.10 Test cases for elasticity
        1. Elasticity (automatic scalability)
          1. Load profile
          2. Boundary values
          3. Process
        2. Manual scalability
          1. Load profile
          2. Boundary values
          3. Process
      11. 5.2.11 Test setup for a performance test
      12. 5.2.12 Representative test environment
    3. 5.3 Testing security
      1. 5.3.1 Assessing network security
      2. 5.3.2 Inventorying supplier security
        1. Authentication
        2. Authorization
        3. Log files and audit trails
      3. 5.3.3 Inventorying customer security
      4. 5.3.4 Testing encryption
      5. 5.3.5 Testing authentication
      6. 5.3.6 Testing authorization
      7. 5.3.7 Testing security robustness against Internet attacks
      8. 5.3.8 Testing log files and audit trails
      9. 5.3.9 Testing security patch routines
      10. 5.3.10 Performing audits
    4. 5.4 Testing for manageability
      1. 5.4.1 Specifications on the supplier side
        1. Interface specifications
        2. Specifications for customer resources
        3. Platform specifications
        4. Infrastructure specifications
      2. 5.4.2 Specifications on the customer side
        1. Infrastructure specifications (IaaS)
        2. System specifications
        3. Architecture documentation
        4. Business process documentation
        5. Business requirements
      3. 5.4.3 User documentation
      4. 5.4.4 Test environment availability
        1. Test environments in IaaS/PaaS
        2. Test environments of the SaaS user
      5. 5.4.5 Test documentation
        1. Customer test documentation
        2. Supplier test documentation
      6. 5.4.6 Incident management procedure
        1. Supplier resolves the incident
        2. Customer resolves the incident
        3. The incident will not be resolved
      7. 5.4.7 Change procedure and version control
      8. 5.4.8 Maintainability of software
    5. 5.5 Availability/continuity testing
      1. 5.5.1 Failure Mode and Effect Analysis
      2. 5.5.2 The role of architecture
      3. 5.5.3 Hardware reliability
      4. 5.5.4 Software reliability
      5. 5.5.5 Guarantees and SLAs
        1. Guarantees
        2. SLAs
        3. Points to note
      6. 5.5.6 Impact of availability mechanisms
      7. 5.5.7 Internet and Internet connection
      8. 5.5.8 Testing failover
        1. Failover test cases
        2. Specific checkpoints in failover tests
        3. Test management aspects for failover tests
        4. Process simulation
      9. 5.5.9 Testing working offline
    6. 5.6 Functionality testing
      1. 5.6.1 Compatibility of service with business processes
      2. 5.6.2 Testing service quality
      3. 5.6.3 Testing user-friendliness
        1. User documentation
      4. 5.6.4 Testing interfaces to other systems
      5. 5.6.5 Testing service configuration
      6. 5.6.6 Customization by the supplier
      7. 5.6.7 Customization by the customer
      8. 5.6.8 Testing web services
        1. WSDL
        2. Tools
        3. Test cases
      9. 5.6.9 Multi-platform testing
        1. Considerations for the supplier
        2. Considerations for the customer
      10. 5.6.10 Testing of and testing with apps
        1. Testing an app (on a mobile device)
      11. 5.6.11 Testing for working offline
      12. 5.6.12 Testing for regression
      13. 5.6.13 Creating a test basis
        1. Process flows
        2. Use cases
        3. Classification tree
        4. CRUD
        5. Authorization table
        6. Interface specifications (agreements)
    7. 5.7 Testing migration
      1. 5.7.1 Migration test strategy
      2. 5.7.2 Minimal interruption of business processes
        1. Trial migration
      3. 5.7.3 Correct data migration in IaaS and PaaS
      4. 5.7.4 Correct data conversion with SaaS
        1. Do conversion rules work correctly?
        2. Does the conversion work properly on the input data?
        3. Is there any data lost?
        4. Are any partially completed transactions lost?
        5. During and after conversion, does any sensitive data remain available?
        6. Is too much data being transferred to the service through migration?
      5. 5.7.5 Migration performance
      6. 5.7.6 Data cleanup
      7. 5.7.7 Test environment migration
      8. 5.7.8 Parallel runs and trial runs
    8. 5.8 Testing due to legislation and regulations
      1. 5.8.1 Inventory of legislation and regulations
        1. Privacy legislation
          1. Data location
          2. Additional agreements
        2. Other legislation
        3. Other obligations
        4. Conflicting laws and rules
        5. Impending laws and rules
      2. 5.8.2 Checking for legislation and regulations
    9. 5.9 Testing in production
      1. 5.9.1 Continuity in production in the case of changes
        1. Changes in the service
        2. Changes in other systems
        3. Changes at the supplier
        4. Changes in the business processes
        5. Growth
        6. Changes in connected resources
        7. Changes to the Internet
        8. Conclusion
      2. 5.9.2 Measuring guarantees
        1. Availability
        2. Performance
        3. Scalability
        4. Security
      3. 5.9.3 Original selection criteria evaluation
      4. 5.9.4 Practical points to note
        1. (E2E) Regression test
        2. Measurement methods for KPIs
  13. 6 Completion
  14. Glossary
  15. Index
  16. Footnote
    1. 5 Test Measures

Product information

  • Title: Testing Cloud Services
  • Author(s): Kees Blokland, Jeroen Mengerink, Martin Pol
  • Release date: September 2013
  • Publisher(s): Rocky Nook
  • ISBN: 9781492000020