Accelerating Modernization with Agile Integration

Accelerating Modernization with Agile Integration

by Adeline SE Chun, Aiden Gallagher, Amar A Shah, Callum Jackson, Claudio Tagliabue, Iliya Dimitrov, James Blackburn, Joel Gomez, Kim Clark, Lee Gavin, Maria Menendez, Martin Evans, Mohammed Alreedi, Murali Sitaraman, Nick Glowacki, Shishir Narain, Timothy Quigly, Tony Curcio, Ulas Cubuk, Vasfi Gucer
Released July 2020
Publisher(s): IBM Redbooks
ISBN: 9780738458366

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

The organization pursuing digital transformation must embrace new ways to use and deploy integration technologies, so they can move quickly in a manner appropriate to the goals of multicloud, decentralization, and microservices. The integration layer must transform to allow organizations to move boldly in building new customer experiences, rather than forcing models for architecture and development that pull away from maximizing the organization's productivity.

Many organizations have started embracing agile application techniques, such as microservice architecture, and are now seeing the benefits of that shift. This approach complements and accelerates an enterprise's API strategy. Businesses should also seek to use this approach to modernize their existing integration and messaging infrastructure to achieve more effective ways to manage and operate their integration services in their private or public cloud.

This IBM® Redbooks® publication explores the merits of what we refer to as agile integration; a container-based, decentralized, and microservice-aligned approach for integration solutions that meets the demands of agility, scalability, and resilience required by digital transformation. It also discusses how the IBM Cloud Pak for Integration marks a significant leap forward in integration technology by embracing both a cloud-native approach and container technology to achieve the goals of agile integration.

The target audiences for this book are cloud integration architects, IT specialists, and application developers.

Table of contents

  1. Front cover
  2. Notices
    1. Trademarks
  3. Preface
    1. Authors
    2. Now you can become a published author, too!
    3. Comments welcome
    4. Stay connected to IBM Redbooks
  4. Chapter 1. Introduction
    1. 1.1 Integration has changed
    2. 1.2 Audience and scope
    3. 1.3 Navigating the book
  5. Chapter 2. Agile integration
    1. 2.1 Agile integration: A brief introduction
      1. 2.1.1 People: Decentralized integration ownership
      2. 2.1.2 Architecture: Delivery focused integration architecture
      3. 2.1.3 Infrastructure aspect: Cloud-portable integration infrastructure
    2. 2.2 The journey so far: SOA, ESBs, and APIs
      1. 2.2.1 The forming of the ESB pattern
      2. 2.2.2 What went wrong for the centralized ESB pattern
      3. 2.2.3 The API economy
    3. 2.3 Microservices
      1. 2.3.1 The rise of lightweight run times
      2. 2.3.2 Microservices architecture: A more agile and scalable way to build applications
      3. 2.3.3 Comparing SOA and the microservices architecture
      4. 2.3.4 Bi-modal IT and decentralization
      5. 2.3.5 Decentralization and integration versus point-to-point
    4. 2.4 The three aspects of agile integration
    5. 2.5 People: Decentralized integration ownership
      1. 2.5.1 Moving to a decentralized and business-focused team structure
      2. 2.5.2 Big bangs generally lead to big disasters
      3. 2.5.3 Decentralized integration ownership and decentralized infrastructures
      4. 2.5.4 Prioritizing project delivery first
      5. 2.5.5 Evolving the role of the architect
      6. 2.5.6 Automation: The key to consistency in decentralization
      7. 2.5.7 Producing multi-skilled developers
      8. 2.5.8 Conclusions on decentralized integration ownership
    6. 2.6 Architecture: Delivery-focused architecture
      1. 2.6.1 Consumer-centric API management
      2. 2.6.2 Fine-grained application integration
      3. 2.6.3 Application-owned messaging and events
      4. 2.6.4 Conclusions on delivery-focused architecture
    7. 2.7 Technology: Cloud-native infrastructure
      1. 2.7.1 Virtual machines, containers, and serverless
      2. 2.7.2 Cloud-native approach
      3. 2.7.3 Portability: Public, private, and multicloud
      4. 2.7.4 Conclusion on cloud-native integration infrastructure
  6. Chapter 3. Agile integration: Capability perspectives
    1. 3.1 Capability perspective: API management
      1. 3.1.1 A brief history of API management
      2. 3.1.2 Cloud-native infrastructure
    2. 3.2 Capability perspective: Application integration
      1. 3.2.1 Moving to a cloud-native approach
      2. 3.2.2 Fine-grained deployment: Breaking up the ESB
      3. 3.2.3 Grouping integrations
      4. 3.2.4 Stateless components
      5. 3.2.5 Image-based deployment
      6. 3.2.6 Elastic, agnostic infrastructure and container orchestration platforms
      7. 3.2.7 Lightweight run times: How the modern integration run time has changed
      8. 3.2.8 Log-based monitoring
      9. 3.2.9 API intra-application communication
      10. 3.2.10 Event-driven architecture
      11. 3.2.11 Agile methods
      12. 3.2.12 Continuous Integration and Continuous Delivery and Deployment
      13. 3.2.13 DevOps
      14. 3.2.14 Creating integrations is becoming easier
      15. 3.2.15 Decentralizing integration ownership
      16. 3.2.16 Using integration run times in a microservices application
    3. 3.3 Capability perspective: Messaging and event streams
      1. 3.3.1 A brief history of asynchronous communication
      2. 3.3.2 Introducing messaging and event streams concepts
      3. 3.3.3 Differentiating capabilities
      4. 3.3.4 A detailed look at messaging
      5. 3.3.5 A detailed look at event streams
    4. 3.4 Capability perspective: Files and Business-to-Business
    5. 3.5 Hybrid and multicloud considerations
      1. 3.5.1 Multicloud: Multiple cloud services
      2. 3.5.2 Hybrid Cloud: Multiple deployment modes (public, private, and legacy)
      3. 3.5.3 Evolution of API deployment modes
    6. 3.6 Use cases driving hybrid and multicloud adoption
      1. 3.6.1 Multicloud strategy
      2. 3.6.2 Cloud bursting and scalability
      3. 3.6.3 Disaster recovery
      4. 3.6.4 Application affinity
      5. 3.6.5 Regional flexibility
      6. 3.6.6 Geographical high availability
    7. 3.7 References
  7. Chapter 4. Cloud-native concepts and technology
    1. 4.1 Defining cloud-native
    2. 4.2 Key elements of cloud-native applications
      1. 4.2.1 Modular components
      2. 4.2.2 Preferring stateless
      3. 4.2.3 Immutable deployment
      4. 4.2.4 Elastic, agnostic infrastructure and container orchestration platforms
      5. 4.2.5 Lightweight run times
      6. 4.2.6 Log-based monitoring
      7. 4.2.7 API-led intra-application communication
      8. 4.2.8 The reprise of event-driven architecture
      9. 4.2.9 Agile methods
      10. 4.2.10 Continuous Integration and Continuous Delivery and Deployment
      11. 4.2.11 Continuous Adoption
      12. 4.2.12 DevOps
    3. 4.3 Twelve-factor apps
      1. 4.3.1 Conclusion on 12-factor apps
    4. 4.4 Container technology: the current state of the art
      1. 4.4.1 Containers
      2. 4.4.2 Container orchestration
      3. 4.4.3 Kubernetes primer
    5. 4.5 Cloud-native is not for everyone, nor for everything
    6. 4.6 Realizing the true benefits of containerization
    7. 4.7 Application boundaries in a container-based world
      1. 4.7.1 Implicit and explicit boundaries
      2. 4.7.2 Why do application boundaries matter?
      3. 4.7.3 How should we choose the application boundaries?
    8. 4.8 Service mesh
      1. 4.8.1 Role of a service mesh
      2. 4.8.2 Service meshes and API management
    9. 4.9 Cloud-native security – an application-centric perspective
      1. 4.9.1 Scope of this section
      2. 4.9.2 Limitations of traditional security models
      3. 4.9.3 Challenges unique to cloud-native
      4. 4.9.4 Securing a cloud-native application
      5. 4.9.5 Hybrid solutions: Securing cloud to cloud and cloud to ground
    10. 4.10 The future of cloud-native
      1. 4.10.1 Are software-as-a-service applications serverless?
      2. 4.10.2 Function-as-a-service: a more accurate term for serverless?
      3. 4.10.3 Could any runtime be provided in a FaaS model?
      4. 4.10.4 FaaS for cloud-native?
      5. 4.10.5 Are there downsides to FaaS?
      6. 4.10.6 Conclusions on FaaS
  8. Chapter 5. IBM Cloud Pak for Integration
    1. 5.1 IBM Cloud Pak for Integration
      1. 5.1.1 One platform supported by common services
      2. 5.1.2 IBM Cloud Pak for Integration - benefits
      3. 5.1.3 License flexibility for other non-containerized architectures
      4. 5.1.4 Getting access to IBM Cloud Pak for Integration for the exercises
    2. 5.2 Red Hat OpenShift Container Platform
    3. 5.3 API Lifecycle: IBM API Connect
      1. 5.3.1 Key phases of the API Lifecycle
      2. 5.3.2 API Lifecycle components
      3. 5.3.3 API lifecycle in combination with other capabilities
      4. 5.3.4 Product deployment options
    4. 5.4 Integration security: IBM DataPower Gateway
      1. 5.4.1 Security Gateway
      2. 5.4.2 API Gateway
      3. 5.4.3 DataPower and agile integration
    5. 5.5 Application integration: IBM App Connect
      1. 5.5.1 User-aligned integration tooling
      2. 5.5.2 No-code RESTful integration services
      3. 5.5.3 Flexible integration patterns
      4. 5.5.4 Broad deployment options
      5. 5.5.5 Extended connectivity
      6. 5.5.6 Situational awareness with insightful and actionable notifications
      7. 5.5.7 Quick utilization of artificial intelligence (AI) services
      8. 5.5.8 Rapid visual orchestration of data and systems for API-driven architectures
      9. 5.5.9 Lightweight integration runtime for cloud native deployment
      10. 5.5.10 Grown from a trusted market leading product
      11. 5.5.11 IBM App Connect on deployment options
    6. 5.6 Enterprise Messaging: IBM MQ
    7. 5.7 Event Streaming: IBM Event Streams
    8. 5.8 High-Speed File Transfer: IBM Aspera
      1. 5.8.1 Fast, Adaptive and Secure Protocol (FASP) technology
      2. 5.8.2 Aspera on Cloud
    9. 5.9 Service Mesh: Istio
  9. Chapter 6. Practical agile integration
    1. 6.1 Introduction
    2. 6.2 Application Integration to front a data store with a basic API
      1. 6.2.1 Db2 setup
      2. 6.2.2 Db2 table setup
      3. 6.2.3 Swagger definitions
    3. 6.3 Expose an API using API Management
      1. 6.3.1 Importing the API definition
      2. 6.3.2 Configuring the API
      3. 6.3.3 Merging two application flows into a single API
      4. 6.3.4 Add simple security to the API
    4. 6.4 Messaging for reliable asynchronous data update commands
      1. 6.4.1 Enable create, update, delete via commands
      2. 6.4.2 Deploy and configure Queue Manager
      3. 6.4.3 Queue manager configuration
      4. 6.4.4 DB commands implementation
      5. 6.4.5 Graphical data maps implementation
      6. 6.4.6 Policy definitions
      7. 6.4.7 BAR file creation
      8. 6.4.8 Override policies for environment specific values
      9. 6.4.9 Global transaction coordination considerations
      10. 6.4.10 Conclusion
    5. 6.5 Consolidate the new IBM MQ based command pattern into the API
      1. 6.5.1 Defining the API data model
      2. 6.5.2 Paths
      3. 6.5.3 Securing the API
      4. 6.5.4 The Assembly
      5. 6.5.5 API testing
      6. 6.5.6 API socialization
      7. 6.5.7 Conclusion
    6. 6.6 Advanced API security
      1. 6.6.1 Import the API into IBM API Connect
      2. 6.6.2 Configure the API
      3. 6.6.3 Add basic security to the API
      4. 6.6.4 Test the API
      5. 6.6.5 Securing the API Using OAUTH
      6. 6.6.6 External client testing
    7. 6.7 Create event stream from messaging
      1. 6.7.1 Creating a new event stream topic
      2. 6.7.2 Running the IBM MQ source connector
      3. 6.7.3 Configuring the connector to connect to IBM MQ
    8. 6.8 Perform event-driven SaaS integration
      1. 6.8.1 Scenario
      2. 6.8.2 IBM App Connect event-driven flow to Salesforce, Google and Slack SaaS applications
      3. 6.8.3 Prerequisites
      4. 6.8.4 Create flows
      5. 6.8.5 Test your flow
      6. 6.8.6 Conclusion
    9. 6.9 Implementing a simple hybrid API
      1. 6.9.1 Business scenario
      2. 6.9.2 Invoking existing APIs from IBM App Connect Designer
      3. 6.9.3 Solution overview
      4. 6.9.4 Preparing the external SaaS applications
      5. 6.9.5 Create simulated on-premises API flow
      6. 6.9.6 Create Hybrid API
      7. 6.9.7 Test the flows
      8. 6.9.8 First, test the simulated on-premises API
      9. 6.9.9 Final Hybrid API integrated testing
      10. 6.9.10 Conclusion
    10. 6.10 Implement event sourced APIs
      1. 6.10.1 Implementing the query side of the CQRS pattern
      2. 6.10.2 Event sourced programming - a practical example
      3. 6.10.3 How to do it?
      4. 6.10.4 Creating the flow in IBM App Connect
      5. 6.10.5 Mapping the received events to the output required
      6. 6.10.6 Sending the new payload to the database
      7. 6.10.7 Client applications
    11. 6.11 REST and GraphQL based APIs
      1. 6.11.1 IBM, GraphQL, and Loopback
      2. 6.11.2 LoopBack models and relationships
    12. 6.12 API testing
      1. 6.12.1 Create a test from an API request
      2. 6.12.2 Update the test case from a Swagger file and publish
      3. 6.12.3 Gain insights into API quality
    13. 6.13 Large file movement using the claim check pattern
      1. 6.13.1 Build the file transfer
      2. 6.13.2 Build an event-driven flow
  10. Chapter 7. Field notes on modernization for application integration
    1. 7.1 IBM App Connect adoption paths
      1. 7.1.1 Agile integration
      2. 7.1.2 Adoption path options
      3. 7.1.3 Conclusion
    2. 7.2 Splitting up the ESB: Grouping integrations in a containerized environment
      1. 7.2.1 What grouping do you have today?
      2. 7.2.2 Splitting by business domain
      3. 7.2.3 What about integrations that span business domains?
      4. 7.2.4 Grouping within a domain
      5. 7.2.5 Stable requirements and performance
      6. 7.2.6 Synchronous versus asynchronous patterns
      7. 7.2.7 Shared lifecycle
      8. 7.2.8 A worked example
      9. 7.2.9 Conclusion
    3. 7.3 When does IBM App Connect need a local MQ server?
      1. 7.3.1 Benefits of being dependency-free in container-based environments
      2. 7.3.2 When can we manage without a local MQ server?
      3. 7.3.3 Can I talk to multiple queues in the same transaction without a local MQ server?
      4. 7.3.4 Coordinating a two-phase commit requires a local MQ server
      5. 7.3.5 When else do I need a local MQ server?
      6. 7.3.6 Why do we have so many integrations that use server connections?
      7. 7.3.7 Conclusion
    4. 7.4 Mapping images to helm charts
      1. 7.4.1 Developing helm charts for Kubernetes
      2. 7.4.2 Upgrading (extending) helm charts
    5. 7.5 Continuous Integration and Continuous Delivery Pipeline using IBM App Connect V11 architecture
      1. 7.5.1 Continuous Integration Delivery and Deployment
      2. 7.5.2 Example pipeline - High-level concepts
      3. 7.5.3 CI/CD pipeline in depth
      4. 7.5.4 Considerations for CI/CD pipelines with IBM App Connect
      5. 7.5.5 Practical example
    6. 7.6 Continuous Adoption for IBM App Connect
      1. 7.6.1 What does Continuous Adoption apply to in IBM App Connect?
      2. 7.6.2 How can Continuous Adoption be implemented with IBM App Connect?
    7. 7.7 High Availability and Scaling considerations for IBM App Connect in containers
      1. 7.7.1 Overview
      2. 7.7.2 Scaling
      3. 7.7.3 High Availability
    8. 7.8 Migrating centralized ESB to IBM App Connect on containers
      1. 7.8.1 Overview
      2. 7.8.2 Considerations for IBM MQ based integrations in containers
      3. 7.8.3 Considerations for Http/WebServices based integration flows in containers
      4. 7.8.4 Considerations for integrations that interact with databases
      5. 7.8.5 Considerations for in containers
      6. 7.8.6 Considerations for TCP/IP based integrations in containers
      7. 7.8.7 Considerations for file-based integration in containers
      8. 7.8.8 Considerations for integrating IBM App Connect with IBM Event Streams
    9. 7.9 Splitting an integration across on-premises and cloud
      1. 7.9.1 Overview
      2. 7.9.2 Using callable flows with IBM App Connect Designer
      3. 7.9.3 Callable flows versus APIs
      4. 7.9.4 Cloud debugger for ACE on Cloud applications
  11. Chapter 8. Field notes on modernization for API lifecycle
    1. 8.1 Move from DataPower only to API Connect
      1. 8.1.1 Security Gateway
      2. 8.1.2 API gateway
      3. 8.1.3 Connectivity and mediation
      4. 8.1.4 DataPower and API Connect compared
      5. 8.1.5 DataPower to API Connect migration
    2. 8.2 Enterprise APIs across a hybrid or multicloud boundary
    3. 8.3 How many API Connect Clouds and Gateways
      1. 8.3.1 Separate API Clouds
      2. 8.3.2 Separate API Gateway Cluster
    4. 8.4 Organization, Catalog and Space responsibilities for APIs
    5. 8.5 Automated provisioning of a new API provider team
    6. 8.6 High availability and scaling on containers for API Management
      1. 8.6.1 High availability in a containerized environment
      2. 8.6.2 Scalability of API Connect in containerized environment
      3. 8.6.3 Conclusion
    7. 8.7 IBM API Connect API Test Pyramid
      1. 8.7.1 Practical Test Pyramid
      2. 8.7.2 Requirements
      3. 8.7.3 Test types
      4. 8.7.4 Automated testing
      5. 8.7.5 Conclusion
  12. Chapter 9. Field notes on modernization for messaging
    1. 9.1 Modernizing your messaging topology with containers
      1. 9.1.1 Typical existing topology
      2. 9.1.2 Removing local connections
      3. 9.1.3 Containerizing queue managers
      4. 9.1.4 Fine-grained queue manager deployment
      5. 9.1.5 Decentralization
      6. 9.1.6 Application containerization and operational consistency
    2. 9.2 IBM MQ availability
      1. 9.2.1 Kubernetes deployment styles: Deployments (replica sets) versus stateful sets
      2. 9.2.2 Multi-instance queue managers in containers
      3. 9.2.3 Further improving service availability with additional independent queue managers
      4. 9.2.4 Connection distribution
    3. 9.3 IBM MQ scaling
    4. 9.4 Automation of IBM MQ provisioning using a DevOps pipeline
      1. 9.4.1 Design for DevOps pipeline
      2. 9.4.2 Building a sample IBM MQ pipeline
  13. Appendix A. Additional material
    1. Locating the GitHub material
    2. Cloning the GitHub material
  14. Related publications
    1. IBM Redbooks
    2. Online resources
    3. Help from IBM
  15. Back cover

Product information

  • Title: Accelerating Modernization with Agile Integration
  • Author(s): Adeline SE Chun, Aiden Gallagher, Amar A Shah, Callum Jackson, Claudio Tagliabue, Iliya Dimitrov, James Blackburn, Joel Gomez, Kim Clark, Lee Gavin, Maria Menendez, Martin Evans, Mohammed Alreedi, Murali Sitaraman, Nick Glowacki, Shishir Narain, Timothy Quigly, Tony Curcio, Ulas Cubuk, Vasfi Gucer
  • Release date: July 2020
  • Publisher(s): IBM Redbooks
  • ISBN: 9780738458366