By David A. Chappell
Book Price: $39.95 USD
£28.50 GBP
PDF Price: $31.99

Cover | Table of Contents | Colophon

Table of Contents

Chapter 1: Introduction to the Enterprise Service Bus

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Across all industries, executives are demanding more value from their strategic business processes. What process is strategic to a given company may vary dramatically by industry, but a common theme is that CEOs want their IT organizations to measurably improve the flow of data and information driving key business decisions. Whether it's a financial services firm seeking a competitive advantage by guaranteeing a higher volume of faster foreign exchange trades, a retail chain looking to accelerate the flow of store data back to brand managers at corporate headquarters, or a building materials supplier striving to optimize order flow through a complex distribution chain, there are common and significant technical challenges to be overcome. Information is locked up in applications within different departments and organizations, and it is both time-consuming and costly to pry that data loose. In short, the enterprise is far from integrated.

The past several years have seen some significant technology trends, such as Service Oriented Architecture (SOA), Enterprise Application Integration (EAI), Business-to-Business (B2B), and web services. These technologies have attempted to address the challenges of improving the results and increasing the value of integrated business processes, and have garnered the widespread attention of IT leaders, vendors, and industry analysts. The Enterprise Service Bus (ESB) draws the best traits from these and other technology trends.

The ESB concept is a new approach to integration that can provide the underpinnings for a loosely coupled, highly distributed integration network that can scale beyond the limits of a hub-and-spoke EAI broker. An ESB is a standards-based integration platform that combines messaging, web services, data transformation, and intelligent routing to reliably connect and coordinate the interaction of significant numbers of diverse applications across extended enterprises with transactional integrity.

An extended enterprise represents an organization and its business partners, which are separated by both business boundaries and physical boundaries. In an extended enterprise, even the applications that are under the control of a single corporation may be separated by geographic dispersion, corporate firewalls, and interdepartmental security policies.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

SOA in an Event-Driven Enterprise

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In an event-driven enterprise, business events that affect the normal course of a business process can occur in any order and at any time. Applications that exchange data in automated business processes need to communicate with each other using an event-driven SOA to have the agility to react to changing business requirements. An SOA provides a business analyst or integration architect with a broad abstract view of applications and integration components to be dealt with as high-level services. In an ESB, applications and event-driven services are tied together in an SOA in a loosely coupled fashion, which allows them to operate independently from one another while still providing value to a broader business function.

In the realm of SOA, events are represented in an open XML format and flow through a transparent pipeline that's open to inspection and subject to intermediation...

— John Udell, InfoWorld

Service components in an SOA expose coarse-grained interfaces with the purpose of sharing data asynchronously between applications. Using an ESB, an integration architect pulls together applications and discrete integration components to create assemblies of services to form composite business processes, which in turn automate business functions in a real-time enterprise.

An ESB provides the implementation backbone for an SOA. That is, it provides a loosely coupled, event-driven SOA with a highly distributed universe of named routing destinations across a multiprotocol message bus. Applications (and integration components) in the ESB are abstractly decoupled from each other, and connect together through the bus as logical endpoints that are exposed as event-driven services.

With its distributed deployment infrastructure, an ESB can efficiently provide central configuration, deployment, and management of services that are distributed across the extended enterprise.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

A New Approach to Pervasive Integration

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The common goal of applying technologies such as SOA, EAI, B2B, and web services is to create an architecture for integration that can be pervasive across an extended enterprise and beyond. For an integration infrastructure to achieve this pervasiveness, it must have the following characteristics:

• It must adapt to suit the needs of general-purpose integration projects across a variety of integration situations, large and small. Adaptability includes providing a durable architecture that is capable of withstanding evolutions in protocols, interface technology, and even process modeling trends.
• It must link together applications that span the extended enterprise using a single unified approach and a common infrastructure.
• It must extend beyond the boundaries of a single corporate IT data center and into automating partner relationships such as B2B and supply-chain scenarios.
• It must have simplicity of design and low barriers to entry, enabling the everyday IT professional to become a self-empowered integration architect.
• It must provide an SOA across the pervasive integration that enables integration architects to have a broad, abstract view of corporate application assets and automated business processes.
• It needs the flexibility and ability to react to and meet the needs of changing business requirements and competitive pressures.

In an ESB, applications and event-driven services are tied together in an SOA in a loosely coupled fashion. This allows them to operate independently from one another while still providing value to a broader business function.

The ESB architecture addresses these needs, and is capable of being adopted for any general-purpose integration project. It is also capable of scaling pervasively across enterprise applications, regardless of physical location and technology platform. Any application can plug into an ESB network using a number of connectivity options, and immediately participate in data sharing with other applications that are exposed across the bus as shared services. This is why the ESB is often referred to as an

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

SOA for Web Services, Available Today

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Web services have bestowed newfound importance on service-oriented architectures by providing a standards-based approach to interoperability between applications. The main objective of web services has been to provide a service abstraction that allows interoperability between applications built using disparate platforms and environments. The achievement of this goal will provide an easier path to pervasive integration between applications.

With the advent of the ESB there is now a way to incorporate web services and SOA into a meaningful architecture for integrating applications and services into a backbone that spans the extended enterprise in a large-scale fashion. An ESB makes web services, XML, and other integration technologies immediately useful with the mature technology that exists today.

The core tenets of SOA are vital to the success of a pervasive integration project, and are already implemented quite thoroughly in the ESB. The web services standards are trending in the right direction, but remain incomplete with respect to the enterprise-grade capabilities such as security, reliability, transaction management, and business process orchestration. The ESB is based on today's established standards in these areas, and has real implementations that are already being deployed across a number of industries. The ESB is quite capable of keeping in step with the ongoing evolution of the web-services equivalents of these capabilities as they mature. Chapter 12 provides a more detailed discussion on this subject.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Conventional Integration Approaches

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

An ESB applies web services and other complementary standards by combining them with technology concepts and best practices learned from EAI brokers. However, an ESB is more than simply a web-services veneer on top of the same old EAI hub.

Traditional formalized approaches to integration have their pros and cons. Chapter 1 shows some of the high-level traits of integration approaches, which range from the least desirable on the lower left point of origin, to the most desirable on the upper right quadrant.

Figure 1-1: Characteristics of traditional EAI brokers, application servers, vanilla MOM, and ESB

Traditional EAI brokers, which include those that are built upon application servers, use a hub-and-spoke architecture. A hub-and-spoke architecture has the benefit of centralized functions, such as management of routing logic and business rules, but does not scale well across departmental or business unit boundaries. Chapter 2 will examine the huge price of early attempts at integration using EAI hubs, as well as their moderate success.

Application servers can interoperate through standard protocols, yet they link things together in a tightly coupled fashion, and intertwine the integration logic and application logic together.

EAI brokers provide increased value by separating the application logic from the integration and process routing logic, yet still suffer from the hub-and-spoke architecture.

Message Oriented Middleware (MOM) provides the ability to connect applications in a loosely coupled, asynchronous fashion. However, MOM by itself requires low-level coding in an application. Using a traditional MOM, along with custom coding techniques, can get you a long way toward a distributed integration solution. However, without a higher level of abstraction of the routing logic, this approach also suffers from having integration logic hard-wired and intertwined with the application logic. Depending on the MOM being used, even the distributed characteristic might be limited because some traditional MOM infrastructure is not capable of spanning physical network boundaries very well.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Requirements Driven by IT Needs

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

A key characteristic of an ESB is to provide the underpinnings to support the needs of distributed, loosely coupled business units and business partners automating supply chains. These capabilities in an ESB were born out of necessity, as a result of middleware vendors working with industry professionals who were trying to create an architecture for large-scale integration. These industry professionals included IT architects in large corporations, and innovators in the e-Marketplace trading hub community who needed to build a B2B trading exchange backbone based on shared services, messaging, XML, and numerous connectivity options, while adhering to industry standards for each component. Chapter 3 will discuss the many catalysts that contributed to the creation of the ESB concept.

At the same time, the biggest needs that had yet to be addressed included how to effectively provide integration capabilities such as application adapters, data transformation, and intelligent routing in a way that could be used for general-purpose integration projects across a variety of integration situations. Also required was a more universal technology and an architectural approach that could be used to connect applications beyond the needs of individual tactical integration projects.

IT professionals had been disappointed by some previous technology trends such as CORBA and EAI. CORBA had the right idea with SOA, but turned out to be too complex to implement and maintain due to its reliance on tightly coupled interfaces between applications and services. EAI also suffered from steep learning curves and expensive barriers to entry on individual projects (more on this in the next chapter). What was really needed was a simple approach to SOA, with an architecture that could be adapted to suit the general needs of any integration effort, large or small. In addition, there needed to be a durable architecture that was capable of withstanding evolutions in protocols, interface technology, and even process modeling trends. The ESB concept was created to address all these needs.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Industry Traction

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Since ESB was first introduced in 2002, the ESB approach to integration has been adopted by numerous significant vendors in the middleware, integration, and web-services markets. Its acceptance continues to grow steadily.

Analyst firms such as Gartner Inc., IDC, and ZapThink have been tracking and writing about the ESB technology trend since early 2002. In a report issued in 2002 (DF-18-7304), Gartner Inc. analyst Roy Schulte wrote the following:

A new form of enterprise service bus (ESB) infrastructure—combining message-oriented middleware, web services, transformation and routing intelligence—will be running in the majority of enterprises by 2005.

These high-function, low-cost ESBs are well suited to be the backbone for service-oriented architectures and the enterprise nervous system.

Those four pillars—MOM, web services, transformation, and routing intelligence—represent the foundation of any good ESB. This book will focus on the role of each piece and many other required components as we explore the ESB. We will examine what the ESB can do for an enterprise, and the role that each basic component plays. We will also explore some advanced topics, including architectural overviews of practical uses across a number of industries.

A number of middleware and integration vendors have either built, or are in the process of building, something that matches the description of the ESB. And the list is growing all the time. The Appendix lists all the known vendors. Some vendors say they are providing an ESB already; some have announced plans to create one; some are just using the terminology in marketing materials but don't really have anything substantial behind it. This technology category is destined to become as hot as application servers were in the late '90s, when more than 25 vendors were competing for the same technology space.

A few vendors in this list deserve special mention. Sonic Software pioneered the concept, and shortly thereafter a number of other smaller vendors got on board, saying they also were providing an ESB or were in the process of building one. Once the incumbent integration companies such as webMethods, SeeBeyond, and IBM finally got "on the Bus" and announced their intent to build an ESB, the ESB term really began to get widespread notice as a growing technology category with staying power.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Characteristics of an ESB

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Due to the fast flurry of vendors trying to gain attention in the growing ESB product category, combined with the number of industry analysts and journalists reporting with their opinions on it, understandably there is much confusion as to what an ESB actually is. This section serves to outline the main characteristics of an ESB.

As illustrated in Chapter 1, the ESB can form the core of a pervasive grid. It is capable of spanning an extended enterprise and beyond, having a global reach across departmental organizations, business units, and trading partners. An ESB is also well suited for localized integration projects, and provides flexible underpinnings enabling it to adapt to any type of integration environment.

Figure 1-2: ESB forms a pervasive grid that can span a global enterprise network

Applications plug into the bus as needed, and are capable of having visibility and of sharing data with any other applications or services that are plugged into the bus. While web-services interfaces are an integral part of an ESB architecture, all applications do not have to be modified to become true web services to participate in the ESB. Connectivity is achieved through multiple protocols, client API technologies, legacy messaging environments, and third-party application adapters.

Standards-based integration is a fundamental concept of an ESB. For connectivity, an ESB can utilize J2EE components such as the Java Message Service (JMS) for MOM connectivity, and J2EE Connector Architecture (JCA or J2CA) for connecting to application adapters. An ESB also integrates nicely with applications built with .NET, COM, C#, and C/C++. In addition, an ESB can easily integrate with anything that supports SOAP and web-services APIs, which includes de facto standard web-services toolkit implementations such as Apache Axis. For dealing with data manipulation, an ESB can use XML standards such as XSLT, XPath, and XQuery to provide data transformation, intelligent routing, and querying of "inflight" data as it flows through the bus. For dealing with SOA and business process routing, an ESB can use the Web Services Description Language (WSDL) to describe abstract service interfaces, and Business Process Execution Language for Web Services (BPEL4WS), WS-Choreography, or some other XML-based vocabulary such as ebXML BPSS, to describe abstract business processes.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Adoption of ESB by Industry

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Many nascent technologies suffer from the issue of gaining adoption by trying to find a problem to solve. ESB concepts, on the other hand, have evolved out of necessity via industry-leading architects working with vendors in the technology community to define and build it, so ESB has been adopted as it has been built. ESBs are already being put to use in a variety of industries, including financial services, insurance, manufacturing, retail, telecom, energy, food distribution, and government. Here are some examples.

• A leading subprime lender implemented an ESB to reduce application processing costs by 60%. This was accomplished by creating a unified view of customer and lending data across an eCredit system, third-party credit bureaus, and their backend systems.
• Leading banks have implemented Straight Through Processing of financial transactions using an ESB, at a considerable saving over manual processing.
• A Derivatives Trading system relies on an ESB to process more than 100,000 transactions a day for 1,200 users, accounting for several billion dollars in revenue.

• The world's largest life and health reinsurer, with $20 billion in annual revenue, generated significant savings using an ESB as a business process management solution to streamline the exchange of back-office transactional information between the main headquarters and the insurance brokers who market and manage their policies.

• A manufacturer of countertops and flooring is using an ESB to improve supply chain predictability and reduce out-of-stock conditions by implementing a co-managed inventory system and "availability to promise" (ATP) query system. In phase 1 of the deployment, the ESB is being used to link the manufacturer and 60 of its distributors in a supply-chain network.
  The deployment model of the ESB allows the manufacturer to deploy ESB service containers at the distributor sites. This is an alternative to deploying an integration broker at each remote distributor.
• A major manufacturer of lighting, televisions, and medical imaging equipment is using an ESB to create a unified integration backbone to connect all its data centers across its global business units, and to create a unified view of product data and billing information to customers worldwide.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Summary

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In summary, an ESB has the following characteristics:

Pervasiveness.

An ESB can be adapted to suit the needs of general-purpose integration projects across a variety of integration situations. It is capable of building out integration projects that can span an entire organization and its business partners.

Highly distributed, event-driven SOA.

Loosely coupled integration components can be deployed on the bus across widely distributed geographic deployment topologies, yet are accessible as shared services from anywhere on the bus.

Selective deployment of integration components.

Adapters, distributed data transformation services, and content-based routing services can be selectively deployed when and where they are needed, and can be independently scaled.

Security and reliability.

All components that communicate through the bus can take advantage of reliable messaging, transactional integrity, and secure authenticated communications.

Orchestration and process flow.

An ESB allows data to flow across any applications and services that are plugged into the bus, whether local or remote.

Autonomous yet federated managed environment.

An ESB supports local autonomy at a departmental and business unit level, and is still able to integrate in a larger managed integration environment.

Incremental adoption. An ESB can be used for small projects.

Each individual project can build into a much larger integration network, which can be remotely managed from anywhere on the bus.

XML support.

An ESB can take advantage of XML as its "native" datatype.

Real-time insight.

An ESB provides the underpinnings to enable real-time insight into live business data. BAM enablement is built right into the ESB fabric.

You should now have enough information about ESBs to whet your appetite. In the later, more detailed chapters, you will learn more about the underlying technical aspects. The next few chapters will discuss the evolution of the ESB, its current state of integration, the benefits of adopting XML in a generic data exchange architecture to mediate between diverse data representations, and asynchronous messaging and MOM.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Chapter 2: The State of Integration

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

A number of factors, both business and technical, have led to the need for a new approach to integration. There are business drivers such as changes in economic conditions, regulatory compliance, and the introduction of new disruptive hardware technology such as Radio Frequency Identification (RFID) tags, all of which foreshadow significant changes in the way businesses view application integration and data sharing. These drivers seem at odds with the current state of integration within enterprises, which is not as advanced as you might think. As we will explore in this chapter, the majority of applications that ought to be integrated simply aren't, and those that are integrated suffer from overly complex integration approaches that have grown unmanageable over time, due to a lack of a cohesive integration strategy that can be applied broadly.

Here are some current business drivers that are affecting the need for a broad-scale integration solution:

Economic drivers.

These have changed the shape of IT spending. Economic trends have caused IT departments to focus on the applications that are available and getting them to work together somehow.

Top priority: Integration.

Survey results show that integration continues to be at the top of the list of priorities for CIOs.

Regulatory compliance.

Sarbanes-Oxley, the PATRIOT Act, and FCC regulations are forcing corporations to build the internal infrastructure required to track, route, monitor, and retrieve business data in more detailed ways than ever before.

Straight-Through Processing (STP).

The goal of STP is the elimination of inefficiencies in business processes, such as manual rekeying of data, faxing, paper mail, or unnecessary data batching. In industries such as financial services, this helps to achieve near-zero-latency processing of financial transactions.

Radio Frequency Identification (RFID).

Seen as the next evolution of bar codes, RFID has the potential to generate new kinds of data in large volumes, which then needs to be routed, transformed, aggregated, and processed.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Business Drivers Motivating Integration

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Economic trends have caused IT departments to focus on making things work with applications they have available to them today. In the year leading up to Y2K, the majority of IT spending was focused on Y2K preparation, which included buying packaged applications that were Y2K-ready.

The subsequent economic downturn, whether attributed to post-Y2K, post-Internet-bubble, 9/11, or wartime uncertainty, has led to dramatic changes in IT spending. This has had a particular impact on integration, both positively and negatively. IT budgets are not what they were in pre-Y2K years. No longer do IT managers have the luxury of multimillion-dollar budgets to spend on integration broker software and services, with projects that could take 18-24 months to show results. IT spending has now become highly visible at the executive level, and individual projects are being highly scrutinized. Only the projects that are critical to business viability are getting funded. Corporations are demanding tangible results and ROI within the 3-6 month timeframe on a per-project basis, though they still maintain the strategic goal of improving overall operational efficiency.

A new era of frugality does not decrease the need for improvement in business processes or the need for integration. The business drivers are still there: the need for improved process cycle times, the need to reduce inventory levels, and the need to eliminate duplicate IT services, to name a few.

An IDC report surveyed 557 CIOs about their high-level priorities for 2004. The report said this about integration:

Of what might be called "market driver" trends, integration has replaced security as the highest priority in IT planning for 2004 in North America among IT and executives interviewed in June 2003.

The report also notes that integration and security rank third and fourth, respectively, among the highest priorities of CIOs, just behind "Infrastructure replacement/upgrade" and "IT cost-cutting."

That total percentage was influenced by the fact that 21% of "midmarket" companies ranked integration important, even above infrastructure replacement/upgrade and IT cost-cutting. Table 2-1 shows the answers to the question: "Which of the following issues do you expect to have single-highest priority in 2004? Select one."

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

The Current State of Enterprise Integration

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This section will explore the details of how enterprise applications are integrated, or not integrated, today. This includes a discussion of a prevailing problem across many organizations: the accidental architecture.

Over the past two decades, numerous distributed computing models have arrived on the scene, including DCE, CORBA, DCOM, MOM, EAI brokers, J2EE, web services, and .NET. However, indications are that only a small percentage of enterprise applications are connected, regardless of the technology being used. According to a research report from Gartner Inc., that number is less than 10%.

Another statistic is even more surprising—of the applications that are connected, only 15% are using formal integration middleware. The rest are using the ETL and batch file-transfer techniques, which are largely based on hand-coded scripting and other custom solutions. More information on ETL and batch file transfer, including their associated problems, can be found in Chapter 9.

The Gartner 15% statistic provides a sobering data point that illustrates the true state of integration today. How are the other 85% of applications connected? A very common situation that exists in enterprises today is what I refer to as "the accidental architecture." The accidental architecture is something that nobody sets out to create; instead, it's the result of years of accumulating one-of-a-kind pointed integration solutions. In an accidental architecture, corporate applications are perpetually locked into an inflexible integration infrastructure. They continue to be treated as "silos" of information because the integration infrastructure can't adapt to new business requirements (Figure 2-1).

Most integration attempts start out with a deliberate design, but over time, other pieces are bolted on and "integrated," and the handcrafted integration code drifts away from the original intent. Through incremental patches and bolt-ons, integrated systems can lose their design integrity, especially if the system is maintained by a large number of people to whom the original design intent may not have been well communicated. It's a fact of life that individual point-to-point integrations will drift away from consistency, as engineers make "just this one little change" that's expedient at the time. Eventually it becomes difficult to even identify the points for making changes, and to understand what the side effects would be as a result. In a deployed system this can lead to disastrous results that will negatively affect your business.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Leveraging Best Practices from EAI and SOA

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Before we go and sacrifice our previous efforts, throw away every preceding technology, and throw up our hands in defeat, there is a way to leverage some good lessons learned from integration brokers and still move away from the accidental architecture—and that is with the adoption of the ESB. Best practices in integration, which have been refined through experience with integration brokers, can now be combined with standards-based infrastructure based on XML, web services, reliable asynchronous messaging, and distributed ESB integration components. These collectively form an architecture for a highly distributed, loosely coupled integration fabric to deliver all the key features of an integration broker, but without all the barriers.

Migrating away from the accidental architecture and refactoring toward a uniform and consistent integration backbone using an ESB involves the prescriptive steps described in the following sections.

While an ESB is certainly capable of transporting many types of data formats, adopting XML as the means for exchanging data between applications (Figure 2-2), as has been done in some traditional EAI approaches, can have numerous benefits. As we will see in Chapter 4, leveraging XML can provide insulation from making global changes to data structures and interfaces across the accidental architecture all at once. The ESB can further facilitate that process by examining the content of XML messages and allowing control over where they are delivered, sometimes altering the path of delivery to include additional services that modify or augment the message.

Figure 2-2: The ESB approach uses XML as the means for sharing data between applications

Thinking and planning in terms of an SOA is a fundamental step in refactoring toward an ESB. As illustrated in Figure 2-3, the introduction of service-level interfaces provides a common layer of abstraction that creates a separation between the interfaces and the implementation. This eases the construction of composite business process definitions composed of coarse-grained service interfaces, using a common interface definition mechanism such as Web Services Description Language (WSDL).

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Refactoring to an ESB

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Getting from the accidental architecture to a uniform integration infrastructure on a global scale may seem like a daunting task. It's not realistic to get everything ready and then flip a switch to get all your applications onto the new infrastructure. This has been a major reason why organizations continue to add on to the accidental architecture as the status quo, even with the knowledge that they are only perpetuating its associated problems.

An ESB provides capabilities that help ease the pain of introduction. Chapter 9 will drill down into a particular case study of how to migrate away from a preexisting integration solution that was based entirely on FTP and nightly batch jobs.

Let's now revisit our discussion of the accidental architecture. In Figure 2-6, the solid, dotted, and dashed lines represent different types of connection technologies and communications protocols used to integrate the applications. Note that there is an existing "island of integration" represented by the integration broker, and that the link between the Point-of-Sale (PoS) application and the Finance application is using FTP file transfer. The link between PoS and the Enterprise Resource Planning (ERP) system had previously been upgraded to use SOAP over HTTP as a protocol, as had the link between the Sales Force Automation (SFA) and the Customer Relationship Management (CRM) applications.

Figure 2-6: Representative accidental architecture using SOAP communications, FTP, and hand-rolled sockets, and including an integration broker

The ESB can be introduced at a departmental level or on a per-project basis. Adopting the ESB at the project level allows you to get accustomed to doing standards-based integration using ESB service containers, with full confidence that the project will fit into a larger integration network and align with the corporate strategic goals of enterprise-wide integration.

The first step in our example ESB adoption is to integrate the front office. In Figure 2-7, the front office CRM, Finance, and SFA are connected into the ESB via "service containers." These containers are key components of the ESB architecture, and are explained in detail in Chapter 6. The nature of the integration through ESB service containers may vary. The interface between the container and the application may be accomplished using a third-party application adapter; the container may expose XML data, which is described using WSDL; or it may be implemented as completely custom-written code.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Summary

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter covered the following issues:

• A number of drivers contribute to the need for a broad-scale, general-purpose integration infrastructure.
• The accidental architecture is the dominant design in use today. In this kind of system, the enterprise is currently not very connected at all.
  • Only 10% of applications are linked.
  • Of these, only 15% use any kind of middleware.
  • To date, distributed computing technologies have perpetuated, not solved, the accidental architecture problem.
• Hub-and-spoke EAI brokers have had moderate success. However, they:
  • Are largely proprietary
  • Failed to provide organizations with a standardized integration platform that could be applied to general-purpose use across an enterprise
• The ESB draws value from lessons learned in EAI broker technology.
• Integration is a departmental and corporate culture issue as much as it is a technical issue.
• The ESB allows incremental adoption to occur in accordance with the individual needs of departmental development schedules.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Chapter 3: Necessity Is the Mother of Invention

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The ESB is a new architecture for integration that is flourishing in corporations around the world. To many casual observers, the ESB as a technology category seems to have come out of nowhere. In reality, though, the ESB has not just "happened." Over time, many catalysts helped it develop and evolve, and lessons were learned from past technology approaches that extend back more than a decade.

This chapter will examine some key concepts of the ESB, including the many requirements, technology drivers, and forces in the IT climate that led to the creation of the ESB concept. All of this will be discussed in the context of the recent history and evolution of the ESB. This discussion will illustrate the point that an ESB is not merely an academic exercise; it was born out of necessity, based on real requirements arising from difficult integration problems that couldn't be solved by any preexisting integration technology. The discussion will conclude with a study of an ESB deployment, with a manufacturer exposing inventory management and supply chain optimization functionality to its remote distributors as shared services through an ESB.

Sometimes solving a problem requires looking at previous attempts at solutions and learning from their drawbacks. Entire trends had to come about as predecessors to ESB for the IT and vendor communities to have something to point at and say, "I like that," "I don't like that," or "That's what I've been trying to build on my own." The Greek philosopher Plato is credited for the phrase "Necessity is the mother of invention." The ESB is a shining example of invention fostered by necessity. In this chapter we will explore those necessities, and how an ESB addresses them.

The ESB concept is the next generation of integration middleware, capable of being applied to a much broader range of integration projects than what could be handled by specialized integration brokers. However, it should be stated that ESB is not just EAI plus web services, nor is it MOM plus web services. A number of recognized trends, both technology-driven and business-driven, have had an equal share of influence on the evolution of the ESB.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

The Evolution of the ESB

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The creation of the ESB has been an evolutionary process. As we just discussed, a number of events in the industry had their part in catalyzing the creation of the ESB (Figure 3-1). This does not mean to imply that the predecessors of ESB were bad or failed technologies. Each contributing technology in the ESB ancestry was the best available for its time and continues to have its "meritt"[sic]. The ESB draws positive influences from its predecessor approaches, and avoids the downsides.

Figure 3-1: ESB catalysts: a timeline of technology and other events affecting the creation of the ESB

The invention of the ESB was not an accident. The ESB is a result of vendors working with forward-thinking customers who were trying to build a standards-based integration network using a foundation of SOA, messaging, and XML. These customers came from the end-user IT community in the manufacturing business, and from e-Marketplace infrastructure and trading exchange companies such as CommerceOne and GE Global Exchange Services (GE GXS).

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

The ESB in Global Manufacturing

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

A global company in the manufacturing sector is an example of an end-user IT organization that helped to catalyze the ESB. This manufacturing company is made up of at least five different major business units located around the world. Their goal was to have a common integration backbone based on message-oriented middleware infrastructure and standards-based interfaces. This effort was started and restarted several times, and had a resurgence of interest a few years ago, when the Java Message Service was first introduced as a way of providing standard interfaces to a common messaging backbone. This manufacturing company had many "islands of integration" and departmental pockets of proprietary and third-party message buses, which had been installed and controlled at a departmental or business-unit level. The requirement was that all departments and business units be integrated with each other to form a more consistent environment in which to plug in applications.

Their IT organization was looking to JMS to provide a standardized interface and common behavioral semantics across all applications, across all departments, across all business units. While many of us in the JMS business were excited about this, the reality is that JMS alone can't meet that requirement. The company also needed integration broker functionality such as routing based on rules and data transformation, all based on an abstraction of loosely coupled shared service interfaces.

At the time, the manufacturing company was faced with a dilemma caused by several things. While JMS had been selected as part of the solution, not all of the messaging vendors supported it. The existing messaging vendors and integration broker hubs that were entrenched in the various business units couldn't support the highly distributed approach required to span across global business units in a reasonable and manageable way. Some of the newer messaging vendors had the right JMS support, as well as the right security, firewall traversal, and message broker clustering to support the distributed topologies required to bring the geographically dispersed business units of the company together. However, there was no model in place for a service abstraction layer upon which to build an SOA, nor did the rest of the integration-stack layers exist that were required to integrate all of those applications across the organization.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Finding the Edge of the Extended Enterprise

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

For the past decade, through the era of EAI and the evolution of the Internet and application server technology, a clear dividing line has developed between the communications and application integration infrastructure within the four walls of a corporation, and the "external" communication with business partners, vendors, and customers. This separation has been driven largely by the capabilities and limitations of the technology. To date, technology such as application server infrastructure has been specifically designed to make clear distinctions between what's inside the firewall and what's outside the firewall. This distinction is evidenced by completely separate architectural approaches, with different programming models required for building applications. In the J2EE application server architecture, for example, this is manifested as a web container versus an EJB container.

Hub-and-spoke EAI brokers could get as far as the corporate boundaries, but were not really built for scaling beyond that. Various bridging technologies were designed to bridge the gap at the "edge" of the network. In many legacy cases, this is "bolted on" as an afterthought. The majority of the work being done in the area of web services has also been focused on this "edge" of the network.

But just where exactly is this "edge" of the network anyhow? Before we get to that, let's explore another ancestor of the ESB, the e-Marketplace, also known as the e-Commerce Trading Hub.

In a trading network of business partners, there is the desire to move away from expensive EDI Value Added Networks (VANs) and use the public Internet as a means of communications wherever possible, and to lower the barrier of entry such that small to medium enterprises can afford to participate. This was the impetus behind the creation of e-Marketplaces and trading exchanges such as those powered by CommerceOne and GE Global eXchange Services (GXS).

A trading exchange acts as an intermediary, or semiprivate business portal, that facilitates electronic commerce between buyers and suppliers in a supply chain. The majority of the interactions within the "portal" are not browser-based—they are performed directly between specialized applications that require little or no human interaction. The interactions occur between applications residing in a trading partner, and backend applications residing in the trading hub. These backend trading hub applications provide value-added functions such as the dispersion of Requests For Quote (RFQs) between a buyer and multiple suppliers, or Availability To Promise (ATP) data from suppliers to buyers (Figure 3-3).

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Standards-Based Integration

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The maturity and adoption of relevant standards for integration have helped to foster the emergence of the ESB as a technology trend. The ESB makes full use of standards for integration wherever possible. The use of such standards can have significant effects on a business, as follows:

• Allows you to leverage existing IT staff, rather than specialist consultants. The amount of information available on XML and web services standards such as SOAP, WSDL, XSLT, XPath, and XQuery is expanding at an ever-increasing rate. There is an educational ecosystem in which information about standards (and budding specifications) becomes available as the standards evolve. The introduction of a popular specification creates a fertile ground for industry experts to write articles, tutorials, and O'Reilly books on the subject, which in turn allows IT professionals to learn more and stay current. This means that the average IT professional can readily attain the expertise he or she needs to become the in-house integration architect.
• Reduces proprietary vendor lock-in. With a proprietary integration stack, an organization can't simply say, "Well, vendor A wasn't what we expected, so let's try vendor B." This would result in an expensive restart and relearning. Adopting standards as part of an integration infrastructure means the ability to pick and choose best-of-breed implementations from different vendors and have them work together.
• Java standards. While the ESB concept can be Java-free, Java provides a set of specifications that don't exist elsewhere and that can standardize components and interfaces between those components. These standards define application interfaces, behavioral semantics of middleware and application adapters, and deployment models. Standards such as JMS, MessageDrivenBeans, and JCA can significantly increase the ability to interoperate between enterprise applications and J2EE application servers. J2EE application servers from various vendors have found their way into most, or perhaps even all, enterprise organizations.
• Increases the ability to integrate with business partners using standard interfaces and standard protocols.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Case Study: Manufacturing

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

As an example of how ESB technology is changing the economics of integration in a real deployment scenario, let's take a look at how a building material manufacturer is using an ESB. We will pay special attention to how they are taking advantage of the selective deployment model of the ESB.

The manufacturer operates 50 plants in 15 countries. They distribute their products through large building supply retailers, such as Home Depot and Lowe's, as well as through a network of more than 60 independent distributors serving retail and wholesale customers in regional markets across the United States. Twenty-eight of their larger distributors are connected to their headquarters using an EDI VAN.

Connecting one supplier with 60 distributors is a simple challenge for an integration infrastructure that is capable of scaling out to thousands of diverse endpoints. However, the point of this case study is not the scale of the deployment, but the traits of the ESB that it utilizes. Its simplicity highlights yet another characteristic of the ESB—that it is capable of scaling up to large global integration networks, but is also well suited for small projects.

To improve operations and optimize the distribution chain, this manufacturing company embarked on building an infrastructure that would enable direct distributor participation in inventory management and ATP as a means of achieving real-time order fulfillment.

Section 3.5.1.1: Inventory management

The inventory management application allows the manufacturer to better anticipate the demand of its distributors by tracking each distributor's monthly inventory consumption and creating a recommended monthly order requisition for each distributor.

By deploying an ESB, the manufacturer can now analyze its distributors' order and sales histories, thus allowing inventory to be jointly managed. This was accomplished by implementing a message exchange that allows the manufacturer to anticipate a distributor's inventory requirements. In this scenario, distributors periodically send product activity data to the manufacturer, which uses that data to anticipate product consumption activity and determine when the distributors need to replenish the stock. They then generate a shipping schedule message indicating the products and quantities that the distributor should order to replenish its stock. The distributor will use this data to generate a purchase order back to the manufacturer.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Summary

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In this chapter, we saw that the ESB concept was developed out of necessity, and had many catalysts. We also explored the following:

• The ESB provides a pervasive, event-driven SOA, which is based on requirements of IT architects working together with vendors to build broad-scale integration networks using messaging, standard integration services, and standard interfaces.
• e-Marketplaces provided a fertile breeding ground for scalable and secure ESB infrastructure capable of supporting the needs of large trading hubs with potentially thousands of trading partners. Out of this environment, the requirements of sophistication routing across segregated data channels were identified.
• The proliferation and reasonable maturity of standards has provided the benefits of standards-based integration.
• Application servers have their place in IT infrastructure as containers for housing business logic. An ESB architecture can provide a loosely coupled integration fabric for integrating application servers with other application servers and cross-platform applications at large.
• Remote ESB service containers obviate the need to install integration brokers at every remote site to be integrated.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Chapter 4: XML: The Foundation for Business Data Integration

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter will explore the use of XML as a means for providing the mediation between diverse data structures and formats as data is passed between applications. We will also examine how ESB-enabled data transformation and content-based routing services can support an XML data exchange architecture that insulates individual applications from wholesale changes in data structures.

XML, the eXtensible Markup Language, is accepted by the industry as the ideal vehicle for sharing structured data among applications and organizations. Putting aside for a moment the benefits of web services technologies such as SOAP and WSDL, there is much to be gained by simply adopting XML as the means for representing data that is shared between applications. XML has many benefits, including the following:

XML is becoming universally understood.

XML has become a lingua franca for representing data and interfaces between applications. Many packaged-application vendors provide inputs and outputs in XML. Think about your own applications and how they could better integrate and share data if they accepted and emitted XML.

XML provides a richer data model

The power of XML to represent data is one reason that it's been adopted so broadly. XML lets you model hierarchies, lists, and complex types directly in the data structure. This is in contrast to an RDBMS, which requires table joins, special schema features, and mapping routines to achieve the same thing.

XML makes data self-describing

XML embeds metadata that describes the hierarchy and data element names. This is the basis for XML parser technology that can use the tags and associated schema to identify elements of data by name and datatype.

XML is dynamic

An XML document is loosely coupled with its schema, unlike data structures in relational tables, which are hard-wired to their schemas. This means that programs that extract data directly from the XML structure using standard XML expressions such as XPath are independent of the schema and loosely coupled with the structure itself.

XML eliminates fixed formats

The order in which the data elements appear in a document doesn't need to be fixed. Each element can be of a variable length and can include a variable number of other fields.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

The Language of Integration

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

XML, the eXtensible Markup Language, is accepted by the industry as the ideal vehicle for sharing structured data among applications and organizations. Putting aside for a moment the benefits of web services technologies such as SOAP and WSDL, there is much to be gained by simply adopting XML as the means for representing data that is shared between applications. XML has many benefits, including the following:

XML is becoming universally understood.

XML has become a lingua franca for representing data and interfaces between applications. Many packaged-application vendors provide inputs and outputs in XML. Think about your own applications and how they could better integrate and share data if they accepted and emitted XML.

XML provides a richer data model

The power of XML to represent data is one reason that it's been adopted so broadly. XML lets you model hierarchies, lists, and complex types directly in the data structure. This is in contrast to an RDBMS, which requires table joins, special schema features, and mapping routines to achieve the same thing.

XML makes data self-describing

XML embeds metadata that describes the hierarchy and data element names. This is the basis for XML parser technology that can use the tags and associated schema to identify elements of data by name and datatype.

XML is dynamic

An XML document is loosely coupled with its schema, unlike data structures in relational tables, which are hard-wired to their schemas. This means that programs that extract data directly from the XML structure using standard XML expressions such as XPath are independent of the schema and loosely coupled with the structure itself.

XML eliminates fixed formats

The order in which the data elements appear in a document doesn't need to be fixed. Each element can be of a variable length and can include a variable number of other fields.

When combined, these principles allow XML to encode the entire contents of a database—all its tables, columns, rows, and constraints—in a single XML file.

Well, some would say that is up for debate, but it is certainly more human-readable than some of its predecessors, such as EDI. XML is generally readable by many tools. You can view XML documents in Microsoft Explorer, and most developer tools have viewers or editors for XML. XML is also writable. XML is well suited for integration tools that can visually create and manage sophisticated data mappings.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Applications Bend, but Don't Break