Just Spring Integration

Integration Strategies

You may have seen applications reading configuration from a file system, persisting data to a database, sending messages to an external client, publishing email, FTPing daily snapshots, and performing other routine tasks. Whether you know it or not, your application is talking to different systems—File System, Database, email, FTP, etc. You may even have developed some sort of adapter that will integrate your application with these external systems.

Integrating disparate systems is not an uncommon task. Before the advent of integration frameworks, there were common strategies in the integration space. The most popular strategies are:

• Shared File Systems: Two or more applications share a common file system; one may write to it while the other may poll the file system to read it. The sender and receiver are decoupled in this case. This solution certainly works, but has drawbacks like performance, reliability, and dependency on the File system.

• Single Database: In this strategy, applications share the data from a single database. One application writes data to the table while the other simply reads from the table. The drawback is that this setup forces applications to use a unified schema throughout the organization. Shared databases also pose a few other issues, such as network lag and lock contention.

• Messaging: This strategy mainly encourages and supports sender-receiver decoupling. A sender application sends a piece of data enclosed in a message to a messaging middleman and forgets about it. A consumer consumes the message whenever it can and begins its own workflow. One of the advantages of using Messaging as the medium is that the sender and receiver are decoupled completely. Also, the messages can be enriched, transformed, routed, and filtered before hitting the end channels.

We will examine the Messaging strategy and how Spring Integration gives us the tools to develop a full-fledged application.

Messaging Patterns

We all know that in our day-to-day life, there are some common problems that may have common solutions. In the messaging domain, too, you can observe such recurring problems and encounter some solutions to them. These common solutions are recorded as patterns. During the last couple of decades, formal patterns emerged for some of the recurring problems. Instead of reinventing the wheel, one could create the solutions using the approach laid out by these patterns. For example, to decouple the sender and receiver, one can introduce a Message pattern—the sender sends a message, while the receiver receives this message. Each party is unaware of the other.

Any messaging system has a few building blocks to work with, such as Messages, Channels, Transformers, etc. These are identified as patterns and discussed later in the chapter.

One pattern that might require a mention is the pipes and filters pattern.

Let’s look at a very simple example to demonstrate this pattern—a Unix pipeline ( | ) command. Most of us should be familiar with this command.

The pipeline command, denoted by |, is used to combine several Unix commands to achieve a complex task. Although it looks simple, this example shows the powerful concept of the pipes and filters architecture.

Our requirement is to find the word count of Just Spring in the just-spring-titles.txt file. Run the command as shown below to achieve this:

mkonda$ cat just-spring-titles.txt | grep "Just Spring" | wc -l

Going into detail, the above command consists of three endpoints and two channels. The cat, grep, and wc are the endpoints while the pipe (|) acts as a channel.

The cat command displays the contents of the file. But the display, instead of being sent to the screen, is sent to the grep command using the pipe. The grep command then picks up the contents and searches for the Just Spring string. The result is then passed on to another command, wc, in this case. This simply displays the word count on the screen.

Note that these commands do not know anything about each other. These are small, narrowly focused tools that take in messages and publish them. They don’t depend on each other’s API and can be developed independently.

If you are familiar with JMS or distributed technologies, you may have heard of Enterprise Messaging. Your application talking to another application over the network can be considered an Enterprise application. You may have to use an application server to host these applications if you want to expose services so other applications can call the service according to their needs.

However, we can also introduce messaging to a standalone program that may run in a single process (single JVM). Spring Integration is one such framework for inter- and intra-application messaging.

While knowing these patterns will help you understand Spring Integration technology, it is not a requirement. The Spring Integration framework is developed to implement the patterns discussed in Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf. I would advise you to read this book to understand the EAI subject in depth.

Traditional Programming Model

Let’s consider an application that loads Trades from an external system (File, in this example). The requirements of processing these Trades are as follows:

• Trades should be segregated based on Trade types (NEW, CANCEL, AMEND, etc).

• Trades are then processed accordingly and persisted.

• An auditor tool should be notified once the Trades are persisted.

These are typical application requirements. Usually, one can code them in a single component, as shown in the example below:

//Pseudo code
public class TradesLoader {
    private List<Trade> trades = null;
    ....
    public void start(){
      trades = loadTrades(tradesFile);
    
      (for Trade t: trades){
         processTrade(t);
         persistTrades(t);
         auditNotify(t);
      }
    }

    public void processTrade(Trade trade){
        if (t.getStatus().equalsIgnoreCase("new")) {
           processNewTrade();
        } else if (t.getStatus().equalsIgnoreCase("ammend")) {
           processAmmendTrade();
        } else {
          processOtherTrade();
        }
    }

    public void loadTrades(File fromFile){ .. }
    public void persistTrades(Trade trade){ .. }
    public void auditNotify(Trade trade){ .. }
    
    public static void main(String[] args) {
      new TradesLoader().start();
    }
}

The loadTrades method reads the file and transforms its content into a list of Trades. Each Trade is then sent through various actions, such as processing, persisting, and notifying, as shown in the snippet above.

The drawback of this model is that the process typically works sequentially. Please see Figure 1-1, which illustrates this model.

Figure 1-1. Serial Processing

There is no harm in adopting this programming model, except that the component is tightly coupled to a business workflow. If we have to add another workflow, such as raising a notification for all big Trades or creating a task to collect Trade patterns, then we have to burn our keyboards writing more if-else statements.

Did you also notice that the above TradesLoader is doing too much work, instead of just doing its basic job of loading Trades?

In order to reduce its burden, we have to refactor it so it will only load Trades. Its responsibility should end once the loading is finished and is decoupled from doing other processes.

So, how can we enhance the above TradesLoader to adapt to various scenarios that develop over a period of time?

One way is to use Messaging in our standalone application.

In this scenario, the TradesLoader fetches the Trades and posts them onto an internal data structure (a queue) before exiting. The relevant components, such as TradeProcessor, TradePersistor, and TradeNotifier will be working on their respective jobs to satisfy the workflow. They all can work at their own pace and not be bothered by the sequential processing anymore.