Enterpise Integration Strategies
In present era no system in the world can be self sufficient to meet all business needs of its own. It needs to interact to number of interconnected systems to achieve business goals . For this interaction it needs to share data/information among those system, mainly we can divide this Interaction or integration strategies in the following categories :
- Shared File Systems
Two or more applications share a common file system; one may write to it while the other may poll the file system to read it. The sender and receiver are decoupled in this case. This solution certainly works, but has drawbacks like performance, reliability, and dependency on the File system. - Shared Database
In this strategy, applications share the data from a single database. One application writes data to the table while the other simply reads from table. The drawback is that this setup forces applications to use a unified schema throughout the organization. Shared databases also pose a few other issues, such as network lag and lock contention. - Remote Procedure Calls
If data or a process needs to be shared across an organization, one way to expose this functionality is through a remote service, for example an EJB or a SOAP/REST service allows functionality to be exposed to the rest of the enterprise. Using a service, the implementation is encapsulated, allowing the different applications to change the underlying implementation without affecting any integration solution as long as the service interface has not changed. The thing to remember about a service is that the integration is synchronous: both the client and the service must be available for integration to occur, and they must know about each other. - Messaging
This strategy mainly encourages and supports sender—receiver decoupling.,A sender application sends a piece of data enclosed in a message to a messaging middleman and forgets about it. A consumer consumes the message whenever it can and begins its own workflow. One of the advantages of using Messaging as the medium is that the sender and receiver are decoupled completely. These styles are disparate because no one solution will work all the time. To build a generic solution, middleware thinkers thought of a system based on these I patterns, typically called an Enterprise Service Bus (ESB). An ESB is the ultimate middle-man; it knows how to talk all languages, over all protocols, and mediate messages being passed. JEE and Spring played a key role in simplifying the enterprise programming model. JEE standardized and made commodities of solutions to common enterprise problems like database access, remote procedure invocation, transactions, authentication, directory services, etc. EAI solutions have no direct support in JEE, aside from the basics: RFC, messaging. There are many patterns for Enterprise Application Integration, and just as many protocols that need to be dealt with.
SEDA (Staged Event-Driven Architecture) - An Integration Pattern
Staged event-driven architecture (SEDA) is an approach to build a system that can support massive concurrency without incurring many of the issues involved with using a traditional thread and event based approach. The basic premise is to break the application logic into a series of stages connected by event queues. Each stage may be conditioned to handle increased load by increasing the number of threads for that stage (increasing concurrency).
Spring Integration
Why Spring Integration?
I found Spring Integration as the most interesting and best of Spring modules. I tried give a short introduction and a quick start here (a high level brief overview)
Spring Integration provides the ability to model ESB - style solutions using the same idioms and conveniences that we learnt from Spring framework
- Does not need a container or separate process space, can be invoked in existing program
- Very light footprint, it is just a JAR which can be dropped with WAR or standalone systems, Spring framework module, goes, easily with existing spring and java project
- Out of the box features:
- Threading support
Using XML based configurations we can easily set pool size, parallel execution etc. which would have been an extremely complex job otherwise. For , optimal performance of threading, polling interval and max messages per poll should be set appropriately. Suppose we have a requirement to process ~10k files in an hour, here thread pool executor can be implemented and configured to process items parallel. - JMX Support
One liner Spring XML based configuration exposes all of the channels on standard JMX console - STS Visual Editor
Eclipse provides excellent support for Spring Integration. In Eclipse market place we find a plugin named Spring Tool Suite (STS). It gives a beautiful visual editor which depicts the flow using structured diagrams.
At a high level Spring Integration is a complete implementation of Enterprise Integration Patterns developed on the top of Core Spring framework. Note that Spring Integration is NOT an integration of Spring with some application (generally when we hear "Spring Integration" it sounds like integrating Spring framework in some application or with some other framework), rather it is a very big module under Spring umbrella. It is a vast integration framework with lot of components and features.
It is an extension/module of Spring framework to support well-known Enterprise Integration Patterns. Spring Integration enables lightweight messaging within Spring based applications and supports integration with external systems via declarative adapters. Spring Integration's primary goal is to provide a simple model for building enterprise integration solutions while maintaining the separation of concerns that is essential for producing maintainable, testable code.
It's a messaging framework that supports lightweight, event—driven interactions within an application as well as adapter based platform that supports easy and quick integration of enterprise applications.
Framework handles the complete routing of actual data from sources to where the business logic should be executed; it directs the message to the appropriate component and where the response should be sent-providing support for message routing and filtering. In addition. Spring Integration provides message transformation support so that the resultant application will be agnostic to the underlying message format and transport. Spring Integration follows the same configuration scenarios as the Spring Framework, providing options for configuration through annotations, Java configuration, XML with namespace support, XML with the "standard" bean element, and direct access to the underlying API.
Spring Integration takes care of the fundamental Spring principle of single-focused POJO beans in complex integrations. There is rarely any need to couple our code to the Spring Integration API. No matter how much complexity of code/logic it carries, still it will be a Spring based POJO/bean.
As per Spring documentation Spring Integration has been designed considering following goals/principles:
Key Goals
- Provide a simple model for implementing complex enterprise integration solutions.
- Facilitate asynchronous message-driven within Spring based application.
- Promote intuitive incremental adoption for existing Spring users
Guiding Principles
- Components should be loosely coupled for modularity and testability.
- The framework should enforce separation of concerns between business logic and integration logic.
- Extension points should be abstract in nature but within well-defined boundaries to promote reuse and portability.
Key Components
- Message
In Spring Integration, a Message is a generic wrapper for any Java object combined with meta-data used by the framework while handling that object. It consists of a payload and headers. Here Java object is the payload property, and the meta-data is stored as a collection of message headers, which is in essence a Map with a and an Object value. - Headers
Headers object is a String/Object Map that typically maintains values for message housekeeping. Message headers are immutable and are usually created using the MessageBuilder API. There are a number of predefined entries (headers), including id, time-stamp, correlation id, and priority. In addition, the headers can maintain values required by the adapter endpoints (e.g., the file name for the file adapter or the email address for the mail adapter) . Headers may be used any key/value pair required by the developer. - Payload
The message payload can be any POJO. Spring Integration does not require message payloads to be in some sort of predefined format (i.e. no implementation or extension is needed for an object to act as payload) . Although we can use transformers to convert any payload into any type of format as required by the message endpoint. - Message Channel
A Message Channel represents the "pipe" of a pipes-and-fliters architecture. Producers send Messages to a channel, and consumers receive Messaged a channel. The Message Channel therefore decouples the messaging components, and also provides a convenient point for interception and monitoring Messages. The channel implementation manages the details of how and where a message is delivered but shouldn't need to interact with the payload of a message.
Two endpoints can exchange messages only if they are connected through a channel. The most important characteristic of any channel is that it logically decouples producers from consumers. Message channels are the key enabler for loose coupling, both the sender and receiver can be completely unaware of each other. - Publish-Subscribe Channel
A Message Channel may follow either Point-to-Point or Publish/Subscribe semantics. With a Point-to-Point channel, at most one consumer can receive each Message sent to the channel. Publish/Subscribe channels, on the other hand, will attempt to broadcast each message to all of its subscribers. Spring Integration supports both of these. - Point-to-Point
Whereas "Point-to-Point" and "Publish/Subscribe" define the two options for how many consumers will ultimately receive each Message, there is another important consideration: should the channel buffer messages? in Spring Integration, Pollable Channels are capable of buffering Messages within a queue. The advantage of buffering is that it allows for throttling the inbound Messages and thereby prevents overloading a consumer. - Pollable Channels Pollable Channels are capable of buffering Messages within a queue. The advantage of buffering is that it allows for throttling the inbound Messages and thereby prevents overloading a consumer.
- Subscribable Channels On the other hand, a consumer connected to a Subscribable Channel is simply Message-driven.
- Message Endpoint
A message endpoint is the abstraction layer between the application code and the messaging framework. An endpoint broadly defines all the types of components used in Spring Integration. It handles such tasks as producing and consuming messages, as well as interfacing with application code, external services, and applications. When data travels through a Spring Integration solution, it moves along channels from one endpoint to another, life should be able to focus on your specific domain model with an implementation based on plain Objects. Then, by providing declarative configuration, can "connect" your domain-specific code to the messaging infrastructure provided by Spring Integration. The components responsible for these connections are Message Endpoints.
Message endpoints have following key characteristics: - Polling or event-driven
- Inbound or outbound
- Unidirectional or bidirectional
- Internal or external
-
Transformer
Converts the message content or structure. It is responsible for converting content or structure of the message and returning the modified message. e.g. converting XML and file based payloads to Domain objects. -
Filter
Determines if the message should be passed to the message channel, all. This simply requires a boolean test method that may check for header etc. e.g. dropping all messages which do not meet the minimum acceptance criteria -
Router
Can determine which channel to sent particular message based on its contents. It is responsible for deciding which channel or channels should receive the message next (if any). e.g. routing a message on the basis of contents; another usecase can be thought of, if an XML is parsed successfully will be passed to the next channel in the flow for further processing otherwise will be sent to retry channel. - Splitter
Can break an incoming message into multiple messages and send them to the appropriate channel. It is another type of message endpoint responsible for accepting a message from its input channel, split that Message into multiple messages, and then send each of those to output channel. e.g. after fetching list of files from a remote FTP server, list of files is put on a splitter, which then puts it on a gateway - one by one so that each file can be fetched. - Aggregator
Can combine multiple messages into one. An aggregator is more complex than a splitter often required to maintain state. It is almost inverse of the splitter. As the name suggests it is a type of endpoint that receives multiple messages and combines them into a single message. Spring Integration provides a completion strategy as well as configurable settings for timeout, whether to send partial results upon timeout, and the discard channel. e.g. we initiate parallel web-service (SOAP/REST) requests to downstream systems and want to wait for response of all the messages. Here aggregator is used to wait for all the downstream systems to respond back (when all responses available for all the requested services) and collect responses together - correlation strategy decides if all responses are available or not. Once available it is passed on for further processing. - Service Activator
It the interface between the message channel and a service instance, many times containing the application code for business logic. It is a generic endpoint for connecting a service instance to the incoming messages . The input Message Channel must be configured, and if the service method to be invoked is capable of returning a value, an output message channel may also be provided. The output channel is optional, since each message may also provide its own 'Return Address' header. e.g. for application logic like persisting some data to database. Generally it contains pure business specific logic. - Adapter
Is used to connect the message channel to another system or transport. It is an endpoint that connects a message channel to some other system or transport. Adapters may be either inbound or outbound. An inbound adapter endpoint Connects a source: system to a message channel. An outbound adapter endpoint connects a message channel to a target system.
 |
| Spring Integration - Adapter Flow |
-
All of above components play important role in an enterprise integration system. Let's start with a very basic setup and maven based example (for the flow shown in above figure).
We will see a running example of Spring Integration in next post
