Enterprise Service Buses (ESBs) Drive SOA Adoption Part 6 - The ESB Channel Model
In the last article, we started to implement messaging applications, specifically Message Routers. We first turned, as usual to the Hohpe and Woolf book to look at the Message Router pattern. We then looked at variants of routers. I then spent some time looking at what is involved in implementing routers with raw WCF. Then I showed you that it is built into Neuron and we needed two lines of code to do it.
This time around, I am going to step back and cover some more Neuron essentials that we will need to understand before we continue to build applications. In Part 4 of the series, I discussed the importance of topics in Neuron and how each topic is implemented as an individual publish-subscribe network, called a topic network. Topic networks are based on a modular channel model and can be powered by technologies ranging from WCF peer networking to BizTalk Server. In this installment, we’ll look at the channel model in more detail.
ESBs have endpoint architecture, meaning there is a common stack all endpoints go through in order to interact with each other. This is conceptually no different than using a WCF client to talk to a WCF service: a common stack on both ends of the connection work together to facilitate communication and cooperative processing. The difference with the ESB stack is the functionality provided, which includes publish-subscribe communication to N parties and the ability to layer integration functionality (transformations, rules, workflow) into your messaging at any point.
How does the ESB stack pull off the publish-subscribe communication between N parties? You might assume all those ESB stacks simply connect to an ESB Server in a hub-and-spoke arrangement, but that’s not the case at all. Or more accurately, it’s up to the channel. A channel is simply a software module that implements publish-subscribe using some base technology. The four channel types included with Neuron implement pub-sub over peer networking, WCF TCP, MSMQ, and BizTalk Server. Each of the channels provide the same functionality but with varying qualities of service. Since each topic is a separate channel choice, you are free to run different qualities of service side by side. You might, for example, favor speed for some business conversations but require reliability for others.
Although the channel’s responsibility is to implement publish-subscribe, applications are not limited to that one messaging pattern. Neuron’s stack also allows request-reply, direct point-to-point connections, and itinerary routing (where the message determines the delivery points). All of these patterns are always available, regardless of which channel you use.
All publish-subscribe systems have to deal with backlog, where published messages come in faster than they can be delivered. A channel’s implementation must provide a strategy for handling backlog.
Peer Networking Channel
Peer Channel uses Windows peer-to-peer networking. In WCF, this technology is packaged up as the NetPeerTcpBinding. With Peer Channel you create a cloud or mesh of messaging nodes. The nodes are interconnected using both TCP and UDP, and will also take advantage of IPv6 on the network if they are able to. Any message sent by a node is delivered to all others nodes on the mesh in a highly parallel fashion that doesn’t require server infrastructure. A mesh is self-organizing, changing its shape in response to nodes joining/leaving and activity. This automatic optimization keeps the number of hops to a minimum and slower nodes are moved to the outside of the mesh.
Peer networking often brings to mind consumer applications such as instant messaging, music and file sharing, or multi-player gaming. However, there are enterprise scenarios where peer networking is ideal. Its great strengths are its high speed, its parallelism, and its scalability.
In Neuron, each topic powered by Peer Channel is a separate mesh. The behavior of a Peer Channel mesh is a natural fit for publish-subscribe messaging. Neuron deployments have been known to operate meshes with as many as 1500 nodes that stretch across the U.S. over multiple geographic locations.
There’s a catch with peer channel, however. The base technology is not reliable enough to suit most enterprises. There are no message delivery assurances: you can lose messages (sometimes large blocks of messages if there’s a lot of CPU contention) nor are messages guaranteed to arrive in order. The good news is that Neuron ESB provides an optional reliability layer that provides delivery assurances through message caching.
In its native form, peer channel is server-less. In fact, you can shut down all of your servers and parties can continue to message. Adding in the reliability layer or enabling certain features such as auditing will require you to keep at least one server up and running however.
The peer networking channel is volatile, meaning there is no durable storage of messages. Performance is very good, but is limited by the slowest node in the mesh.
Peer channel is best suited for scenarios requiring low-latency (high speed) communication and/or the need to broadcast to many recipients simultaneously.
WCF TCP Channel
The WCF TCP channel interconnects parties using WCF TCP with reliable messaging. The channel connects parties to an ESB server via TCP. A massively efficient TCP publishing engine on the server copies incoming published messages out to connected subscribers with matching subscriptions.
The server publishing engine can become backlogged, where published messages are coming in faster than they can be relayed out. The engine will buffer backlogged messages up to a threshold; once that threshold is exceeded, flow control kicks in and publishers will be slowed down as needed.
In a server farm scenario, each server will have a different collection of parties connected to it. The servers share incoming messages with each other laterally so that any incoming messages are relayed to all servers for distribution.
The WCF TCP channel is volatile, meaning there is no durable storage of messages. There are delivery assurances within the scope of a connection, but sudden client or server loss can lose messages.
The TCP channel is a good choice for applications that don’t require durable or transacted messaging.
MSMQ Channel
The MSMQ channel implements publish-subscribe over queues. Messaging is durable, transacted, and reliable. There is a publish queue for a topic, and a recipient queue for each subscriber. An MSMQ publishing engine on the server moves messages between queues.
Parties do not have to be online in order to receive messages from each other because undelivered messages persist in the queues. This characteristic allows you to take systems offline and online without being disruptive.
The MSMQ queues used by the channel are durable, providing reliable store-and-forward messaging. In a backlog situation, messages that cannot be immediately processed accrue in publisher or subscriber queues until they can be processed.
The server’s MSMQ publishing engine is transactional. A delivery either fully completes or the queue reads and writes are rolled back. The engine processes messages in batches when it can for efficiency.
The MSMQ channel is a good choice for transacted, durable communication.
BizTalk TCP Channel
The BizTalk TCP channel performs publish-subscribe over BizTalk Server. The channel connects parties to BizTalk Server via TCP, and BizTalk’s messaging engine performs the publish-subscribe messaging.
Making changes to the Neuron ESB configuration automatically programs BizTalk Server to make matching changes to its configuration. For example, defining new publishers and subscribers in Neuron will cause new send ports and receive ports to be created in BizTalk.
With BizTalk as the messaging engine, you gain the many features of BizTalk including strong retry/recovery capabilities. With a small amount of BizTalk configuration you can link messages to features such as Business Activity Monitoring (BAM),, transformations , the rules engine, and orchestrations.
The BizTalk channel is durable and is well-suited for mission-critical applications that require high reliability and/or the unique characteristics and features of BizTalk Server’s. It is not the best choice for low-latency applications, as BizTalk’s message box database limits performance.
What about Services, Legacy Systems, and Line of Business Apps?
Does this mean there are only 4 protocols you can use to communicate with in Neuron? Not at all--the endpoint architecture and channel model we’ve been describing is how Neuron makes publish-subscribe happen, but this is the internal side of the bus. Externally, you can connect a wide, wide variety of services and non-services to the bus… everything from WCF services to WebSphereMQ. We will talk about the external side of the bus in future articles.
Where are We?
The Neuron ESB channel model provides a modular system for switching in different technologies for messaging. This not only makes it possible to leverage a wide range of existing technologies, but anticipates new technologies and provides a path for accommodating them.
Now that we have talked about the internal side of the bus, it’s time to go back to the external side of the bus, with the concept of Endpoints
