Understanding ESB

Question

though I understand what system integration is, I am a bit new to all the newest approaches. I am farily familiar with web services and JMS but I feel utterly confused by the concept of an ESB.

I have done some research but I still don't really get it. I work much better by example rather than theory.

So can someone please illustrate a simplistic example to demonstrate why one would use an Enterprise Service Bus vs just a Queue , a web service , the file system or else?

I would like the example to amplify the capabilities of the ESB which could not be achieved by any other conventional intgration method or at least not with the same efficiency.

All replies are greatly appreciated.

Thanks, Bob

Answer 1

This is going to sound a bit harsh, but basically if you needed an ESB, you'd know you needed an ESB.

For a majority of use cases, the ESB is a solution looking for a problem. It's a stack of software over engineered for most scenarios. Most folks simply do not do enough variety of processing to warrant it. The "E" for "Enterprise" is notable here.

In a simple case:

tail -F server.log | grep SEVERE >> severe.log

THAT is a trivial example of an instance of an ESB scenario.

"But that's just a UNIX command pipeline!"

Yes, exactly.

The "ESB" part is the "|" and the ">>"

The ESB is the run time within which you can link together modules, monitor traffic, design all sorts of whacky scenarios like fan outs and joins, etc. etc.

ESBs are notable for having a bunch of connectors to read a bunch of sources and write a bunch of destinations. They're notable for weaving more complicated graphs and workflows for processing using rather coarse logic blocks.

But what most folks typically do is:

input -> DO_STUFF -> output

With an ESB they get:

ESB[input -> DO_STUFF -> output]

In the wild, most pipelines simply are not that complicated. They tend to have one off logic that's not reusable, and folks tend to glob it together in to a single logic module.

Well, heck, you can do that with a Perl script.

Long pipelines in ESBs tend to be more inefficient than not. With lots of marshaling of data in to and out of generic modules (since you rarely use a binary payload).

So, say, CSV comes in, converts to XML, process it, output XML for input to another step as XML, which marshals it, works on it, converts it back in to XML for Yet Another step. Rinse and repeat until the CPU hits 400% (multi-core FTW).

Then some one comes up with "Hey, if I drag an drop these modules together in to a single routine, we skip all this XML junk!", and you end up with "input -> DO_STUFF -> output".

For large systems, with lots of web services that need to do casual, ad hoc integration, they can be fine. If you're in a business that does that a lot, they can work really well. When you have dozens of pipelines, they can help with the operational aspect of managing them.

But for complicated pipelines, if you have a lot of steps, maybe it's not such a good idea beyond prototyping, especially if there's any real volume involved. Mind, you may not having any choice, depends on the systems you're integrating.

If not, if you have a single interface you need to stand up -- then just do it. Do it in Perl, in Java, in C#, in whatever. Don't run out and spool up some odd 100MBs of infrastructure and complexity that you now to get to learn, master, and maintain.

So, again, if you needed an ESB, you'd know it. Really. You'd have whatever system you've built together of disparate stuff, you'd be fighting it, talking to colleagues about what a pain all this stuff is, and you'd stumble across some link to some site to some vendor and read a white paper and go "THAT'S IT!", but if you haven't done that yet, then you're not missing anything.

Answer 2

ESB is for the cases where you do have that web service and queue and file system all in the same system and need to integrate them. An ESB product usually solves the following

1. Security
2. Message routing
3. Orchestration (which is advanced message routing)
4. Protocol transformation
5. Message transformation
6. Monitoring
7. Eventing

You can do all of these with other tools as well and if you just need one or two of these capabilities you can probably do without and ESB (as it introduced additional complexity) but when you need several of them an integrated solution in the form of an ESB can be a better solution.

Answer 3

As @WillHartung concluded, ESBs tend to be properly used in large, complex situations. And that's why it's named Enterprise Service Bus.

Now, to actually answer your question, ESBs typically:

• Communicate over several protocols (e.g. HTTP, Message Queue, etc.), for both input and output

• Establish a common message format, and often translate from other formats into the 'canonical' format

• Provide endpoint transparency (e.g. you send a message to the bus, and get an answer back, but you don't explicitly know what service, also connected to the bus, handled your request.

• Provide monitoring and management capabilities

• Facilitate versioning of services and messages

• Enforce security, when needed.

So, as you can see, it's for when doing a lot of point-to-point communication ("just do it") would be a huge, unmanageable pile of spaghetti. Indeed, most places that I've seen SOA implemented, it's replacing that huge pile of spaghetti that already exists.

Answer 4

An ESB is an enterprise service bus, an infrastructure backplane if you like for a service-oriented architecture. Imagine the chaos of hundreds of services happily reusing each other. How do manage such an environment? How do you provide flexible, decoupled routing between your services? How do you avoid point-to-point spaghetti architecture? How do you manage transactions and security across a hybrid technology landscape? How do you track where messages are in complex flows across multiple systems?

You use an ESB.

ESBs typically allow you to design flows across multiple systems in an XML configuration language, offering you a host of EIS adaptors, transformation and mediation plugins etc. Some will offer an IDE to help you design flows. Some ESBs are very expensive, some are open source.

If you want to get a feel for ESB, check out either Mule or WSO2, both good open source products, or even Spring Integration which is a non-clustered solution but excellent for decoupling Java from the underlying external interface points.