Hi, I am Gregor Hohpe, co-author of the book Enterprise Integration Patterns. I work on and write about asynchronous messaging systems, distributed architectures, and all sorts of enterprise
computing and architecture topics.
Find my posts on IT strategy, enterprise architecture, and digital transformation at ArchitectElevator.com.
The beginning of the New Year is the time to reflect on the past and make resolutions for the future. As long as you make it a resolution to reflect more on the past, you can check off items with one stone. One of my resolutions was to write more regularly, so I am trying to start the year on the right foot by writing this rambling. Another resolution was to get myself a new laptop (the company issued model is all but unusable), so I am writing this on a Panasonic Let’s Note N8 running Japanese Windows 7. I am pouring water into the keyboard as we speak. Just kidding. The battery meter shows 8時10分 at 88%, so it looks like I can type along for quite some time.
It's been over 6 years now since Enterprise Integration Patterns first hit the shelves. The book continues to quite popular, so I am contemplating whether we could call it a “bestseller” soon. Actually, I have no idea what that label really means: I have always been equally amused and disturbed by the notion of a “bestselling author” or “bestsellers” in general. The exact criteria to call one's work a bestseller appear to be even less well understood than the Amazon Sales Rank algorithm. To my relief, Wikipedia states that "Bestsellers tend not to be books considered of superior academic value or literary quality." So maybe it’s a label we don’t actually want.
Technical book sales are certainly not what they used to be, and are by no means comparable to fiction titles. The Harry Potter series apparently topped 400 million copies and The Lord of the Rings (my teenage year favorite) logged 150 million copies. These days a technical book that sells more than 10,000 copies is probably considered a solid seller, whereas 50,000 copies constitute quite a success. At a minimum, the publisher is going to encourage you to write another one. With a little luck, EIP may hit the 50,000 mark this year, which would be a happy cause for celebration. It's not that we make that much money from it, but once you put this much effort into a book, you really do want people to read it. Write one and you'll know the feeling. Martin Fowler has aptly described the scary "will they like my baby" moment when a book hits the shelves. It's not like you can go back and fix it afterwards. It feels good if they like your baby.
What excites us the most is that the book has remained relevant over so many years. Sure, there are some "antique" TIBCO Integration Manager screen shots (maybe Mike Rettig will show these to his children some time, who will never understand why integration was difficult), and our code samples are refreshingly "generics free", but nevertheless quite generic. Of course the problems (and solutions) of asynchronous messaging have not changed terribly much in six years, making EIP as relevant as ever. It’s nice to see that EIP also still receives favorable reviews. What caught my attention in Mike Hadlow's blog post is that he originally got the book in 2004, but now actually applies it as he is working on messaging products. It's certainly nice to read statements about EIP like "I don’t think anyone can have an intelligent discussion about integration architecture without referencing it." Mike also understands well that one does not have to read the book cover to cover, but that it doubles up as a reference book. Jan Van Ryswyck agrees with him here, and points out that reading the 600+ pages back-to-back is quite an undertaking. Jan is aptly pointing out that the almost 50 page introduction contains many (hidden) gems about what messaging is, and why it works so well. In hindsight, we probably should have made it Chapter 1. This follows the advice from a smart fellow author (whose name escapes me), who solves the problem that people don't read introductions by renaming the introduction to "Chapter 1". Our introduction actually runs past page "li", i.e. lowercase roman 51, which is likely an indication that it would have made a good chapter.
I mentioned quite a few times before that we owe a lot of the success of EIP to the ESB community and particularly the open source implementations. Virtually all of them, including Mule, Apache Camel, Spring Integration, and OpenESB aka Project Fuji support our patterns out of the box and use our vocabulary to describe their products' capabilities. I even found a (short) paper comparing the implementation of EIP patterns in different open source ESB's. Lastly, seeing our patterns at the JavaOne keynote certainly ranks as one of my proud moments (as #2 behind Grady Booch listing EIP as one of his favorite pattern books at OOPSLA).
In his blog post, Jan also echoed the most common criticism about the book: the fact that it's not about the entirety of enterprise integration, but just about messaging. So the title is in fact a little misleading. The snip reply is to read the subtitle more carefully, which clearly hints at messaging. The real reason, though, is that when I was writing EIP with Bobby, I had something bigger in mind than a single volume.
Messaging is the foundation for many connected systems, but as many people have pointed out, it's only one part of a comprehensive integration solution. Messaging is great because it makes for an elegant interaction model which favors asynchronous, stateless solutions, and allows high throughput. As much as all those buzzwords make you think it's the best thing since sliced bread, messaging also has limitations. For example, the absence of state (beyond the position of the message) makes it difficult to discuss topics like error handling or conversations. Many readers probably noticed the "cliff hanger" pattern in our book, the Process Manager. We could not really hide the fact that this pattern is really a window into a different world, with design trade-offs rules, vocabulary, and patterns.
Process managers are often employed to track the exchange of multiple messages over time, i.e. a conversation. If you followed some of my (somewhat) recent publications in IEEE Internet Computing, SOA Expertenswissen, or EuroPLoP, or my Dagstuhl workshop report, you realize that conversation patterns have been the center of my attention for some time. Conversations are interesting to me because they are commonplace, both in connected systems as well as the real world. As Pat Helland discussed, multiple interacting systems engage in mini activities aka conversations to compensate for the absence of 2-phase-commit transactional guarantees. Since the real world also does not provide such guarantees, humans are very used to holding conversations, i.e. to exchange messages over time, to achieve an ultimate goal. For example, planning a dinner with your friends may involve a series of e-mail exchanges to pick a suitable time and location, and to inform everyone of the decision. Because these days e-mail is the asynchronous messaging infrastructure of choice for humans, many conversations are "hidden" behind threads of e-mail discussions. Not quite coincidentally, Google Wave is aiming to put the conversational aspect of e-mail exchanges into the center by choosing a document-oriented model of interaction. In some ideological universe we could call Google Wave is the blackboard architecture of asynchronous human interaction.
There is a lot more to say about conversations, but it becomes apparent fairly soon that designing and describing them is not an easy task. I am hoping that once again design patterns can help us tackle these design challenges by defining a consistent vocabulary and building a collection of design trade-offs and pitfalls. Enter EIP 2: Conversations between Loosely Coupled Systems. I concluded 4 years ago that interaction gets (and deserves) a lot of attention, so I think it's time we collect some guidance on how to do it right.
Conversation Patterns make a very natural extension to EIP I:Messaging, but the series does not have to stop there. Conversation participants almost invariably hold state, to track which messages were sent and received already, to count the number of retries, and so on. Osten, this state is held in a process engine of sorts, which expresses the conversation in form of orchestrations and process models. This is yet again a whole new world and probably deserves its own volume. You could consider it the dualistic (but different) view of conversation patterns.
But that's not it. CEP (Complex Event Processing) is getting more and more attention and I had a chance to connect with some very smart folks from the "CEP tribe" at EuroPLoP 2008 (man, why does everything seem ages ago?). CEP takes a bottom-up approach in the sense that instead of prescribing rules of conversations, the event processor observes a stream of events travelling the "event cloud" and interprets them to gain a higher-level understanding. A fascinating topic, which has been garnering vendor support, including an attractive open source engine, Esper.
Am I in all honesty proposing 4 volumes of Enterprise Integration Patterns? I guess so:
I think it'd be a valuable body of work, if I can only ever get around to writing it! The volumes would be in somewhat logical order from the bottom up with messaging being the foundation. One could also say the volumes follow the trend of basic EAI (messaging) to SOA (orchestration) and then SOA 2.0 (event processing). I feel a little dirty after typing SOA 2.0, so I won't get into the debate about "2.0" technologies.
With all these grand dreams, where does reality settle in? The EIP world domination plan stands about here:
So obviously I have some work ahead of me. Sadly, progress has been grinding to a halt for almost two years now, but I have not given up yet. Wish me luck!
Gregor is a cloud architect and author. He is a frequent speaker on asynchronous messaging,
IT strategy, and cloud.