Hi, I am Gregor Hohpe, co-author of the book Enterprise Integration Patterns. I work on and write about asynchronous messaging systems, distributed architectures, and all sorts of enterprise
computing and architecture topics.
Find my posts on IT strategy, enterprise architecture, and digital transformation at ArchitectElevator.com.
Developing an architecture is a bit like a guessing game. You are anticipating future changes to the system and try to devise a system architecture that can (most likely) accommodate these changes. This is admittedly not the only purpose of architecture but certainly a major component of it. Well, are architects then essentially a bunch of fairground charlatans who suck money out of people's pockets for some form of "clairvoyant" trick performance just to have moved on to the next village by the time their scam is being revealed? (Rob Zombie's Living Dead Girl video comes to mind) I am sure Hani would concur after inserting the words "wanker", "bastard" and "$%^%@#". And some of you might feel that "finally we have a definition of architects that is based on real life!". OK, but you'll have to admit that some people are better guessers than others. For example, experience can make you a better guesser, ahem, architect and so can the ability to abstract problems, for example by using patterns.
One person was recently willing to watch me rub the crystal ball and proclaim what kind of variability and extensibility points their application, which is currently under development, should have. A lot of the themes we came up with relate to integrating with other applications, but this time we view them from the perspective of an application. Here is a brief list of items I came up with.
I am sure there are many more, but one of the well-known secrets of consulting is the "lingerie principle" -- never bare everything right away!
If your application models a portion of an overall business domain it is likely that the domain objects that you use are also represented by other applications, albeit with a slightly different emphasis. Since businesses without customers are rare (hey, it's not 1999 anymore!), the customer entity is a good candidate for an example. An order entry system might be most interested in the customer's preferences, while the shipping system is most interested in their shipping address, and the financial system might be interested in the customer's billing address and tax exemption code. This is all well as some of these additional fields are likely to live in related entities anyway (e.g., the address entity) and each application is allowed to have their version of customer and address . However, the users of your application are likely to want to see some of the extra data. The fact that the additional data fields reside in a different "application" will seem terribly irrelevant to the users if they have a customer on the line asking for just that information. The extra data itself can come from different sources: it can be entered by the user or it can be replicated from another application via Data Replication.
How can an application anticipate the need for these additional data elements? Let's look at a few options (see figure).
This option must be a close contender to "screen scraping" in the categories of wide-spread use and disdain. These are the kind of techniques that none of us use but that are to be found everywhere. All purpose fields are these "data1" varchar(255) fields that linger at the end of tables in case someone needs to stuff data in there that was not anticipated. This option certainly does not align well with the relational model but is unbeatable for convenience. Hey, I need to store the customer's pet's name, so there I go -- I have a varchar column to spare. Most systems will allow the display of this field on the UI screens, likely even with a custom label. Unfortunately, convenience often leads us down dangerous paths. In fact, the popularity of this option with early ("legacy") systems has shown us all too well where the path is likely to lead: before long you'll end up with a system where the first 3 characters of field "foo" contain some cryptic code. If that code is "bar" the next 5 characters are the person's birth date in hexadecimal except if they are born on a Tuesday...
Extension tables provide a little better separation between core application data and ancillary data. This approach suggests the existence of a CUSTOMER table and a CUSTOMER_Z (or whatever naming scheme you prefer), which is keyed to the customer table but is under the control of the administrator. This allows the addition of properly named and typed fields to be associated with a customer. The business logic layer would loop through this extra data to the UI so that it can be rendered. This approach can work reasonably well of you choose to (or have to) integrate with other applications by Maintaining Data Copies. As with the previous solution, the business logic layer typically provides a means of pulling the data through to the user interface without having to know much about the detailed composition of the data. The pros and cons of this approach are pretty well understood:
This option is a simple form of Portal Integration. Rather than extending the data model of the existing application, you create the user interface in such a way that additional application front-ends can be plugged in. Several portal standards, e.g. WSRP, simplify this task. One advantage of this approach is that the core application does not have to be modified internally. However, if close interactivity between the existing and the new data elements is required, the portal might provide a less than ideal user interface experience. Also, updates to both core application data and extension data are no longer under the scope of a single database transaction.
A more sophisticated approach along the same lines is a composite application. In this case, we assume that the user has to interact with a number of applications simultaneously and we provide a separate layer whose role it is to render a coherent user interface on top of a number of disparate back-end applications or service. While in Option (B) one application plays the primary role of the user interface and allows a few other items to the "plugged in", this approach separates the user interface completely from the different back-end systems. A composite application is often viewed as a key part of a service-oriented architecture where new applications are composed of existing services. This requires, however a solid services infrastructure and a good handle on how to expose your application's business functions as services (see below).
Sometimes you can achieve very simple integration and extension of the UI by using the ability of browsers to link to another application. This is essentially option (B) but without the ability to incorporate the new data into the existing screen. Instead, the front-end simply links to another user interface. This makes the user still work with two applications (i.e., there is no integration) but it makes it easier for the user to switch from one application to another. In a sense, this options is a more efficient version of "swivel chair integration". It also carries the same disadvantages, for example, the user might have to re-key information from one application to the next.
While there are already a lot of options for just adding simple fields to an existing domain object, sometimes an application requires the addition of entirely new domain objects or the interjection of new relationships or indirections into existing objects graphs. When dealing with a packaged application, these kind of changes are pretty hard to introduce without touching core application code and data. Some applications try to protect against this kind of scenario by employing a very generic domain model. This can work well to some extend, but you need to be careful that the application does not start to resemble "Microsoft Access", i.e. a generic database with a generic user interface builder. Yes, this kind of tool provides ultimate flexibility but does not bring much value out of the box. Once again, finding the correct compromise between specificity and extensibility is the secret.
The implementation choices essentially mimic the choices for the simple domain model extensibility. Either you need to be able to modify or extend the application core itself or you need to have a user interface that is flexible enough to have the separate applications appear as one.
So far, we only considered extending the data side of the model. More likely than not, some business logic will also depend on the new fields or entities. The need for new logic makes approaches such as All Purpose Fields or Extension Tables tougher because now you need to be able to add code to the application that deals with the new, generic fields. This no longer makes it possible to simply pull through generic data to the UI, but we need the ability to add new code modules that can process data values entered by the user for these new fields. IF a lot of logic crosses existing and new elements, a Composite Application approach starts to appear more likely.
A less intrusive way to extend an application is to allow other applications to receive relevant business events from your application. In the currently preferred style of service-oriented architecture or event-driven architecture, this ability is expected from any "enterprise class" application. For example, other applications should be able to find out about major activities happening in your application, for example, updates to entities such as the customer, new orders etc. The main implementation decisions for this feature are as follows:
In addition to publishing relevant business events, your application should also allow other applications to invoke relevant business functions provided by your application. This is the essence of a services layer that is accessible via a service bus or a similar communications backbone. Design decisions for this interface are similar to the ones above. For example, once again, you want to make sure your application provides metadata for the available functions.
Admittedly, this was a superfast sweep across extensibility requirements for an "enterprise class" application. One thing is for sure, though. Developing an enterprise application is as much about extensibility and integratability as it is about core application functionality. Not paying enough attention to these requirements can turn today's new application into tomorrow's silo. It is therefore critically important that you document specific extension scenarios (for example, "need to show an additional field in the user interface") and outline what kind of effort it takes to fulfill this use case. Will code changes be required? Will the vendor or the original developers have to get involved? Will such a change impact future upgrades? The only thing worse than not having an answer is to never ask the question!
Gregor is a cloud architect and author. He is a frequent speaker on asynchronous messaging,
IT strategy, and cloud.