RESTful Conversations

Find my posts on IT strategy, enterprise architecture, and digital transformation at ArchitectElevator.com.

As indicated a good while ago I spent some time thinking about patterns that instead of following a message through multiple systems, looks at the message exchange over time between a (mostly) fixed set of systems. I call these message exchanges "conversations", partly because it avoids collision with terms like "orchestration" or "choreography" and partly because it makes for a nice bridge between communicating systems and communicating humans. "interactions" would have been another good name, but I found it a tad too generic, even though I did register interactionpatterns.com.

Vocabulary aside, my writing efforts on EIP Vol 2 fell very much asleep around 2008, first due to lack of time, but then also to my fear that with the rise of REST, stateful conversations between systems would step into the background as most integration problems are now solved with a simple POST or GET. A presentation and conversation at Stuttgart University's IAAS (not IaaS!) PhD Seminar setup by Frank Leymann enlightened me that this is not at all true: systems following the REST architectural style very much engage in conversations.

To REST or not to REST

I am probably the last person to admonish everyone that neither using XML or JSON nor HTTP verbs alone makes an interface "ReSTful" in the intended sense. First up, REST is an architecture style, not a specific technology or encoding. Mary Shaw and Dave Garlan defined an architectural style as "defining a vocabulary of components and connector types, and a set of constraints on how they can be combined". [side note: in my ranking of proudest moments, being listed as a SATURN 2015 keynote speaker alongside Mary Shaw vies for 1st place with Grady Booch's mention of our book at OOPSLA 2005]. Second, ReSTfulness is not a binary property. Leonard Richardson's REST Maturity Model describes multiple levels of maturity that make an API actually ReSTful. Martin Fowler wrote a nice explanation of this model, including a halo to be achieved in the REST Nirvana, the mismatching religious metaphors being my contribution. Not quite coincidentally, simply sending POX over HTTP gets you only maturity level 0, the lowest possible one.

Application State

The connection between REST and conversations starts at level 3 of the model, where we encounter what must be one of the worse acronyms in recent IT history: HATEOAS, which is supposed to stand for "Hypertext as the Engine of Application State". I could not actually make out who blessed us with this acronymial masterpiece -- it does not appear in Roy Fielding's thesis or blog. If it was up to me, I would change its meaning to Hideous Acronym That Elaborates On Application State.

The unlucky name aside, the idea behind level 3 of the maturity model is that application state is expressed through hypermedia (read URI's). In reality this means that a service response describes possible follow-on on resources to acted upon by the client in form of embedded URI's, aka "links". Essentially, this set of URI's defines the legal state transitions of the application (and thus the allowed states) to the client as Roy Fielding explained in a somewhat emotional post from 2008: "From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user's manipulation of those representations". As the application state transitions drive the conversation a client can engage in, this representation of application state translates fairly directly into a description of allowed conversations. Compared to other ways of describing conversations, such as choreography or orchestration, describing conversations with links embedded in response messages is a dynamic and relatively loosely coupled mechanism. Speaking of loose coupling, my favorite quote from Roy's post is "There is so much coupling on display that it should be given an X rating" -- I'll make sure to borrow that some time.

Putting the idea of embedding application state transitions in URI's to practice, let's draw on Jim Webber's, Savas Parastatidis' and Ian Robinson's RestBucks example, which was in turn inspired by a popular rambling of mine -- it's a small world! In their example, once you placed a drink order to RestBucks via an HTTP POST, the service will supply you with a response similar to this:

201 Created
Location: http://starbucks.example.org/order/1234
Content-Type: application/xml

<order xmlns="http://starbucks.example.org">
  <drink>latte</drink>
  <cost>3.0</cost>
  <next xmlns="http://example.org/state-machine"
    rel="http://starbucks.example.org/payment"
    uri=" http://starbucks.example.org/payment/order/1234"
    type="application/xml"/>
</order>

After the customer has just ordered a drink the next action he or she can take is to submit a payment. Correspondingly, a URI for the payment resource is included in the response message under the "next" element. The semantics of "next" URI's are indicated by the "rel" attribute http://starbucks.example.org/payment, meaning client and server have to share a common vocabulary and semantics of resources such as "payment". The set of the resources and associated actions is also the vocabulary for the conversation to be had: you can place an order, make a payment etc. The links do not show, though, that you can also potentially update your order -- this is handled by the HTTP protocol via the OPTIONS verb or the expect header.

Conversation State

The inclusion of "next" links in responses defines the set of valid conversations from the perspective of the service provider. Unlike choreography and orchestration approaches, the conversations are dynamically defined at the time of interaction as opposed to an a priori definition. This approach reduces coupling (and thus avoids an X rating), however, it does not allow the upfront creation of client workflows that model allowed conversations. Instead, the client has to start interacting with the service and see what it is being offered to do next. For example, constraints that define valid conversations, such as whether a customer can pick up a drink before payment is made or whether the drink is started to be made before a payment is received, are represented as a set of links embedded in multiple response messages and not known upfront. This means that clients cannot make strong assumptions on message order -- the service is in the driver's seat. Once again, reduced coupling means fewer assumptions, but you (or your code) have to be more flexible, which can bring complexity or at least a different way of thinking about process flow. If (slightly) stronger coupling is allowed, a REST service could describe legal transitions upfront through a special media type. The transitions are then mapped to concrete resources via the URI's embedded in next elements.

This relatively simple example also makes it nice and clear that REST services are indeed engaged in conversations just like traditional web services, and maybe even more so because the individual interactions tend to be simpler, causing services and clients to engage in longer conversations to accomplish their ultimate goal.

Conversation Patterns for REST Services

Having learned that REST services engage in conversations we should hope that some of the Conversation Patterns I started to document a while ago still have relevance. In fact, one can assume that many new patterns are waiting to be "mined" from the magnitude of REST services out there, from OpenStack to the Amazon API's. This should motivate me to pick the topic of conversation patterns back up and make a push towards EIP 2.

Florian Haupt, a PhD student at IAAS Stuttgart presented conversation pattern candidates at the PhD Seminar and documented them in his upcoming paper A conversation based approach for modeling REST APIs, which he is presenting at WICSA 2015 in May. His interest in conversations stems from his work in model-driven development for REST services. Modeling languages are particularly useful if they include higher-level constructs for frequently recurring solutions. Such solutions are not patterns per se, but could be an indicator that there are related patterns to be harvested. His conversational constructs originated from analyzing REST services, but, like all good patterns, are not limited to REST services alone.

Florian shared the example of a long-running tasks, which in the REST world defines three resources: a manager, a task, and a result for the client to interact with. This could well be an implementation strategy to a polling or callback conversational pattern. A second example, a Try-Confirm-Cancel sequence, is indeed a common conversation pattern that is as useful with REST services as with any distributed system that lacks global transactions. In my paper on Conversation Patterns I deal with these conversations under the chapter "Reaching Agreement", which is still incomplete.

Conversation Patterns Quo Vadis?

The realization that conversation patterns are as relevant as ever begs the questions where I am with EIP 2: Conversation Patterns. I did post the paper I workshopped at EuroPLoP 2008 on-line, but I think I really need two things: a good vehicle to share work in progress and a community to provide input and feedback to give me energy. For the former, you may have noticed small changes to the Web site's navigation and the renaming of the pattern catalog to "Messaging Patterns" in preparation of making room for another, work-in-progress, catalog. Alas, the site is still being built by Ant 1.7 and XSLT, which may soon find a place in the Computer History Museum, but is not a great tool for content management or collaboration. For EIP we used a simple mailing list and archives for discussion because our Wiki was flooded by spam. Maybe mailing lists are becoming "retro" and actually work? There is no writing on conversation patterns without engaging in conversations!

Join my Team

Our team is still growing! I am looking for one more senior Cloud Infrastructure Architect to join my team at Allianz in Munich.