 |
An application is using Messaging. However, it cannot process messages as fast as they’re being added to the channel.
How can a messaging client process multiple messages concurrently?
Create multiple Competing Consumers on a single channel so that the consumers can process multiple messages concurrently.
Competing Consumers are multiple consumers that are all created to receive messages from a single Point-to-Point Channel. When the channel delivers a message, any of the consumers could potentially receive it. The messaging system's implementation determines which consumer actually receives the message, but in effect the consumers compete with each other to be the receiver. Once a consumer receives a message, it can delegate to the rest of its application to help process the message. (This solution only works with Point-to-Point Channels; multiple consumers on a Publish-Subscribe Channel just create more copies of each message.)
Competing Consumers Sequence
... Read the entire pattern in the book Enterprise Integration Patterns
Example: Apache KafkaNEW
Apache Kafka is a distributed streaming platform that allows clients to consume messages in both Publish-Subscribe Channel and Competing Consumers semantics. Kafka is optimized for high throughout and horizontal scalability and therefore tries to avoid the overhead that can be inherent in coordinating across multiple Competing Consumers. So while the client interface allows multiple consumers to consume messages off a topic in a competing fashion, guaranteeing that each message is only consumed by one of the consumers, the implementation actually relies on multiple partitions implemented as Point-to-Point Channels:
In the example above, a Kafka topic consists of three partitions divided across two consumers. A Message Router distributes the messages across the three Point-to-Point Channels. This router is generally a Content-Based Router that works off the message key. This allows the sender to control the distribution of messages, e.g. to assure related messages go onto the same partition. This can be necessary to preserve order because in-order delivery is not guaranteed across partitions.
This design implies that the decision which consumer receives the message is made a priori, meaning consumers don't actually compete but rather read from independent channels. This limits the ability to load balance between consumers, e.g. if one channel runs empty, the corresponding consumer won't be able to aid other consumers whose channels still have messages. Also, as the number of partitions is set independently, not all consumers will service the same number of partitions, as illustrated in the example above.
However, as Kafka partitions are relatively inexpensive, there are usually lots of them, meaning that each consumer is in reality likely to read of many partitions and therefore channels. This highlights another trade-off: there can't be more competing consumers in a consumer group than there are partitions on that topic.
Related patterns:
Content-Based Router, Event-Driven Consumer, Message, Message Channel, Message Dispatcher, Message Router, Messaging, Point-to-Point Channel, Polling Consumer, Publish-Subscribe Channel, Transactional Client
Integration Pattern Language
