Priority Messages in Distributed Systems
Distributed systems might use message queues or message streams for asynchronous communication between services. System designers can choose from many products, taking into account specific features, like message prioritization. In this article, I will focus on how to design said prioritization in virtually all message systems.
Message Systems
Simply put, message systems allow producers to send messages to consumers using brokers. Messages usually contain:
- data (a sequence of bytes),
- metadata (e.g., creation timestamp, creator information, or transmission-control information).
Systems that support topics (channels, buckets) require producers to declare the intended destination in message metadata.
A broker receives a message and then makes it available to consumers.
Or not…
Depending on a system, a broker might give way for more urgent messages first or it can hold the message back based on its metadata. You might think that brokers decide when messages reach consumers but there happens more than meets the eye.
All consumers in a system might operate under a heavy load, all of them may become unavailable due to a network failure or they all reject messages because of a bug introduced in the latest deployment. There remains the cold hard truth about any distributed system — a lot of possible points of failure always exist. Even relaying data between two components utilizing a proxy provides no guarantee of completion.
Native Message Prioritization
I subtly hinted before that message systems might not support prioritization natively. For it to work, all 3 component types (producers, brokers, and consumers) need to work aligned. The actual implementation depends on the chosen solution.
Architectures that support prioritization ad hoc cannot provide reliable ordering of messages. I find it evident in event-based and command-based systems, where processing messages in arbitrary order might introduce tremendous problems. Maintaining prioritization might indicate that some data can disappear due to a lack of natural order of messages.
Custom Message Prioritization
Other architectures task system designers to tackle prioritization on their own. How? By devising a priority-aware layer on top of the chosen message system.
Priority topics
Systems that use topics (sometimes called channels or buckets) come to our advantage. We can split topics into subtopics, each responsible for a different priority. For instance, I can replace the topic called orders with orders-0, orders-1, and orders-2 (please note I did not use the term priority here on purpose!).
Producers will send most data to orders-1 while emitting hasty messages to orders-0; orders-2 shall receive the least significant information. Designers should allot enough consumers to process the load from three different topics.
The aforementioned solution has several issues. Firstly, producers and consumers alike need to acknowledge the existence of additional topics. Secondly, producers have to choose the topic before posting a message. Thirdly, the topic division does not account for different load scenarios.
Priority metadata
Instead of leveraging multiple topics for prioritization, we might pass priority hints in message metadata. Such an approach allows brokers and consumers to move messages in the execution queue, if feasible at a particular moment. I have created a TypeScript project for demonstrating this method, available here.
The Priority Message System permits multiple producers and consumers to exist, but just one broker. Messages flow from producers to the broker, ordered by their arrival timestamp. The Broker transmits messages to consumers in constant intervals; only one consumer can obtain a particular message.
A crucial finding — consumers acting as producers — allows adding messages back to the queue. I will use it to achieve prioritization, using consumer strategies. I have prepared three of them as an example.
The Predicate-based Consumer Strategy
The predicate-based strategy will run a function to tell if a message requires processing now or later. If it does not, the consumer will send it back to the queue and increment the number of predicate executions. If the aforesaid number has increased to an arbitrary maximum, the message will get processed anyway.
The predicate might not only take into account the message data, but also the system state. This feature enables the consumers to adapt to the current load. As a failsafe, all messages get eventually processed after a couple of cycles.
The Priority-based Consumer Strategy
The priority-based strategy enforces a priority threshold, below which no processing occurs. Different consumers might operate employing various thresholds. Changes in the system environment might impact the numbers as well.
The strategy compensates messages with lower priorities by incrementing that value before shipping them back to the broker. This ensures the successful processing of each message.
The TTL-based Consumer Strategy
The TTL-based strategy deals with messages in a specific time-to-live range. Consumers will discard low-TTL messages and decrement the TTL of high-TTL messages. Similar to the preceding examples, the acceptance range might change in the runtime.
The strategy might remove data from the system if the administrators modify the TTL range in the wrong way. I will leave it to you to figure out potential failure cases. I added this concept to illustrate how simple ideas can become problematic when used carelessly.
You can use distinct strategies interchangeably with caution. I always recommend using tests first before applying changes to production systems.
Summary
I believe that understanding the limitations of the system and finding ways around them serves as the key takeaway from the article. Despite some message systems not supporting message prioritization natively, we can always build it on top of the existing technology. If you have other ideas about adding priority handling, please leave a comment in the comment section!
