Broadcasting Messages with Apache Kafka and .NET

Download full source code.

Apache Kafka is often used to deliver a message to a single instance of an application or to deliver copies of a message to multiple different applications. But it can also broadcast messages to all instances of a single application.

Why would you want to broadcast messages to all instances of an application? I can think of a few reasons -

• You want to update all instances of an application with the same data.
• You want to tell all instances of an application to perform some action.
• You want to wake up all instances of an application from some sleep state.
• You want to let all instances of an application know that something has happened elsewhere in your system.

With a broadcast message, you don’t need to worry about coordinating between instances of an application.

Competing Consumer vs Publish-Subscribe in Kafka

To keep the text easy, when I say consumer and consumers I mean a single or multiple instances of the same application. I am not talking about different applications acting as consumers.

The most common use case for Kafka I have seen is to have a producer and lots of consumers. Each consumer reads a message from a topic and processes it. But only a single consumer will read a particular message. This is called a “competing consumer” pattern. Whoever reads the message first gets to process it. To follow this pattern, every consumer must be in the same consumer group.

But what if you want to broadcast a message to multiple consumers? This is called a “publish-subscribe” pattern. In this pattern, every consumer gets a copy of the message. To follow this pattern, every consumer must be in a different consumer group.

Kafka has some complications around partitions in the case of of the “competing consumer”. But in the “publish-subscribe” pattern, you don’t need to worry about partitions. You can have as many partitions as you want, and every consumer will get a copy of the message no matter which partition it is in.

From a code perspective, the only difference is the consumer group id. In the “competing consumer” pattern, every consumer must have the same consumer group id. In the “publish-subscribe” pattern, every consumer must have a different consumer group id.

Broadcasting Messages with Publish-Subscribe

As mentioned above, to broadcast messages to multiple consumers you need to use a different consumer group id for each consumer. There isn’t much else to it.

I’m going to re-use the code from my previous post on using Kafka with .NET and make the required change.

The producer code remains the same. The consumer code changes slightly. Here is the code for the consumer -

 1using Confluent.Kafka;
 2
 3public class Consumer 
 4{
 5    ConsumerConfig _consumerConfig;
 6
 7    public Consumer(string connection)
 8    {
 9        string groupId = Guid.NewGuid().ToString();
10        Console.WriteLine($"Consumer group ID: {groupId}");
11
12        _consumerConfig = new ConsumerConfig
13        {
14            BootstrapServers = connection,
15            GroupId = groupId,
16            AutoOffsetReset = AutoOffsetReset.Latest,
17            // the rest is the same
18        };
19    }

Running the Code

Start the producer, and start two instances of the consumer. Each will get a unique consumer group id, and each will get a copy of every message.

Partitions

When telling a friend about this, he brought up partitions, concerned that they would cause problems.

Download full source code.