Command Line Interface Consumer for Kafka in C#
When I started working with Kafka, I installed it locally on Docker and used a combination of the Confluent Command Line Interface (CLI) and C# programs I wrote.
One of the CLI tools from Confluent let me produce and consume messages on a Kafka topic by specifying the topic name and broker address. It was an easy way try things out.
But when I used this more recently, I couldn’t get it to work. I had updated the Docker image and the CLI. There seem to be new authentication requirements when using the CLI, authentication requirements that are not needed when using a C# consumer. I also could not find a tutorial on Confluent about local Kafka usage.
I messed around with it for a while, but eventually decided to write my own simple CLI consumer in C# using the new dotnet run app.cs approach.
Here is the code -
1#!/usr/bin/dotnet run
2#:package Confluent.Kafka@2.5.0
3using Confluent.Kafka;
4
5if (args.Length < 2 || args.Length > 3 )
6{
7 Console.WriteLine("Usage: url topic [groupId]");
8 return;
9}
10
11string url = args[0];
12string topic = args[1];
13string groupId = args.Length == 3 ? args[2] : Guid.NewGuid().ToString();
14Console.WriteLine($"Using groupId: {groupId}");
15ConsumerConfig _consumerConfig = new ConsumerConfig
16{
17 BootstrapServers = url,
18 GroupId = groupId,
19 AutoOffsetReset = AutoOffsetReset.Latest,
20 EnableAutoCommit = true, // this is the default but good to know about
21 EnableAutoOffsetStore = true // this is the default but good to know about
22};
23
24using var consumer = new ConsumerBuilder<string, string>(_consumerConfig).Build();
25consumer.Subscribe(topic);
26
27Console.WriteLine($"Looking for messages on topic: {topic}");
28while (true)
29{
30 var consumeResult = consumer.Consume();
31 Console.WriteLine($"{consumeResult.Message.Value}");
32}This can be executed like a script after it is made executable with chmod +x consumer.cs.
I’ll write a follow up with an example of a producer in a day or two.