Getting Confluent Kafka Working in Linux with Docker Desktop

Dec 23, 2024 # Kafka, Linux, Docker, Container

Some time ago I wrote that getting Kafka running on Windows was not easy to do, there was a lot of information but it was spread across many different sources, and not simple to follow.

In the end, I used the Confluent executable, it abstracted away the complexity of setting up Kafka behind a simple command. That was Windows. But when I tried the same in Ubuntu Linux I got all sorts of errors.

Like in Windows, I use the Docker Desktop in Linux because it is easier to work with. I don’t have to start all the required services, and then remember to stop them when I’m done. But as far as I know, they work a bit differently, in Windows when the Docker Desktop starts it fires up more background services than the Linux version and it seems the Confluent executable relies on these services.

Trying to get Confluent Kafka working in Linux

I downloaded the Confluent executable for Linux and tried to run the same command, but it complained about not being able to connect to the Docker service, I resolved that, and it complained about being able to connect to the Docker port. At this point, I gave up. I had the feeling if I solved this problem another would arise.

It seems the Confluent executable is very opinionated about how Docker should be running and I couldn’t find a way to make it work, and I had no intention of changing my Docker setup just to make it work.

In the end, all the Confluent executable does is run some Docker commands so I needed to figure out what they were and use them in Linux.

Actually getting Confluent Kafka working in Linux

I went back to Docker in Windows, and got the command used to start the container.

I noticed it uses a local network, so I need to create that too.

Back in Linux, I ran two commands, the first was to create the network -

docker network create -d bridge confluent-local-network

and the second to start the container this is one very long line, make sure to copy it all -

docker run --hostname=confluent-local-broker-1 --user=appuser --env=KAFKA_BROKER_ID=1 --env=KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT --env=KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://confluent-local-broker-1:51257,PLAINTEXT_HOST://localhost:64886 --env=KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 --env=KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0 --env=KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 --env=KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 --env=KAFKA_PROCESS_ROLES=broker,controller --env=KAFKA_NODE_ID=1 --env=KAFKA_CONTROLLER_QUORUM_VOTERS=1@confluent-local-broker-1:51258 --env=KAFKA_LISTENERS=PLAINTEXT://confluent-local-broker-1:51257,CONTROLLER://confluent-local-broker-1:51258,PLAINTEXT_HOST://0.0.0.0:64886 --env=KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT --env=KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER --env=KAFKA_LOG_DIRS=/tmp/kraft-combined-logs --env=KAFKA_REST_HOST_NAME=rest-proxy --env=KAFKA_REST_LISTENERS=http://0.0.0.0:8082 --env=KAFKA_REST_BOOTSTRAP_SERVERS=confluent-local-broker-1:51257 --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=container=oci --env=LANG=C.UTF-8 --env=UB_CLASSPATH=/usr/share/java/cp-base-lite/* --env=KAFKA_ZOOKEEPER_CONNECT= --env=CLUSTER_ID= --volume=/etc/kafka/secrets --volume=/var/lib/kafka/data --network=confluent-local-network --workdir=/home/appuser -p 64886:64886 -p 8082:8082 --restart=no --label='architecture=x86_64' --label='build-date=2024-01-16T18:44:24' --label='com.redhat.component=ubi8-minimal-container' --label='com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI' --label='description=Common base image for Confluent lightweight Docker images.' --label='distribution-scope=public' --label='io.buildah.version=1.29.0' --label='io.confluent.docker=true' --label='io.confluent.docker.build.number=6' --label='io.confluent.docker.git.id=f38aed949' --label='io.confluent.docker.git.repo=confluentinc/kafka-images' --label='io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.' --label='io.k8s.display-name=Red Hat Universal Base Image 8 Minimal' --label='io.openshift.expose-services=' --label='io.openshift.tags=minimal rhel8' --label='maintainer=partner-support@confluent.io' --label='name=confluent-local' --label='release=7.6.0' --label='summary=Kafka with Rest Proxy' --label='url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.9-1108.1705420507' --label='vcs-ref=7e7670a8c8c9a3be83beaa2787f3703b404d4a1d' --label='vcs-type=git' --label='vendor=Confluent' --label='version=f38aed949' --runtime=runc -d confluentinc/confluent-local:7.6.0

The above command will set -

  • Kafka REST Port to 8082
  • Plaintext Ports to 64886

You can change these to whatever you want by finding and replacing the values in the command.

After this, I tried using the Confluent executable again to create a topic and produce/consume messages, but it still didn’t work properly. But my .NET application did work!

Kafka Linux Docker Container
comments powered by Disqus

Related