In the world of IT, everything is constantly changing. Therefore upgrading software is one of the most important operations to do. Apache Kafka is no different. It is recommended to upgrade whenever a new version of Kafka is out. The longer you wait, the harder it will be.

One of the main reasons why you should upgrade your Kafka cluster is that there are many improvements and bug fixes in every new version released.

Many people are afraid of upgrades, especially if steps for upgrading seem hard and complicated, like in the case of Kafka. But with a little effort and with the help of official documentation and this blog you should do it with ease.

In our case, we were upgrading Kafka in production from version 2.0 to 2.5. We needed to do it automatically with no downtime so we decided to use Ansible scripts for automation.

Just a quick disclaimer before we start. Here, we are focusing on upgrading plain Kafka. If you are using Kafka on a confluent platform please refer here for Ansible scripts.

Without further due, let’s see what you need to do in order to upgrade Kafka with Ansible scripts.

Upgrade steps

There are four steps to successfully upgrade Kafka. These steps work for upgrading all versions from 0.8.x to 2.5.x. If you are using a version of Kafka that is prior to 0.8.x please refer to the official documentation for steps that you need to do, because they are not all the same as here.

• Update server properties

• Update Kafka binaries

• Update inter broker protocol version

• Update log message format

After all of these steps, you will need to perform a rolling restart of your brokers.

An important thing to notice here is that if you are upgrading from a version prior to 2.1.x once you do step 3 there is no turning back. That means that there is no way to downgrade to a version prior to 2.1.x. This is because of the change of the schema that used to store consumer offsets.

1. Update server properties

The first step is to update server.properties on all Kafka brokers. You need to add the following properties:

• inter.broker.protocol.version=CURRENT_KAFKA_VERSION(etc. 2.0) — “CURRENT_KAFKA_VERSION” is referring to your current version of Kafka that you are upgrading from.
• log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION — If you have previously overridden the log message format then you don’t need to override it again. If you haven’t, then you will need to set the current log message format.

In our case, we were upgrading from 2.0 and we haven’t overridden the log message format earlier so we added those two parameters to the end of our server.properties:

inter.broker.protocol.version=2.0log.message.format.version=2.0

After you set these properties to all brokers in the cluster you need to perform a rolling restart of your brokers for these settings to take effect.

Automation of upgrade with Ansible

In Ansible you need to create a new role for each step. These are four Ansible roles for the upgrade. Every role has it’s main.yml which is a task that role is going to do. Also, every role has properties that are actually server.properties in Kafka.

Task main.yml for role first role(kafka-set-current-version):

- name: Update Kafka configurationtemplate:
src: “kafka.server.properties.j2”
dest: “{{kafka_install_path}}/kafka/config/server.properties”
owner: kafka
group: kafka
mode: 0770

Properties are one that you are currently using with the addition of two parameters that you can place to the end of properties. Those are:

inter.broker.protocol.version=2.0log.message.format.version=2.0

Task main.yml for second role(kafka-update-binaries):

- name: Unzip Kafka
unarchive:
src: “{{ artifacts_path }}/kafka_2.12–2.5.0.tgz”
dest: “{{kafka_install_path}}/”
owner: kafka
group: kafka
mode: 0770- name: Link Kafka installation dir
file:
src: “{{kafka_install_path}}/kafka_2.12–2.5.0”
dest: “{{kafka_install_path}}/kafka”
state: link
owner: kafka
group: kafka
mode: 0770- name: Create Kafka configuration
template:
src: “kafka.server.properties.j2”
dest: “{{kafka_install_path}}/kafka/config/server.properties”
owner: kafka
group: kafka
mode: 0770

Properties for the second task are the same as for the first task.

Task main.yml for third role(kafka-update-inter-broker-protocol):

- name: Update inter broker protocol
template:
src: “kafka.server.properties.j2”
dest: “{{kafka_install_path}}/kafka/config/server.properties”
owner: kafka
group: kafka
mode: 0770

Now you need to slightly change properties. You need to update the inter broker protocol so in properties you need to specify the version that you are upgrading to. The log message format is still unchanged at this point.

inter.broker.protocol.version=2.5log.message.format.version=2.0

Task main.yml for fourth role(kafka-update-log-message-format)

- name: Update log message format
template:
src: “kafka.server.properties.j2”
dest: “{{kafka_install_path}}/kafka/config/server.properties”
owner: kafka
group: kafka
mode: 0770

In this final step, you need to change properties one more time. You need to increase the version of the log message format to the version of Kafka that you are upgrading to.

inter.broker.protocol.version=2.5log.message.format.version=2.5

To automatize starting Kafka you also need one role for starting Kafka. You can name that role “service-kafka-start” and provide the following task:

- name: (Re)start service kafka
service:
name: “kafka”
state: restarted
enabled: yes

To be sure if your Kafka is started after each step you need to make one more role “kafka-check”. It will check if your Kafka is up and running.

- name: Waiting kafka service to be readyshell: ‘{{kafka_install_path}}/kafka/bin/kafka-broker-api-versions.sh — bootstrap-server {{internal_hostname}}:9092 | grep -c “id: {{ groups.kafkaservers.index(inventory_hostname) + 1 }}”’
register: app_check_ok
until: app_check_ok.stdout.find(“0”)
retries: 30
delay: “1”
- name: sleep for 5 minutes
pause:
minutes: 5

To ensure that you don’t lose any data during the upgrade you can make a pause for 5 minutes between each step on each broker. The pause can vary depending on how much traffic you have in your Kafka cluster.

In our case, we had 3 brokers and with four steps that are 12 restarts overall(each broker has 4 restarts). That means that with 5 minutes of pause, upgrading will take a minimum of one hour and that is something to be considered.

The final thing that you will need to do is to create an Ansible playbook that will run all tasks and upgrade Kafka.

#FIRST STEP — set inter.broker.protocol.version and log.message.format.version to current version- name: set inter.broker.protocol.version and log.message.format.version to current version
hosts: kafkaservers
serial: 1
tags:
- kafka-upgrade
roles:
- role: kafka-set-current-versions
- role: service-start
- role: kafka-check##SECOND STEP — link kafka binaries to new version
- name: link kafka binaries to new version
hosts: kafkaservers
serial: 1
tags:
- kafka-upgrade
roles:
- role: kafka-update-binaries
- role: service-start
- role: kafka-check#THIRD STEP — set inter.broker.protocol.version to newest version
- name: set inter.broker.protocol to newest version
hosts: kafkaservers
serial: 1
tags:
- update-versions
roles:
- role: kafka-update-inter-broker-protocol
- role: service-start
service_name: kafka
- role: kafka-check#FOURTH STEP — set log.message.format.version to newest version
- name: set log.message.format to newest version
hosts: kafkaservers
serial: 1
tags:
- update-versions
roles:
- role: kafka-update-log-message-format
- role: service-start
- role: kafka-check

The hosts file that is specified in the Ansible playbook is information about your Kafka cluster.

With the parameter “serial: 1” you can ensure that every step is done on one broker at a time. If you don’t specify it then every step will be done on all brokers at the same time and that is something that you don’t want.

It is recommended that you separate the first two steps into one command, and the last two steps into the second command. Like we already said, after the first two steps you can still downgrade Kafka if something is not working.

Firstly you will enter:

ansible-playbook -i hosts_kafka -e artifacts_path=/home/user/kafka-upgrade -t kafka-upgrade playbook-upgrade-kafka.yml

When that finishes you need to enter the second command:

ansible-playbook -i hosts_kafka -e artifacts_path=/home/user/kafka-upgrade -t update-versions playbook-upgrade-kafka.yml

The artifacts path in this Ansible command refers to the path where you have downloaded new Kafka.

After you enter the starting commands you only need to watch and relax. Ansible will do everything now.

Our Kafka upgrade project is available on our Github for everyone. Enjoy!

And that’s it. Maybe it seems a little complicated at first, but trust me, if you follow this blog step by step you can do it.