Upgrading Apache Cassandra cluster


At thethings.io we use Cassandra as our timeseries database for thing data. Weeks ago we needed to upgrade our cassandra version to a major release version.

This post aims to group all steps that we use for upgrade our Cassandra cluster to a newer version. There are a lot of documentation and sometimes it is a little bit annoying to look at different pages.

This section describes how to upgrade Apache Cassandra with zero downtime if your cluster has a replication factor of at least 3.

Note:

  • Be careful when you run these steps on production environments.
  • For command simplicity I don’t include the user and password. Ex: nodetool drain -u username -pw password

Prerequisites

  • Make sure you have an up-to-date daily backup. We use nodetool snapshot tool for that but there are other options as well like OpsCenter.
  • Familiarize yourself with the changes and features in this new release.
  • Remove all dead nodes.
  • Do not run nodetool repair or other tasks.

Please check this link for more information.

Steps for upgrade cassandra version

1. Run nodetool drain before shutting down the existing Cassandra service. This flushes all memtables from the node to SSTables on disk. Cassandra stops listening for connections from the client and other nodes. Also prevents overcounts of counter data, and speeds up restart post-upgrade.

2. Stop cassandra services.

3. Back up your cassandra configuration files from the old installation to safe place.

4. Update java version.

5. Install the new version of Apache Cassandra.

If you are running Cassandra from a source you should download the latest tar.gz instead of using the package manager.

6. Configure the new product. Review, compare, merge and/or update any modifications you have previously made into the new configuration files for the new version (cassandra.yml, cassandra-env.sh, etc.). Take care with the configuration options and check version restrictions.

7. Start the cassandra services.

Check the logs for warnings, errors, and exceptions.

8. Run nodetool upgradesstables

9. Check the logs for warnings, errors, and exceptions.

10. Check the status of the cluster

11. Repeat theses upgrade steps on each node in the cluster.

References


Also published on Medium.