Cloudify High Availability Cluster Upgrade Guide

Overview

This guide explains how to upgrade a high-availability Cloudify Manager cluster to a new version.
Since the old version of Cloudify cluster was active-passive, the guide is the same for both this case and the Cloudify All-In-One case.
The best-practice to upgrade Cloudify is to deploy the new version on a new set of hosts, therefore a new host should be created for every existing Manager.

Upgrade to new hosts

  1. Create and download a snapshot from the existing Manager. Please refer to the Backup and restore guide for further instructions.
  2. Install a new Cloudify cluster. Please refer to the Cloudify cluster installation guide for further instructions.
  3. After the cluster and CLI installation is done, ssh into all Managers except one (best practice - 2 out of 3) and run cfy_manager stop. This would stop Cloudify services on these hosts.
  4. Copy the snapshot to the last Manager (the only active one).
  5. Ssh into this Manager, and run the following code:

    sudo /opt/patroni/bin/patronictl -c /etc/patroni.conf show-config > /etc/patroni.conf.bak
    sudo /opt/patroni/bin/patronictl -c /etc/patroni.conf edit-config -s ttl=86400 -s retry_timeout=86400 --force
    sudo sed -i.bak -e 's/timeout client.*$/timeout client 1440m/' -e 's/timeout server.*$/timeout server 1440m/' /etc/haproxy/haproxy.cfg
    sudo service haproxy reload

    This will increase the Patroni and HAproxy timeouts, which is necessary for large snapshots to restore correctly.

  6. On the same Manager, upload the snapshot, and restore it. Please refer to the Backup and restore guide for further instructions.

  7. After the restore is done, revert the Patroni and HAproxy timeouts to their defaults:

    sudo /opt/patroni/bin/patronictl -c /etc/patroni.conf edit-config --apply patroni.conf.bak --force
    sudo mv /etc/haproxy/haproxy.cfg.bak /etc/haproxy/haproxy.cfg
    sudo service haproxy reload
  8. Connect to the inactive Managers, and on each one run cfy_manager start.

  9. Reinstall agents. Please refer to the Agents installation guide for further instructions.

  10. Delete old host/s.