Upgrade

CDF can be upgraded from version 2017.03 to 2017.06. The upgrade will update all components such as Docker, Kubernetes, Heapster, Vault, Etcd, Flannel and the CDF core to the 2017.06.001 version.

To upgrade CDF, do the following:

1. Make sure all CDF nodes are running.

2. Run the command: upgrade.sh -g on any one of the master nodes to generate the CDF_upgrade_parameters.txt under the current directory.

Enter the fully qualified domain name (FQDN) of the host name for external users to access the clusters.

For a single master node, the specified FQDN must be resolved to the master node IP address. For multiple master nodes, the specified FQDN must be resolved to the IP address you specified in the HA_VIRTUAL_IP setting.

Note This step will try to remove resources from the yaml file. If you receive an error when deleting the resources, remove the resources manually before the upgrade. Otherwise, the upgrade may fail due to the existing resources.

3. Stop all the master nodes one after another with the following commands and note which master node you stopped first:

   docker -H unix:///var/run/docker-bootstrap.sock stop etcd_container

   ${K8S_HOME}/bin/kube-stop.sh

4. Stop all the worker nodes one after another with the following command:

   ${K8S_HOME}/bin/kube-stop.sh

5. Optional. If the machine has multiple network interfaces, the user can set the value of FLANNEL_IFACE in the CDF_upgrade_parameters.txt file to specify the interface to be used for docker inter-host communication before the upgrade.
6. Copy the CDF_upgrade_parameters.txt to all nodes.

7. Upgrade the master node that was stopped first with the command: 

   upgrade.sh -u /<Parameter file path>/CDF_upgrade_parameters.txt

8. Upgrade the rest of the master nodes one after the other (no preferred order) with the following command:

   upgrade.sh -u /<Parameter file path>/CDF_upgrade_parameters.txt

9. Upgrade the worker nodes one after the other (no preferred order) with the following command:

   upgrade.sh -u /<Parameter file path>/CDF_upgrade_parameters.txt

You may have to retry the upgrade of one or multiple nodes if an error message popped up about the upgrade on the node having failed.

To retry the CDF upgrade on that failed node, do the following:

1. Run the following command: source /etc/profile.

2. Optional. Run the following command: upgrade.sh -d /<Parameter file path>/CDF_upgrade_parameters.txt only when the failed master node is not the first stopped master. This command helps to remove the etcd member from the etcd cluster.

3. Check if the backup-complete file is available under the /<backup directory>/CDF_201703_backup directory.

   • If the backup-complete file does not exist, the CDF backup step has failed. Back up the CDF again with the following steps:

     1. Delete the /<backup directory>/CDF_201703_backup folder.
     2. Run the following command: upgrade.sh -u /<Parameter file path>/CDF_upgrade_parameters.txt

   • If the backup-complete exists, the CDF backup has been completed.

4. Check the status of the kubelet service with the command: systemctl status kubelet

   • If the kubelet service is not active, delete the kubelet.service file under the /usr/lib/systemd/system directory.
   • If the kubelet service is active, run the command systemctl stop kubelet to stop the kubelet service. Then delete the kubelet.service file under the /usr/lib/systemd/system directory.

5. Check the status of the docker service with the following command: systemctl status docker

   • If the docker service is not active, delete the docker.service file under the /usr/lib/systemd/system directory.
   • If the docker service is active, run the command systemctl stop docker to stop the docker service. Then delete the docker.service file under the /usr/lib/systemd/system directory.

6. Check the status of the docker-bootstrap service with the command: systemctl status docker-bootstrap.

   • If the docker-bootstrap is not active, delete the docker-bootstrap.service file under the /usr/lib/systemd/system directory.
   • If the docker-bootstrap is active, run the command systemctl stop docker-bootstrap and then delete the docker-bootstrap.service file under the /usr/lib/systemd/system directory.

7. Unmount the mounted data with the following commands:

   for data in $(mount | grep "${K8S_HOME}/data/" | cut -d" " -f3 | sort -r);do umount -f -l $data; done

   for data in $(mount | grep "/usr/lib/kubelet" | cut -d" " -f3 | sort -r);do umount -f -l $data; done

8. Reboot the machine on which you are retrying the upgrade.

9. Delete the ${K8S_HOME} directory with the following command:

   rm -rf ${K8S_HOME}

10. Rollback the ${K8S_HOME} directory with the following command:

    mv /<backup directory>/CDF_201703_backup ${K8S_HOME}

11. Delete the backup-complete file under the ${K8S_HOME} directory.

12. Recover the docker.service and the docker-bootstrap.service files with the following commands:

    mv ${K8S_HOME}/docker.service /usr/lib/systemd/system/

    mv ${K8S_HOME}/docker-bootstrap.service /usr/lib/systemd/system/

13. Optional. Restore the data on the NFS server manually before retrying the upgrade only for the first stopped master node that fails to upgrade.

14. Retry the upgrade with the following command:

    upgrade.sh -u /<Parameter file path>/CDF_upgrade_parameters.txt