Administer > Administer the Operations Bridge Suite > Configure scaling and high availability

Configure scaling and high availability

You can improve your system availability and reliability by scaling your suite resources as required. You can scale single nodes, as well as multiple nodes.

By managing your resources, you can scale your system out or in. For example, by increasing the number of pod replicas on a deployment, the load of the deployment can be automatically distributed across all pods.

A high availability configuration offers continuous service despite power outages, machine downtime, and heavy load.

ClosedScale a BVD deployment horizontally

To ensure a high availability of BVD even if one of the worker nodes fails, you can increase the number of BVD receiver (bvd-receiver-deployment) and/or web server (bvd-www-deployment) pod replicas.

Tip  

  • Scale out the BVD receiver load if you send a lot of data to BVD. By scaling out, the number of data samples that can be processed by BVD increases.
  • Scale out the BVD web server load if BVD is accessed by multiple people at the same time. By scaling out, a higher number of users will be able to access BVD concurrently.

  1. Launch the Management Portal from a supported web browser:

    https://<external_access_host>:5443

    <external_access_host> is the fully qualified domain name of the host which you specified as EXTERNAL_ACCESS_HOST in the install.properties file during the Container Deployment Foundation installation. Usually, this is the master node's FQDN.

  2. Log on as the admin user.
  3. Go to Resources, and click All namespaces. In the drop-down list, select the namespace for the Operations Bridge Suite that was assigned during the installation (for example opsbridge).

  4. Go to Workloads > Deployments. You can either scale out the receiver load (bvd-receiver-deployment), or the web server load (bvd-www-deployment).

    Caution Do not scale out any of the of the other BVD pods.

  5. For the deployment you want to scale out, click Actions and select View/edit YAML. The Edit a Replica Set dialog box opens.
  6. Edit the line spec replicas : 1. Increase the number of pod replicas as required.
  7. Click Update.
  8. Wait until the deployment is updated. This might take a few minutes.

  9. Optional. You can verify that the deployment has been updated correctly:

    1. Refresh the Deployments page. For the deployment you selected, the number of pods should have increased (for example 2/2 instead of 1/1).

    2. Go to Workloads > Replica Sets, and verify that the number of pod replicas for the deployment has increased as specified. The age displays for how long the pods have been running.

ClosedConfigure high availability

To achieve a high availability of your system, use the following techniques:

Important Only by using all of these techniques, your system will be highly available.

  • Highly available redundant storage. Use a redundant NFS server for your container deployment.
  • Highly available database instances. Use a redundant external database.
  • Kubernetescluster with multiple master nodes. See High availability of the Kubernetes cluster.
  • Operations Bridgeservices on multiple worker nodes. To achieve a high availability of your Operations Bridge capabilities, deploy the capabilities on multiple worker nodes. By scaling your deployment horizontally, worker nodes can take over loads from failed worker nodes. Scaling is currently only available for BVD. See Scale a BVD deployment horizontally and the example in Hardware requirements.
  • Keep-alive monitoring of Kubernetes. Kubernetes automatically detects failures of single services and complete nodes and restarts pods on other nodes.
ClosedHigh availability of the Kubernetes cluster

When installing multiple master and worker nodes, set the EXTERNAL_ACCESS_HOST to an FQDN which is resolved to the HA_VIRTUAL_IP. This way, you make sure you can access the CDF with the FQDN defined in EXTERNAL_ACCESS_HOST.

By specifying these properties, an ingress instance and keepalived are launched on each master node. keepalived binds the Virtual IP to a master node. The node with the Virtual IP is in master mode while the other nodes are in standby mode. If the master node with the Virtual IP is down, the Virtual IP is bound to another master node.

ClosedHigh availability of Suite capabilities

BVD

BVD pods are highly available by default, even if you have not scaled them out. However, if one of the worker nodes fails, restarting the pods will cause a short downtime of your system.
To ensure a constant system availability even if one of the worker nodes fails, you can increase the number of BVD receiver (bvd-receiver-deployment) and web server (bvd-www-deployment) pod replicas.
A short downtime will then only occur if the BVD Redis pod is affected by the crash of a worker node or the Redis process. In this case, Kubernetes restarts the BVD Redis pod. This takes usually less than a minute.
Any data that was sent during the downtime of the BVD Redis pod will be buffered by the BVD receivers, so no data is lost.

For more information, see Scale a BVD deployment horizontally.

Other Operations Bridge capabilities

Apart from BVD, no other capability supports running pods multiple times on different worker nodes. Therefore, when one of the worker nodes fails, all corresponding pods have to be restarted. The downtime of the capabilities depends on the restart time of those pods. For Performance Engine and Operations Bridge Reporter, this can take several minutes. For Operations Manager i, this can take more time, depending on the amount of events and CIs.

Send Help Center feedback