Troubleshoot

Troubleshoot

This section provides information that can help you troubleshoot problems you may encounter when installing and using the ITOM Platform and the Operations Bridge Suite.

Mouse over and click the boxes below for more information.

ClosedManual verification commands

The following commands can be used to troubleshoot the ITOM Platform and the Operations Bridge Suite container deployment, for example to list namespaces and services.

./kube-status.sh
Displays the status of the K8S cluster.
./kube-stop.sh
Stops the K8S cluster.
./kube-restart.sh
Restarts the K8S cluster.
./kube-start.sh
Starts the K8S cluster.
kubectl

The command to interact with Kubernetes (K8S).

Tip To shorten the kubectl command, run the following command:

ln -s /usr/bin/kubectl /usr/bin/kl

This enables you to type kl instead of kubectl.

kubectl cluster-info
Summarizes information about some of the services that are running on the cluster, including Kubernetes master, KubeDNS for service discovery, and the endpoints of the KubeRegistry (if you are running a registry).
kubectl get nodes
Lists all nodes in the cluster.
kubectl describe nodes <node_IP>
Provides more specific information about the node, such as labels, events, capacity, CPU, memory, the maximum number of pods that the node can support, system information on the node, external IP address, the pods that are running, the list of namespaces, and resources.
kubectl get pods
Lists all pods in the default namespace (used to separate the ITOM Platform services from the deployed suites).
kubectl get pods -n=<namespace>

Lists all the pods that are running on the specified namespace.

For example, run kubectl get pods -n=opsbridge1 to get a list of the pods running in the namespace opsbridge1.

kubectl get pods --all-namespaces

Lists all the pods that are currently running in the cluster.

kubectl describe pod <pod_name> --namespace=<namespace>

Displays details about a specified pod in a specified namespace, such as the image it is running, the port it is exposing, and the command (/hyperkube) that is running inside the container itself with their options, volumes, and more.

kubectl exec <pod_name> -c <container> -n <namespace>

Executes a command in the specified container. If no container is specified, the first container in the pod is selected.

Example: kubectl exec omi-1949254658-p3ipj -c omi -n opsbridge1 bash -ti

Executes a bash shell in the OMi container with the pod name omi-1949254658-p3ipj and the namespace opsbridge1. By executing a bash shell in the OMi container, you can call CLIs from inside the container. For more information, see Access Command Line Interfaces.

kubectl get services --all-namespaces

Displays all the services running in the cluster.

kubectl logs <pod_name> --namespace=<namespace>

Displays the log output for the specified pod.

ClosedLog files

To troubleshoot your issue, you can review the following log files.

Installation

/opt/kubernetes/install-<date><time>.log

NFS share

  • /var/vols/itom/opsbridge/<namespace>/omi/opt/HP/BSM/log/topaz_all.log
  • /var/vols/itom/opsbridge/<namespace>/omi/opt/HP/BSM/log/jboss7_boot.log
  • /var/vols/itom/opsbridge/<namespace>/omi/opt/HP/BSM/log/supervisor/nanny_all.log
  • /var/vols/itom/opsbridge/opsbridge-opsbridge/pe/logs

Login

/var/vols/itom/opsbridge/<namespace>/omi/opt/HP/BSM/log/jboss/login.log

OBR

  • /var/vols/itom/opsbridge/<namespace>/obr/obr-server/adminServer/logs (Administration UI)
  • /var/vols/itom/opsbridge/<namespace>/obr/obr-server/Flink/log (Flink)
  • /var/vols/itom/opsbridge/<namespace>/obr/obr-server/log (OBR platform)
  • /var/vols/itom/opsbridge/<namespace>/obr/obr-server/data (OBR data)
  • /var/vols/itom/opsbridge/<namespace>/obr/obr-server/config (OBR configuration)
ClosedCommon problems and limitations

You may encounter the following problems and limitations when installing or administering the ITOM Platform and the Operations Bridge Suite.

ClosedITOM Platform is not accessible

Description

After the installation of the ITOM Platform, the ITOM Platform cannot be accessed at https://<external_access_host>:5443.

Possible solutions

ClosedITOM Platform is not accessible: nginx controller is Pending

Description

After the installation of the ITOM Platform, the ITOM Platform cannot be accessed at https://<external_access_host>:5443.
When running kubectl get pods --all-namespaces, the nginx ingress controller status is Pending.

Cause and solution

The map hash bucket size might be too small. Check if that is the case by running the following commands:

kubectl describe nginx-ingress-controller-u69gg

kubectl logs nginx-ingress-controller-u69gg

If an error is displayed similar to nginx: [emerg] could not build map_hash, increase the map_hash_bucket_size as follows:

  1. Access the file /opt/kubernetes/objectdefs/nginx-ingress.yaml
  2. Locate the specified map_hash_bucket_size (32 by default) and increase it, for example to 128

  3. Run the following commands to recreate the nginx-ingress.yaml file:

    kubectl delete –f /opt/kubernetes/objectdefs/nginx-ingress.yaml

    kubectl create –f /opt/kubernetes/objectdefs/nginx-ingress.yaml

  4. Optional. If you get a warning about failed scheduling, the scheduling constraints could not be fulfilled. Execute the following command to fix this:

    kubectl label nodes role=loadbalancer –all

    The nginx pod container should then be started automatically.

  5. After the OMi configuration, you must repeat steps 2 and 3 for the OMi nginx controller located at
    /var/vols/itom/core/suite-install/opsbridge/output/suite-ingress-controller-configmap.yaml

ClosedITOM Platform is not accessible: Gateway time out

Description

After the installation of the ITOM Platform, the ITOM Platform cannot be accessed at https://<external_access_host>:5443. The Docker daemon cannot be started, and displays the error message Gateway time out when logging into IDM.

Cause and solution

Kubernetes might not be running. Run the following commands to start Kubernetes:

cd $K8S_HOME/bin

./kube-start.sh

ClosedLogin to ITOM Platform is not possible: IDM service is not ready yet

Description

After the installation of the ITOM Platform, the ITOM Platform cannot be accessed at https://<external_access_host>:5443. The login failure error The IDM service is not ready yet is displayed, and the pods autopass-lm, idm, and suite-installer all have the status CrashLoopBackOff.

Solution

  1. Run the following command:

    kubectl delete -f autopass-lm.yaml; kubectl delete -f autopass-pg.yaml; kubectl delete -f idm.yaml; kubectl delete -f idm-pg.yaml; kubectl delete -f suite.yaml

  2. Delete the subfolders located in the NFS subdirectories <NFS_HOME>/baseinfra-1.0/autopass_db, <NFS_HOME>/baseinfra-1.0/idm_db, and <NFS_HOME>/baseinfra-1.0/suite_db.

  3. Run the following command:

    kubectl create -f idm-pg.yaml; kubectl create -f idm.yaml; kubectl create -f autopass-pg.yaml; kubectl create -f autopass-lm.yaml; kubectl create -f suite.yaml

Closed"502 Bad Gateway" error when attempting to launch OMi

Description

After the installation of the Operations Bridge Suite, a 502 Bad Gateway error is displayed when trying to access OMi.

Cause and solution

The 502 error is displayed because OMi is not yet up and running. Depending on the host machine, it might take up to one hour for OMi to start after the initial configuration.

ClosedNo server connection: invalid character "{" in host name

Description

A connection to the server could not be established. The log displays that the invalid character "{" is used in the host name.

Cause and solution

The firewall might still be enabled on the NFS server. Make sure that the firewall is disabled.

ClosedPod is in ImagePullBackOff or ErrImagePull status: Image not found

Description

After the installation of the ITOM Platform, one of the pods has the status ImagePullBackOff or ErrImagePull. When running the command kubectl describe pod <pod_name> -n <namespace>, the following error message is displayed:

Image <image_name> not found

Cause and solution

Make sure the images are pushed into the private docker registry. To confirm, run the following command:

docker pull <image_name>

ClosedPod is in ImagePullBackOff or ErrImagePull status: Error while pulling image

Description

After the installation of the ITOM Platform, one of the pods has the status ImagePullBackOff or ErrImagePull. When running the command kubectl describe pod <pod_name> -n <namespace>, the following error message is displayed:

Error while pulling image: Get http://localhost:5000/v1/repositories/xxx: dial tcp [::1]:5000: getsockopt: connection refused

Cause and solution

To resolve this issue, delete the Docker registry and the registry proxy pods, and then restart them.

ClosedWorker node installation fails with a Flannel related error

Description

Setting up one or multiple worker nodes fails during the ITOM Platform installation due to an error related to Flannel.

Cause and solution

To troubleshoot and resolve this issue, do the following:

  • Double check if the FQDN is resolved to the correct IP address on the master node.

  • On the master node, run kube-restart.sh

  • Reinstall the worker node from the ITOM Platform.

Closed"503 nginx error" when attempting to run the Suite Installer

Description

After the installation of the ITOM Platform, a 503 Nginx error is displayed when trying to access the Suite Installer.

Cause and solution

This error might be displayed because the time on the master and worker nodes is different. To resolve this issue, synchronize the time on your nodes by using, for example, NTP or VMWare tools.

Send Help Center feedback