Troubleshooting the IBM Storage Enabler for Containers

Checking logs

Detecting errors

1. Run the ubiquitu_cli.sh -a status_wide command to check if:
   • All Kubernetes pods are in Running state.
   • All PVCs are in Bound state.
   • ubiquity-k8s-flex pod exists on each master node in the cluster. If you have three master nodes and five worker nodes, you must see a eight ubiquity-k8s-flex pods.

   Note: The output of the ubiquitu_cli.sh -a status_wide is similar to the ./ubiquity_cli.sh -a status output, illustrated in the What to do next section of Performing installation of IBM Storage Enabler for Containers.

2. If you find no errors, but still unable to create or delete pods with PVCs, continue to the next step.
3. Display the malfunctioned stateful pod ($> kubectl describe pod pod_ ID). Usually, pod description contains information about possible cause of the failure. Then, proceed with reviewing the IBM Storage Enabler for Containers logs.
4. Display the IBM Storage Kubernetes FlexVolume log for the active master node (the node that the controller-manager is running on). Use the $> kubectl logs -n ubiquity ubiquity-k8s-flex-<pod_ID_running_on_master_node> command. As the controller-manager triggers the storage system volume mapping, the log displays details of the FlexVolume attach or detach operations.
   Additional information can be obtained from the controller-manager log as well.
5. Review the IBM Storage Kubernetes FlexVolume log for the worker node, on which the container pod is scheduled. Use the $> kubectl logs -n ubiquity ubiquity-k8s-flex-<pod_ID_running_on_worker_node> command. As the kubelet service on the worker node triggers the FlexVolume mount and umount operations, the log is expected to display the complete volume mounting flow.
   Additional information can be obtained from the kubelet service as well, using the $> journalctl -u kubelet command.
6. Display the IBM Storage Enabler for Containers server log ($> kubectl logs -n ubiquity deploy/ubiquity command) or its database log ($> kubectl logs -n ubiquity deploy/ubiquity-db command) to check for possible failures.
7. Display the IBM Storage Dynamic Provisioner log ($> kubectl logs -n ubiquity ubiquity-k8s-provisioner) to identify any problem related to volume provisioning.

   Note: In general, you can use a request ID of a log entry to identify a particular event in the IBM Storage Enabler for Containers log. This will help you understand if the event is related to FlexVolume or Dynamic Provisioner. For example, if you have this event stored in the Provisioner log INFO provision.go:141 volume::Provision [9a48c08c-6e3e-11e8-b510-a2547d4dae22-Provision] PVC with capacity 1073741824, the string section 9a48c08c-6e3e-11e8-b510-a2547d4dae22 serves as a request ID. It identifies the event as related to the volume provisioning. Then, in the Enabler for Containers log, you can easily detect all entries with the same request ID, identifying them as relevant to volume provisioning. The same identification method can be applied to events, originating from the FlexVolume log.

8. View the Spectrum Connect Spectrum Connect log (hsgsrv.log) for list of additional events related to the storage system and volume operations.

mv /etc/kubernetes/manifests/kube-controller-manager.yaml  /tmp
sleep 5
mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/
sleep 15
#check the control-manager pod is running.
$> kubectl get pod -n kube-system  | grep controller-manager
# Verify it is in Running state.

$> TOKEN=$(kubectl get secret --namespace default $(kubectl get secret
--namespace default | grep service-account | awk '{print $1}') -o yaml | 
grep token: | awk '{print $2}' | base64 -d)

$> kubectl config set-credentials <mycluster.user> --token=${TOKEN}