Resolving the terminating state of pods

Sometimes if the volume path is not available, the pod might go into a Terminating state. The volume path could becomes unavailable due to one of the following reasons:

IBM Spectrum Scale is not running.
The file system is unmounted.
The fileset is unlinked.

When a PVC is created using IBM Storage Enabler for Containers, a soft link is created between /var/lib/kubelet/pods/<pod id>/volumes/ibm~ubiquity-k8s-flex/<pv id> and the fileset link path as part of the volume mount operation.

If the fileset link path is unavailable, then the soft link becomes broken. At this stage, if a pod gets deleted, then it might go into a Terminating state for infinite period of time. Follow these step to clean up the pod which is in a Terminating state:

Note: These steps are only applicable if the PVC used in a pod is created using IBM Storage Enabler for Containers.

Find the id of the pod which is in Terminating state using the following command:
```
kubectl get pod --no-headers -o custom-columns=wwn:metadata.uid <pod name>
```

Find the node on which the pod was scheduled using the following command:

 kubectl get pod --no-headers -o custom-columns=wwn:spec.nodeName <pod name>

Check if a broken soft link exists on the node found in step 2 using the following command:
```
ls /var/lib/kubelet/pods/<pod id>/volumes/ibm~ubiquity-k8s-flex/
```
If the soft link is broken, then delete the soft link using the following command:
```
rm /var/lib/kubelet/pods/<pod id>/volumes/ibm~ubiquity-k8s-flex/<pv id>
```
Note: Trying to mount the above IBM Spectrum Scale file system might fail. You might need to stop the Docker service to mount the IBM Spectrum Scale file system.