IBM Cloud Docs
Why does my cluster upgrade fail due to Pod Security upgrade prerequisites?

Why does my cluster upgrade fail due to Pod Security upgrade prerequisites?

When you upgrade your IBM Cloud Kubernetes Service from Kubernetes version 1.24 to version 1.25, the upgrade fails and you see an error message similar to the following example.

Version update cancelled. CAE009: Cannot complete cluster master operations because the cluster does not pass Pod Security upgrade prerequisites. Reason: [ClusterRoleBinding 'restricted-psp-user' does not have expected subjects]. For more information, see the troubleshooting docs: 'https://ibm.biz/master_pod_security_upgrade_iks_125'

The Kubernetes PodSecurityPolicy admission controller was removed in Kubernetes 1.25 and replaced with a new Pod Security Admission controller.

To safely upgrade IBM Cloud Kubernetes Service clusters from version 1.24 to version 1.25, the cluster PodSecurityPolicies (PSP) and associated role-based access control must satisfy the following prerequisites.

  • No PSPs beyond the 5 IBM Cloud defined PSPs can exist.
  • The IBM Cloud defined cluster role bindings that give all users and service accounts authority to use the IBM Cloud defined privileged and restricted PSPs must exist.

These prerequisites ensure that the cluster's version 1.24 PodSecurityPolicy configuration is equivalent to version 1.25 Pod Security Admission configuration and the upgrade and switch to Pod Security Admission does not break existing applications. Note these prerequisite do not preclude use of third party pod security providers.

Before you begin

If you are not ready to migrate to Pod Security Admission, you can clear the status message by performing a cluster master refresh.

If you already performed the Pod Security Admission upgrade prerequisite actions, the error message indicates an action that you might have missed or an unexpected change to IBM Cloud Kubernetes Service defined resources that you need to address. Complete the following steps based on the message you are seeing.

Could not get PodSecurityPolicies

  1. Run the following to command to get your PSPs.
    kubectl get podsecuritypolicies
  2. If there is no error, try again to upgrade the cluster master.

Found non-IBM PodSecurityPolicy

There are additional PSPs that need to be removed.

Before you upgrade your cluster to version 1.25, verify that only the following PSPs exist.

  • ibm-privileged-psp
  • ibm-anyuid-psp
  • ibm-anyuid-hostpath-psp
  • ibm-anyuid-hostaccess-psp
  • ibm-restricted-psp

  1. List your PSPs.

    kubectl get podsecuritypolicies

  2. If the additional PodSecurityPolicy is no longer in use, delete it. If there are additional pod security policies, review the migration guide.

    kubectl delete podsecuritypolicies PSP

  3. Retry the cluster upgrade.

Could not get ClusterRoleBinding privileged-psp-user

This message indicates the privileged-psp-user cluster role binding does not exist or that some other error prevented the upgrade operation from reading the resource.

  1. List the resource.
    kubectl get clusterrolebindings privileged-psp-user
  2. If you get a NotFound error, create or update the privileged-psp-user cluster role binding.

ClusterRoleBinding privileged-psp-user does not have expected roleRef

This error indicates an unexpected change and that the cluster role binding is referencing the wrong cluster role.

  1. Delete the cluster role binding.

    kubectl delete clusterrolebinding privileged-psp-user

  2. Create the privileged-psp-user cluster role binding.

ClusterRoleBinding privileged-psp-user does not have expected subjects

If this change was intentional, you must first determine whether your apps can run with the cluster role binding. If they can, Create or update restricted-psp-user cluster role binding.

Could not get ClusterRoleBinding restricted-psp-user

This message indicates the restricted-psp-user cluster role binding does not exist or some other error prevented the upgrade operation from reading the resource.

  1. Get the details of the restricted-psp-user cluster role binding.
    kubectl get clusterrolebindings restricted-psp-user
  2. If you get a NotFound error, create or update restricted-psp-user cluster role binding.

ClusterRoleBinding restricted-psp-user does not have expected roleRef

This error indicates is an unexpected change and that the cluster role binding is referencing the wrong cluster role.

Complete the following steps to change the roleRef.

  1. Delete the cluster role binding.
    kubectl delete clusterrolebinding restricted-psp-user
  2. Create or update restricted-psp-user cluster role binding.

ClusterRoleBinding restricted-psp-user does not have expected subjects

If this change was intentional, you must first determine if applications can run properly with the cluster role binding.

Then, create or update restricted-psp-user cluster role binding.

Creating or updating the privileged-psp-user ClusterRoleBinding

Create or update privileged-psp-user cluster role binding by running the following command.

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: privileged-psp-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ibm-privileged-psp-user
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated
EOF

Creating the restricted-psp-user cluster role binding

Create or update the restricted-psp-user ClusterRoleBinding by running the following command.

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: restricted-psp-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ibm-restricted-psp-user
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated
EOF