HNC: if a RoleBinding is quickly deleted and recreated with a new Role, HNC may fail to update it

[adrianludwin]

I was recently reminded that RoleBindings can't have their Roles changed after they're created. This suggests that if we quickly recreate an RB in an ancestor namespace - fast enough that by the time that HNC notices, the new RB exists - HNC could see it as an update (not a delete + create) and attempt to update the RBs in all descendant namespaces. This will fail.

We should add a test that exhibits this issue and find a way to fix it.

I don't think this needs to be backported to HNC v0.4. I'd imagine that in most cases, HNC will notice that the RB has been deleted before it's been recreated, even if the change is being made by some automated tool. But I'll update the release notes to indicate this issue.

[adrianludwin]

/assign @yiqigao217

[adrianludwin]

/unassign @yiqigao217
/assign @GinnyJI

Here's a great bug for you. The problems here will be:

• How do we reproduce this issue? It might need a new e2e test where we delete the HNC deployment.
• How do we fix it? What do we do if "Update" fails because some objects can't be updated (they can only be deleted and recreated)?

The function where objects get updated is: https://github.com/kubernetes-sigs/multi-tenancy/blob/43f9bd442f85d20707b19e9b436a577b000b128d/incubator/hnc/internal/reconcilers/object.go#L515

[adrianludwin]

For the RoleBinding being immutable (i.e. unchangable), see https://kubernetes.io/docs/reference/access-authn-authz/rbac/#clusterrolebinding-example:

After you create a binding, you cannot change the Role or ClusterRole that it refers to. If you try to change a binding's roleRef, you get a validation error. If you do want to change the roleRef for a binding, you need to remove the binding object and create a replacement.