Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Public IP Address orphaned when updating LoadBalancer type Service to a user-provided PIP from another RG #8113

Open
jfchevrette opened this issue Jan 22, 2025 · 4 comments · May be fixed by #8419
Open

Public IP Address orphaned when updating LoadBalancer type Service to a user-provided PIP from another RG #8113

jfchevrette opened this issue Jan 22, 2025 · 4 comments · May be fixed by #8419
Assignees
@nilo19
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@jfchevrette
Copy link

jfchevrette commented Jan 22, 2025 •
edited
Loading

What happened:

Updating an existing LoadBalancer type kubernetes Service to bind it to a user-created PIP does not release the existing PIP when that PIP is in a different resource group

What you expected to happen:

The PIP which was previously bound to the service should be released and removed

How to reproduce it (as minimally and precisely as possible):

  1. Set variables to use in future commands
export TEST_RG=test-orphaned-pip
  1. Create a resource-group to test with
az group create \
  --location eastus \
  --resource-group $TEST_RG
  1. Create an AKS cluster
az aks create \
  --resource-group $TEST_RG \
  --name my-cluster \
  --node-count 1 \
  --generate-ssh-keys
  1. Export cluster credentials to a temporary file and set KUBECONFIG
az aks get-credentials \
  --resource-group $TEST_RG \
  --name my-cluster \
  -f /tmp/my-cluster.kubeconfig

export KUBECONFIG=/tmp/my-cluster.kubeconfig
  1. Create a test loadbalancer service on the cluster
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: my-loadbalancer-service
spec:
  ports:
  - port: 80
  type: LoadBalancer
EOF
  1. Observe the service's external IP address is created and assigned
  2. Get the cluster's identity principal and note it
az aks show \
  --resource-group $TEST_RG \
  --name my-cluster \
  --query "identity.principalId" \
  --output tsv
  1. Create a PIP in $TEST_RG
az network public-ip create \
  --resource-group $TEST_RG \
  --name my-loadbalancer-service
  1. Get the PIP's ID
az network public-ip show \
  --resource-group $TEST_RG \
  --name my-loadbalancer-service \
  --query "id" \
  --output tsv
  1. Assign the Network Contributor role to the cluster identity principal onto the PIP
az role assignment create \
  --assignee <PRINCIPAL_ID> \
  --role "Network Contributor" \
  --scope <PIP_ID>
  1. Annotate the loadbalancer service so that it is reassigned to our user-created PIP
kubectl annotate svc my-loadbalancer-service \
  service.beta.kubernetes.io/azure-load-balancer-resource-group=$TEST_RG \
  service.beta.kubernetes.io/azure-pip-name=my-loadbalancer-service
  1. Observe that the service has been reconciled and now has our PIP assigned
  2. Observe that the original PIP remains untouched in the cluster's node resource group

Note: If the user-created PIP is created in the cluster's node resource group, the PIP gets released properly as expected. However according to documentation it is a bad practice to create resources in this RG

Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:
@jfchevrette jfchevrette added the kind/bug Categorizes issue or PR as related to a bug. label Jan 22, 2025
@jfchevrette
Copy link
Author

jfchevrette commented Jan 23, 2025 •
edited
Loading

I've been looking at the code and while this is not my area of expertise, I'm starting to think the issue lies in that the code looks at the Service object, sees the annotation service.beta.kubernetes.io/azure-load-balancer-resource-group and operates on the Resource Group specified, disregarding the possibility that prior to the Service object update the annotation may not have been there and thus it never looks at the PIPs in the cluster's Resource Group

The breadcrumbs I followed are as follow:

@jfchevrette
Copy link
Author

I also want to reiterate: If the Service is deleted after it has been annotated and reconciled with the user-provided PIP from the other RG, the original PIP remains and is never deleted

@jfchevrette
Copy link
Author

Since k8s's cloud-provider interface does not pass the old object to the controller, I think one way to solve this would be to also consider az.ResourceGroup in addition to the resource group that is specified in the annotation when enumerating and iterating over PIPs in az.reconcilePublicIPs. This would make it so that even though we're looking for a PIP in the specified Resource Group, we also consider the cluster's Resource Group to determine whether PIPs need to be removed

@nilo19
Copy link
Contributor

nilo19 commented Jan 30, 2025

/assign

@nilo19 nilo19 linked a pull request Feb 24, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nilo19 nilo19

Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Remove managed pip if switched to another pip in a different res… nilo19/cloud-provider-azure
2 participants
@jfchevrette @nilo19