[API server] use loadbalancer to expose ingress by default #4926
Conversation
Signed-off-by: Aylei <[email protected]>
Signed-off-by: Aylei <[email protected]>
Signed-off-by: Aylei <[email protected]>
Signed-off-by: Aylei <[email protected]>
Signed-off-by: Aylei <[email protected]>
| helm upgrade --install -n $NAMESPACE $RELEASE_NAME skypilot/skypilot-nightly --devel \ | ||
| --set ingress.nodePortEnabled=false |
There was a problem hiding this comment.
Will this also delete the old NodePort service?
There was a problem hiding this comment.
Yes, this command is solely for deleting the old NodePort service
| # Use ClusterIP here to disable the LoadBalancer created by nginx. | ||
| # If not using NodePort, set this to LoadBalancer. | ||
| type: ClusterIP | ||
| type: LoadBalancer |
There was a problem hiding this comment.
IIUC, in the future this can directly be set to NodePort for users who want to use nodeport? If so, we should probably remove the ingress.nodePortEnabled right before 0.9.0 release.
There was a problem hiding this comment.
Same for charts/skypilot/templates/ingress-nodeport.yaml - that file should be removed before 0.9.0. Good to have a TODO there.
There was a problem hiding this comment.
Yes, I think setting ingress-nginx.controller.service.* is recommended after this PR for new deployments. But it requires the similar migration steps for legacy deployments to abandon the additional node port service:
- Enable the nginx NodePort service, since 30050 and 30051 have been used, the new service should pick up new ports;
- Ask clients to migrate to the new server URL (with new ports)
- Set
ingress.nodePortEnabled=falseto delete the old NodePort service
Not sure whether we should keep the additional node port service to simplify upgrades for nightly users, but we can document the above steps and leave the choice to users. It is reasonable to take no action if they are happy with current setup.
| # Specific nodePort to use for the ingress controller | ||
| # If not set, Kubernetes will assign random ports in the NodePort range (default 30000-32767) | ||
| # Deprecated: use ingress-nginx.controller.service.nodePorts.http instead |
There was a problem hiding this comment.
Can we add a note to remove this before 0.9.0 release? Same for the nodePortEnabled field.
Co-authored-by: Romil Bhardwaj <[email protected]>
close #4920
Motivation of changing the defaults: LoadBalancer (if available) is more stable and secure compared to NodePort, correspondingly better UX and lower support costs.
Note:
LoadBalancermight be unavailable on some k8s infra, we still make it default because it is the de-facto way to expose L4 endpoint. As an example, nginx-ingress also useLoadBalanceras default service type https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginxTested (run the relevant ones):
bash format.sh/smoke-test(CI) orpytest tests/test_smoke.py(local)/smoke-test -k test_name(CI) orpytest tests/test_smoke.py::test_name(local)/quicktest-core(CI) orconda deactivate; bash -i tests/backward_compatibility_tests.sh(local)