Kustomize doesn't pass namespace to helmCharts, resulting in inconsistent output

[Skaronator]

What happened?

When using Kustomize with Helm charts, the namespace defined at the top level in the kustomization.yaml file is not correctly propagated to the helmCharts releases. As a result, the generated YAML files contains inconsistent values.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmGlobals:
  chartHome: ./charts

namespace: the-actual-namespace   # <- defines namespace for all resources

helmCharts:
- name: service
  releaseName: service
  valuesFile: values.yaml

Helm Chart Template

apiVersion: v1
kind: Service
metadata:
  name: the-bug
  namespace: {{ .Release.Namespace }}
  annotations:
    this-service-is-deployed-in-namespace: {{ .Release.Namespace }}

Result

$ kustomize build . --enable-helm
apiVersion: v1
kind: Service
metadata:
  annotations:
    this-service-is-deployed-in-namespace: default # <- {{ .Release.Namespace }} still points to "default"
  name: the-bug
  namespace: the-actual-namespace                  # <- Namespace overwritten by Kustomization to the correct value

This inconsistency can cause issues when deploying resources, such as Bitnami Minio Chart, which relies on the {{ .Release.Namespace }} to dynamically create the service URL.

https://github.com/bitnami/charts/blob/e8999fedbd8e35c96099aa44d3ca4ea947bfbe22/bitnami/minio/templates/distributed/statefulset.yaml#L150

What did you expect to happen?

I expected that the namespace defined in the top level of the kustomization.yaml file would be passed down to all helmCharts releases.

When a Helm chart entry already specifies a namespace, that namespace should take precedence.

How can we reproduce it (as minimally and precisely as possible)?

Repository to reproduce it is available here: https://github.com/Skaronator/kustomize-namespace-bug

$ tree
.
├── charts
│   └── service
│       ├── Chart.yaml
│       ├── templates
│       │   └── service.yaml
│       └── values.yaml
├── kustomization.yaml
├── README.md
└── values.yaml

4 directories, 6 files

# ./kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmGlobals:
  chartHome: ./charts

namespace: the-actual-namespace   # <- defines namespace for all resources

helmCharts:
- name: service
  releaseName: service
  valuesFile: values.yaml

# ./charts/service/Chart.yaml
apiVersion: v2
name: service
description: A Helm chart for Kubernetes
type: application
version: 1.0.0
appVersion: "1.0.0"

# ./charts/service/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: the-bug
  namespace: {{ .Release.Namespace }}
  annotations:
    this-service-is-deployed-in-namespace: {{ .Release.Namespace }}

Expected output

$ kustomize build . --enable-helm
apiVersion: v1
kind: Service
metadata:
  annotations:
    this-service-is-deployed-in-namespace: the-actual-namespace # <- {{ .Release.Namespace }} should be correctly set
  name: the-bug
  namespace: the-actual-namespace                               # <- Namespace overwritten by Kustomization to the correct value

Actual output

$ kustomize build . --enable-helm
apiVersion: v1
kind: Service
metadata:
  annotations:
    this-service-is-deployed-in-namespace: default # <- {{ .Release.Namespace }} still points to "default"
  name: the-bug
  namespace: the-actual-namespace                  # <- Namespace overwritten by Kustomization to the correct value

Kustomize version

v5.3.0

Operating system

Linux

[koba1t]

@Skaronator
Thanks for reporting an issue.

namespace transformer only changes the namespace field in the k8s resource, And I think helmCharts generates k8s yaml when that is called.
This means your helm template is executed and only contains generated k8s yaml when the namespace transformer is called.
I think {{ .Release.Namespace }} is only enabled in helm, and Kustomize can't find that namespace place.

So please consider using the helmCharts.namespace field.

helmCharts:
- name: service
  releaseName: service
  valuesFile: values.yaml
  namespace: the-actual-namespace 

/triage out-of-scope

[Skaronator]

Hi @koba1t, thanks for taking your time.

The current implementation causes inconsistent YAML files. Defining the namespace within helmCharts is often not possible. For example, when ArgoCD dynamically sets the namespace in kustomize.

I'd expect if helmCharts.namespace is not defined, kustomize should set that value to the top level namespace.

[allandegnan]

This really seems closer to a bug @koba1t.

Setting the namespace at the kustomize level inflates the helm chart, rewrites all the objects to the namespace, but then misses things like connection strings, thus deploying the chart in a broken state.

Example, the airflow helm chart here:

https://github.com/apache/airflow/blob/main/chart/templates/secrets/metadata-connection-secret.yaml#L24

You can see it relies on {{ .Release.Namespace }} as part of the default connection to its database, thus if you don't match the namespaces between kustomize and the helmChart, then you have something that looks functional but isn't.

[panteparak]

I too agree that this looks to be a bug on Kustomize side. I believe the correct strategy would be

if a namespace is define in kustomization.yml like below

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: the-actual-namespace

helmCharts:
- name: service
  releaseName: service
  valuesFile: values.yaml

kustomize should not transform the namespace, but instead let helm do the transformation.

This would ensure that any built-in variable such as {{.Release.Namespace}} is being rendered correctly.
It might be worth to support specifying namespace at helm chart levels too, such that It allows the ability to override namespace to specific chart, like so.

helmCharts:
- name: service
  releaseName: service
  valuesFile: values.yaml
  namespace: the-actual-namespace-2

This looks to be partially related to #4593

[aviadhaham]

Hi, I found a workaround solution, to at least not repeat yourself in the patch resources.

Instead of adding the namespace to the patch, I only added the name of the deployment, and the things I want kustomiza to patch, and it worked.

Ref kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: amp-airflow-prod
helmCharts:
  - name: airflow
    repo: https://charts.bitnami.com/bitnami
    releaseName: amp-airflow
    namespace: amp-airflow-prod
    version: 17.2.4
    valuesFile: values.yml
resources:
  - secrets.yml

patches:
  - path: amp-airflow-exporter-patch.yml
    target:
      kind: Deployment
      name: amp-airflow-exporter
      # namespace: amp-airflow-prod

The patch:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: amp-airflow-exporter
  # namespace: amp-airflow-prod
spec:
  template:
    spec:
      containers:
        - name: airflow-exporter
          env:
            - name: TEST
              value: "test"

The patch was applied to the right resource, and the env var is there :)

[ThomasSteinbach]

I have the exact same behavior with this namespace...
https://github.com/ansible/awx-operator/blob/devel/.helm/starter/templates/awx-deploy.yaml#L7

... when deploying it as kustomization with argocd.

[QuadmanSWE]

I get this problem when deploying vernemq helm chart with kustomize through argocd. All of its resources except a service account, a role, and a rolebinding end up in the right place.

https://vernemq.github.io/docker-vernemq

[tsloughter]

namespace transformer only changes the namespace field in the k8s resource, And I think helmCharts generates k8s yaml when that is called. This means your helm template is executed and only contains generated k8s yaml when the namespace transformer is called. I think {{ .Release.Namespace }} is only enabled in helm, and Kustomize can't find that namespace place.

The desire isn't for kustomize to do anything with .Release.Namespace but instead to set a default for helmCharts.namespace to the given namespace. Doing what you suggest:

helmCharts:
- name: service
  releaseName: service
  valuesFile: values.yaml
  namespace: the-actual-namespace 

Means any overlay like:

kind: Kustomization
namespace: other-namespace
...

Will still result in the use of the-actual-namespace in the helm templates.

[Skaronator]

@koba1t

Given the numerous comments and the 13 thumbs up on my original issue, could you please reconsider if it's still out-of-scope and update the labels accordingly?

Thanks

[tsloughter]

On slack someone explained why kustomize can't pass the namespace in the Kustomization when not defined in the helm chart definition but I didn't completely understand. Something around the order things are done.

I also know @koba1t is working on new helm support #4401 (comment) and curious if it'll be done there.

[koba1t]

Hmm.. I am worried that this feature may have a huge change due to this function, which means any transformer output affects another transformer...
But I understand many people want to do it.

/kind feature
/area helm
/triage under-consideration
/priority important-longterm

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[Skaronator]

/remove-lifecycle rotten

[wibed]

selfish bump

[hobbit44]

Just ran headlong into this. Because of the use of the helm binary and it not passing the namespace through. It picked up the namespace defined in my kube config file as the .Release.Namespace when creating resources.

So as an example, using the following kustomization.yaml:

namespace: cert-manager
helmCharts:
  - name: cert-manager
    repo: https://charts.jetstack.io

You can use kubectl config set-context --current --namespace something-else to change the context namespace. You can then see the problem like so:

❯ kustomize build --enable-helm | yq ea 'select(.kind == "ValidatingWebhookConfiguration").metadata.annotations'
cert-manager.io/inject-ca-from-secret: something-else/cert-manager-webhook-ca

koba1t As a quick patch to stop the unexpected behaviour, would it be possible to just adding the --namespace arg to the call to the helm binary with the namespace arg as the parameter? That way even if you dont pass the release namespace, it would save any wierdness happening.

[koba1t]

@hobbit44

koba1t As a quick patch to stop the unexpected behaviour, would it be possible to just adding the --namespace arg to the call to the helm binary with the namespace arg as the parameter? That way even if you dont pass the release namespace, it would save any wierdness happening.

I think we already have a namespace parameter on helmCharts.
Could you try to use this parameter to fix your problem?
https://kubectl.docs.kubernetes.io/references/kustomize/builtins/#_helmchartinflationgenerator_