replacement not working with overlay configmap values

[ferronsw]

Describe the bug

I have the following base/kustomization.yaml where I'm doing some replacements:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ex-secret.yaml

configMapGenerator:
- name: test
  envs:
    - env.properties

replacements:
- source:
    name: test
    kind: ConfigMap
    fieldPath: data.KEY
  targets:
    - select:
        kind: ExternalSecret
        name: prime-manager
      fieldPaths:
        - "spec.data.[secretKey=PRIME_MANAGER_CONSOLE_DB_password].remoteRef.key"
- source:
    name: test
    kind: ConfigMap
    fieldPath: data.KEY2
  targets:
    - select:
        kind: ExternalSecret
        name: prime-manager
      fieldPaths:
        - "spec.data.[secretKey=PRIME_MANAGER_WEB_DB_password].remoteRef.key"
- source:
    name: test
    kind: ConfigMap
    fieldPath: data.KEY3
  targets:
    - select:
        kind: ExternalSecret
        name: prime-manager
      fieldPaths:
        - "spec.data.[secretKey=PRIME_MANAGER_PARAMS_admin_password].remoteRef.key"

The base/env.properties file in the base contains the following properties and values:

KEY=test
KEY2=test2
KEY3=test3

base/ex-secret.yaml file for reproduce:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: prime-manager
  labels:
    app.kubernetes.io/name: prime-manager
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: keyvault-secret-store

  data:
  - secretKey: PRIME_MANAGER_CONSOLE_DB_password
    remoteRef:
      key: prime-manager-console-postgres-password-staging
  - secretKey: PRIME_MANAGER_WEB_DB_password
    remoteRef:
      key: prime-manager-web-postgres-password-staging
  - secretKey: PRIME_MANAGER_PARAMS_admin_password
    remoteRef:
      key: prime-manager-ferrontest-admin-password-staging

Now I have created an overlay overlay/kustomization.yaml to overwrite the values like

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../base

configMapGenerator:
- name: test
  behavior: replace
  envs:
    - env.properties

overlay/env.properties:

KEY=testen
KEY2=testen2
KEY3=testen3

File tree:

C:.
├───base
│       env.properties
│       ex-secret.yaml
│       kustomization.yaml
│
└───overlay
        env.properties
        kustomization.yaml

Expected output

I would expect testen, testen2 and testen3 to show up in the output.

apiVersion: v1
data:
  KEY: testen
  KEY2: testen2
  KEY3: testen3
kind: ConfigMap
metadata:
  name: test-hbtgc5f7dm
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  labels:
    app.kubernetes.io/name: prime-manager
  name: prime-manager
spec:
  data:
  - remoteRef:
      key: testen
    secretKey: PRIME_MANAGER_CONSOLE_DB_password
  - remoteRef:
      key: testen2
    secretKey: PRIME_MANAGER_WEB_DB_password
  - remoteRef:
      key: testen3
    secretKey: PRIME_MANAGER_PARAMS_admin_password
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: keyvault-secret-store

Actual output

But I'm still getting the base values in the output.

apiVersion: v1
data:
  KEY: testen
  KEY2: testen2
  KEY3: testen3
kind: ConfigMap
metadata:
  name: test-hbtgc5f7dm
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  labels:
    app.kubernetes.io/name: prime-manager
  name: prime-manager
spec:
  data:
  - remoteRef:
      key: test
    secretKey: PRIME_MANAGER_CONSOLE_DB_password
  - remoteRef:
      key: test2
    secretKey: PRIME_MANAGER_WEB_DB_password
  - remoteRef:
      key: test3
    secretKey: PRIME_MANAGER_PARAMS_admin_password
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: keyvault-secret-store

Kustomize version

v4.5.7

Platform

Windows

[k8s-ci-robot]

@ferronsw: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[annasong20]

Hi @ferronsw,

I get the same behavior on my mac, but this isn't a bug. Kustomize is designed to run the generators and transformers in each layer only once on the resources available at the time of run. In your case, replacements ran in the base, but won't run again, even though you've changed configMapGenerator in the overlay.

To get an "up-to-date" ExternalSecret, you can move the replacements to the top-most overlay or include it in each overlay.

[nourspace]

I'm facing the same situation. I guess it is intuitive to override configs in overlays and assume that the base replacements will do their job as intended. Including the replacements in each overlay removes the whole point of having overlays to start with. Or am I getting this completely wrong?

I was so excited hoping that I could finally have a way to provide simple config overrides in overlays and let the base take care of distributing these values. It seems that Kustomize is not just disallowing templating, but also anything close to it xD

[nourspace]

I think I just came much closer to what is intended.

The base should be divided into resources and transformers. Each folder has its own kustomization.yaml that loads resources and transformers manifests.

base/resources/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ex-secret.yaml

base/transformers/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - config-replacements.yaml

base/transformers/config-replacements.yaml

apiVersion: builtin
kind: ReplacementTransformer
metadata:
  name: not-important-patch-ingress-mcert
replacements:
  # ... they go here

In the overlay, we would then import resources, generate config, and finally run transformers.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../base/resources

configMapGenerator:
- name: test
  behavior: replace
  envs:
    - env.properties

transformers:
- ../base/transformers

The overlay is minimal and works as expected since transformers run last. There is part of the documentation that mentions this layout, but no concrete example of using the built-ins.
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/#transformed-transformers

[ferronsw]

@nourspace I have almost the same solution in place now.

Base kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

configMapGenerator:
- name: prime-manager-config
  envs:
    - env.properties

The replacements are located in the base folder.

overlay kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

configMapGenerator:
- name: prime-manager-config
  behavior: replace
  envs:
    - env.properties

replacements:
  - path: ../../base/replacements.yaml 

The only thing is you need to specify:

--load-restrictor LoadRestrictionsNone after the kustomize command, because the replacement in located the base folder.