Wait for jobset operator on starting e2e tests.

[mbobrovskyi]

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

Wait for jobset operator ready before start e2e tests.

Which issue(s) this PR fixes:

Fixes #2428

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue ready!

Name	Link
🔨 Latest commit	47f733c
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/66741eab850bb000082057a0
😎 Deploy Preview	https://deploy-preview-2456--kubernetes-sigs-kueue.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[mbobrovskyi]

/test all

[mimowo]

I think we might also need to have an update to JobSet to wait for the webhook service with probes as we do here:

kueue/cmd/kueue/main.go

Lines 294 to 320 in 5be1c20

	func setupProbeEndpoints(mgr ctrl.Manager, certsReady <-chan struct{}) {
	defer setupLog.Info("Probe endpoints are configured on healthz and readyz")
	if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
	setupLog.Error(err, "unable to set up health check")
	os.Exit(1)
	}
	// Wait for the webhook server to be listening before advertising the
	// Kueue replica as ready. This allows users to wait with sending the first
	// requests, requiring webhooks, until the Kueue deployment is available, so
	// that the early requests are not rejected during the Kueue's startup.
	// We wrap the call to GetWebhookServer in a closure to delay calling
	// the function, otherwise a not fully-initialized webhook server (without
	// ready certs) fails the start of the manager.
	if err := mgr.AddReadyzCheck("readyz", func(req *http.Request) error {
	select {
	case <-certsReady:
	return mgr.GetWebhookServer().StartedChecker()(req)
	default:
	return errors.New("certificates are not ready")
	}
	}); err != nil {
	setupLog.Error(err, "unable to set up ready check")
	os.Exit(1)
	}
	}

. Otherwise JobSet can say it is available but it is really not.

[mbobrovskyi]

Thanks @mimowo. I've created an issue kubernetes-sigs/jobset#607.

[alculquicondor]

I guess this doesn't fix #2428 but itself, but it gets us closer.

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 1fc671aaa089ff878e2337bb762efa2c1a1fab4a

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, mbobrovskyi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/OWNERS [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alculquicondor]

/cherry-pick release-0.7

[k8s-infra-cherrypick-robot]

@alculquicondor: once the present PR merges, I will cherry-pick it on top of release-0.7 in a new PR and assign it to you.

In response to this:

/cherry-pick release-0.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-infra-cherrypick-robot]

@alculquicondor: #2456 failed to apply on top of branch "release-0.7":

Applying: Wait for jobset operator on starting e2e tests.
Using index info to reconstruct a base tree...
M	test/e2e/multikueue/suite_test.go
M	test/e2e/singlecluster/suite_test.go
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/singlecluster/suite_test.go
Auto-merging test/e2e/multikueue/suite_test.go
CONFLICT (content): Merge conflict in test/e2e/multikueue/suite_test.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Wait for jobset operator on starting e2e tests.
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-0.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[alculquicondor]

@mbobrovskyi can you prepare a cherry-pick for this?