[Flaky test] Scheduler when Scheduling workloads on clusterQueues when Hold LocalQueue at startup Should admit workloads according to their priorities

[mimowo]

/kind flake
What happened:

Failed test: https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-kueue-test-integration-main/1891370482536550400

What you expected to happen:

no failures

How to reproduce it (as minimally and precisely as possible):

Run CI

Anything else we need to know?:

{Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/kubernetes-sigs/kueue/test/util/util.go:324 with:
Not enough workloads are pending
Expected
    <int>: 2
to equal
    <int>: 3 failed [FAILED] Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/kubernetes-sigs/kueue/test/util/util.go:324 with:
Not enough workloads are pending
Expected
    <int>: 2
to equal
    <int>: 3
In [It] at: /home/prow/go/src/kubernetes-sigs/kueue/test/integration/singlecluster/scheduler/scheduler_test.go:565 @ 02/17/25 06:28:13.731
}

Environment:

• Kubernetes version (use kubectl version):
• Kueue version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[mimowo]

/kind flake

[KPostOffice]

export GINKGO_ARGS='--focus "Scheduler when Scheduling workloads on clusterQueues when Hold LocalQueue at startup Should admit workloads according to their priorities"'
export INTEGRATION_TARGET='test/integration/singlecluster/scheduler'
count=0

while make test-integration ;
do
  count=$[$count + 1]
  echo $count
done

ran this trying to reproduce flake ~100 times and have not been able to reproduce locally

[mimowo]

ran this trying to reproduce flake ~100 times and have not been able to reproduce locally

Thanks for looking at the issue. The other tactics I use for reproducing the issues locally:

• add some stress (./bin/stress --cpu=N)
• add --race to GINKGO_ARGS
• recompile with instrumentation, here are the steps for k8s tests, but it should be possible to adapt for Kueue:

# install stress command
go install golang.org/x/tools/cmd/stress@latest

# recompile with race instrumentation
go test ./pkg/controller/job -race -c

# run (it loops and reports failures)
stress ./job.test -test.run TestJobApiBackoffReset