Create an e2e customconfig test that tests WaitForPodsReady

[PBundyra]

What would you like to be added:
An e2e tests with falls under the customconfigs/ directory that would test WaitForPodsReady feature. We already have integration tests for WaitForPodsReady but they don't run job controller, so it would be nice to have some scenarios with job controller covered.

Scenarios:

• .timeout is surpassed before all pods reach readiness - the workload is evicted, and requeued
• during runtime a pod fails, and .recoveryTimeout is surpassed - the workload is evicted and requeued
• during runtime a pod fails, and the recovery pod is scheduled before .recoveryTimeout is surpassed - the workload continues to run

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Code changes

The artifacts should be linked in subsequent comments.

[mimowo]

+1, waitForPodsReady is quite an important feature for our users for failure recovery. I think it deserves at least a baseline e2e test