Enable sorting of queue items based on a custom timestamp extracted from an annotation

[gbenhaim]

What would you like to be added:

Enable sorting of queue items based on a custom timestamp extracted from an annotation.

Why is this needed:

Queuing plain pods created by Tekton can be highly beneficial for sharing quotas and achieving more consistent scheduling (FIFO based on the pod's creation timestamp). A short demo on this topic can be found here.

The experience could be further improved by allowing pods in the queue to be sorted based on the creation timestamp of the Tekton PipelineRun they belong to.

This would prioritize pods associated with "older" pipelines, helping them complete first and reducing timeouts.

Completion requirements:

1. Kueue should be able to read a timestamp from an annotation set on the workload and use it for sorting workloads in the queue.

2. It should be possible to copy annotations from the Job to the Workload, similar to how labels can be copied.

3. Tekton should set the creation timestamp of the PipelineRun as an annotation on the pods belonging to that PipelineRun (this should be implemented in Tekton).

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[kannon92]

I am a bit confused.

https://cloud.google.com/kubernetes-engine/docs/tutorials/kueue-intro

This seems to say that we have support of FIFO for queueing.

[kannon92]

https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/#queueing-strategy

[tenzen-y]

We have received similar requests many times. However, we could not accept it since such custom timestamp could violate fairness. All workloads should be fair under the Kueue algorithms.

Such annotation allows evil users to prioritize their tasks.