Create EBS CSI Driver scale-test tool

[AndrewSirenko]

What type of PR is this?

/kind documentation

What is this PR about? / Why do we need it?

Validate that the EBS CSI Driver is capable of managing volumes at the scale of your largest EKS clusters with our scale-test tool. See /hack/ebs-scale-test/README.md for more info.

Karpenter cluster-type, more scalability test-types, and more scalability test observability features coming soon.

Scope of this PR: One-step run pre-allocated scale-sts scalability tests.

Running ./scale-test create && ./scale-test run && ./scale-test clean with no extra environment variables set will:

1. Create an EKS cluster
2. Install EBS CSI Driver with metrics enabled
3. Deploy StorageClass and StatefulSet, then scale that sts to 1000 replicas (which will provision and publish 1000 volumes)
4. Delete sts (unpublishing and deleting volumes)
5. Export metrics, ebs-plugin logs, driver deployment + daemonset yaml, and other test artifacts to a local directory and an S3 bucket
6. Clean up all scalability test resources from your AWS account

How was this change tested?

Revision 2 edit: Ran through scale-test create, run, and clean loop again. Also manually tested extra features, the results of which I included in the review comments of those features. Thanks.

Please follow the README.md to run these scale tests yourself.

Timing:

Cluster Setup: ~18 min
Run: ~8 min for 1000 replica test
Cluster Cleanup: ~10 min

❯ ./scale-test setup && ./scale-test run && ./scale-test clean
Deploying EKS cluster. See configuration in /tmp/tmp.FEdux6Eoal/cluster-config.yaml
2025-01-16 14:17:05 [ℹ]  eksctl version 0.194.0-dev+02ef28ee3.2024-10-22T18:42:34Z
...
Applying /workplace/andsirey/aws-ebs-csi-driver/hack/ebs-scale-test/helpers/scale-test/scale-sts-test/scale-sts.yaml. Exported to /tmp/tmp.FGNUXsiFTA/scale-manifest.yaml
...
partitioned roll out complete: 101 new pods have been updated...
Deleting StatefulSet
statefulset.apps "ebs-scale-test" deleted
storageclass.storage.k8s.io "ebs-scale-test" deleted
Waiting for all PVs to be deleted
101 PVs still exist, waiting...
...
Exporting everything in /tmp/tmp.FGNUXsiFTA to S3
Metrics exported to s3://ebs-scale-tests/ebs-scale-pre-allocated-scale-sts-101-2025-01-16T14:35UTC/
2025-01-16 14:44:57 [ℹ]  deleting EKS cluster "ebs-scale-pre-allocated"
2025-01-16 14:52:04 [✔]  all cluster resources were deleted

Does this PR introduce a user-facing change?

Add `scale-test` tool for running EBS CSI Driver scalability tests

[github-actions]

Code Coverage Diff

This PR does not change the code coverage

[ElijahQuinones]

/lgtm

[ElijahQuinones]

/lgtm

[AndrewSirenko]

/retest

Flake: Error: validation failed: wait time exceeded during validation

[ElijahQuinones]

/approve

Ran through scale tests locally and confirmed they run as intended send metrics and logs to S3 and clean up all resources when done.

./scale-test create && ./scale-test run && ./scale-test clean;
...
2025-01-31 16:11:46 [✔]  saved kubeconfig as "/home/elijahlq/.kube/config"
...
2025-01-31 16:11:46 [✔]  all EKS cluster resources for "ebs-scale-pre-allocated" have been created
...
2025-01-31 16:33:15 [✔]  all cluster resources were deleted

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ElijahQuinones

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ElijahQuinones]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment