NCCL integration tests

[annuay-google]

What?

NCCL integration tests for A+ machine families

Why?

1. NCCL tests help validate functionality and baseline performance of the cluster
2. Regressions have crept in due to our current integration tests only validating provisioning and nvidia-smi. An example was the change to rename kubernetes network objects was not caught for breaking the NCCL tests we ship to customers

Testing

Validated by running the tests via cloud build:
A3 Ultra: https://pantheon.corp.google.com/cloud-build/builds;region=global/234a1ddf-b194-4129-a640-0b5b081373a1?e=13803378&mods=monitoring_api_prod&project=hpc-toolkit-dev
A3 Mega: https://pantheon.corp.google.com/cloud-build/triggers;region=global/edit/e2eb58fd-613e-4f32-b107-debed1cd573b?e=13803378&mods=monitoring_api_prod&project=hpc-toolkit-dev
A3 High: https://pantheon.corp.google.com/cloud-build/builds;region=global/6ee8fc8f-f718-4446-b11b-01bfa8cc647d?e=13803378&mods=monitoring_api_prod&project=hpc-toolkit-dev

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #

[annuay-google]

No longer required since the subnet names are now prefixed with deployment-name

[SwarnaBharathiMantena]

LGTM