[Feature]: Support Iceberg on S3 or other non-hadoop storage system.

[baiyangtx]

Description

Currently, Arctic only supports running within the Hadoop ecosystem, and it cannot be used effectively with other storage types like Glue + S3. We should support more types of storage systems.

Use case/motivation

We should support managing Iceberg tables with Arctic in S3 + Glue scenarios, optimize it for better performance, and enable features like cleaning up orphan files.

Describe the solution

1. ArcticFileIO interface should be refactored to reduce its strong dependency on Hadoop.
2. The Optimizer logic needs to be retested to ensure that it supports non-Hadoop systems.
3. The logic for cleaning orphan files needs to be adapted to non-file system scenarios. In extreme cases where the storage system does not support list operations, this functionality may need to be downgraded or even unavailable.
4. Due to the fact that cloud object storage is usually billed based on the number of API calls, the logic of the optimizer checker needs to be adjusted to avoid generating unnecessary costs from frequent checks.

Related issues

• [Subtask]: Remove dir operation from ArcticFileIo in order to support S3 storage. #1362

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct