[Improvement]: Improve major optimizing: compact segment files to target size

[zhongqishang]

Search before asking

• I have searched in the issues and found no similar issues.

Backgroud

The continuously optimized Iceberg format table has a large number of segment files (16m~128m), and they are all close to the lower limit. A large number of small files will bring the following impacts:

• Too many small files will occupy a lot of memory and directly affect the performance of NameNode.
• Will cause query performance loss to the query engine (SparkSQL, Trino)

What would you like to be improved?

Most file sizes are close to target size.

How should we improve?

Fix current segment planner logic. Design Doc

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct