[Bug]: Failed to remove orphan files after dropping a partition field

[rfyu]

What happened?

After dropping a parition field, if we use the latest PartitionSpec to build DataFile or DeleteFile under the historical partition, the following exception will be throw.

Affects Versions

master

What engines are you seeing the problem on?

AMS

How to reproduce

1. create table ... partition by (bucket(id,16))
2. append data
3. drop partition field bucket(id,16)
4. expire snapshot
5. add some orphan files manually(backup deleted old files)
6. Wait for ams to clean up orphan files

Relevant log output

2024-03-03 02:42:42,237 ERROR [async-orphan-files-cleaning-executor-12] [com.netease.arctic.server.table.executor.OrphanFilesCleaningExecutor] [] - hms_ctripdi.dw_ctripdi.o_ta_tickect_choice(tableId=8738070) failed to clean orphan file
java.lang.IllegalArgumentException: Wrong class, expected java.lang.Long, but was java.lang.Integer, for object: 24
    at org.apache.iceberg.PartitionData.get(PartitionData.java:117) ~[iceberg-core-1.4.3.jar:?]
    at org.apache.iceberg.util.StructProjection.get(StructProjection.java:209) ~[iceberg-api-1.4.3.jar:?]
    at org.apache.iceberg.DataFiles.copyPartitionData(DataFiles.java:55) ~[iceberg-core-1.4.3.jar:?]
    at org.apache.iceberg.DataFiles$Builder.withPartition(DataFiles.java:248) ~[iceberg-core-1.4.3.jar:?]
    at com.netease.arctic.scan.TableEntriesScan.buildDataFile(TableEntriesScan.java:326) ~[amoro-core-0.7.0-SNAPSHOT.jar:?]
    at com.netease.arctic.scan.TableEntriesScan.buildContentFile(TableEntriesScan.java:302) ~[amoro-core-0.7.0-SNAPSHOT.jar:?]
    at com.netease.arctic.scan.TableEntriesScan.lambda$entries$0(TableEntriesScan.java:241) ~[amoro-core-0.7.0-SNAPSHOT.jar:?]
    at org.apache.iceberg.io.CloseableIterable$7$1.next(CloseableIterable.java:202) ~[iceberg-api-1.4.3.jar:?]
    at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:65) ~[iceberg-api-1.4.3.jar:?]
    at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:49) ~[iceberg-api-1.4.3.jar:?]
    at com.netease.arctic.server.utils.IcebergTableUtil.getAllContentFilePath(IcebergTableUtil.java:100) ~[amoro-ams-server-0.7.0-SNAPSHOT.jar:?]
    at com.netease.arctic.server.optimizing.maintainer.IcebergTableMaintainer.orphanFileCleanNeedToExcludeFiles(IcebergTableMaintainer.java:328) ~[amoro-ams-server-0.7.0-SNAPSHOT.jar:?]
    at com.netease.arctic.server.optimizing.maintainer.IcebergTableMaintainer.cleanContentFiles(IcebergTableMaintainer.java:288) ~[amoro-ams-server-0.7.0-SNAPSHOT.jar:?]
    at com.netease.arctic.server.optimizing.maintainer.IcebergTableMaintainer.cleanOrphanFiles(IcebergTableMaintainer.java:123) ~[amoro-ams-server-0.7.0-SNAPSHOT.jar:?]
    at com.netease.arctic.server.table.executor.OrphanFilesCleaningExecutor.execute(OrphanFilesCleaningExecutor.java:61) ~[amoro-ams-server-0.7.0-SNAPSHOT.jar:?]
    at com.netease.arctic.server.table.executor.BaseTableExecutor.executeTask(BaseTableExecutor.java:84) ~[amoro-ams-server-0.7.0-SNAPSHOT.jar:?]
    at com.netease.arctic.server.table.executor.BaseTableExecutor.lambda$scheduleIfNecessary$4(BaseTableExecutor.java:95) ~[amoro-ams-server-0.7.0-SNAPSHOT.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_291]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_291]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_291]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_291]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_291]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_291]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct