[AMORO-2401] Support reading encrypted iceberg data files

[liaoyt]

Why are the changes needed?

Close #2401.

Brief change log

• As described in [Feature]: Support reading encrypted iceberg data files #2401

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? no

[CLAassistant]

All committers have signed the CLA.

[zhoujinsong]

@liaoyt Thanks a lot for your contribution!

You may need to sign NetEase's CLA before you start to contribute to Amoro, you can conveniently complete the signing via the link: https://cla-assistant.io/NetEase/amoro

[liaoyt]

@liaoyt Thanks a lot for your contribution!

You may need to sign NetEase's CLA before you start to contribute to Amoro, you can conveniently complete the signing via the link: https://cla-assistant.io/NetEase/amoro

thanks for reminding, already signed

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (928159a) 53.52% compared to head (aba6745) 53.51%.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2402      +/-   ##
============================================
- Coverage     53.52%   53.51%   -0.01%     
+ Complexity     4371     4368       -3     
============================================
  Files           517      517              
  Lines         29900    29912      +12     
  Branches       2917     2917              
============================================
+ Hits          16003    16008       +5     
- Misses        12608    12613       +5     
- Partials       1289     1291       +2     

Flag	Coverage Δ
core	53.99% <100.00%> (+<0.01%)	⬆️
trino	50.93% <ø> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shidayang]

@liaoyt Thanks for your contribution，I want to know if you are currently using the encryption feature.

[liaoyt]

@liaoyt Thanks for your contribution，I want to know if you are currently using the encryption feature.

yes, although iceberg has only PlaintextEncryptionManager currently, we implement a custom encryption manager

[shidayang]

@liaoyt Thanks for your contribution，I want to know if you are currently using the encryption feature.

yes, although iceberg has only PlaintextEncryptionManager currently, we implement a custom encryption manager

I am very interested in this area. How does your EncryptionManager work? Currently, the implementation of BaseTable does not provide a custom method. Did you wrap your own Table object?

[liaoyt]

@liaoyt Thanks for your contribution，I want to know if you are currently using the encryption feature.

yes, although iceberg has only PlaintextEncryptionManager currently, we implement a custom encryption manager

I am very interested in this area. How does your EncryptionManager work? Currently, the implementation of BaseTable does not provide a custom method. Did you wrap your own Table object?

There is not custom method provided currently, we just overwrite the TableOperations.encryption() method to provider our custom EncryptionManager.
Also, we patch some of the PR in this project: https://github.com/apache/iceberg/projects/5

[liaoyt]

As for the behavior of reading data file of iceberg table, iceberg already implement it. This PR attempt to keep this behavior the same as iceberg do.

[shidayang]

@liaoyt Thanks for your contribution，I want to know if you are currently using the encryption feature.

yes, although iceberg has only PlaintextEncryptionManager currently, we implement a custom encryption manager

I am very interested in this area. How does your EncryptionManager work? Currently, the implementation of BaseTable does not provide a custom method. Did you wrap your own Table object?

There is not custom method provided currently, we just overwrite the TableOperations.encryption() method to provider our custom EncryptionManager. Also, we patch some of the PR in this project: https://github.com/apache/iceberg/projects/5

I got you, BTW, GenericDeleteFilter also need EncryptionManager

[liaoyt]

@liaoyt Thanks for your contribution，I want to know if you are currently using the encryption feature.

yes, although iceberg has only PlaintextEncryptionManager currently, we implement a custom encryption manager

I am very interested in this area. How does your EncryptionManager work? Currently, the implementation of BaseTable does not provide a custom method. Did you wrap your own Table object?

There is not custom method provided currently, we just overwrite the TableOperations.encryption() method to provider our custom EncryptionManager. Also, we patch some of the PR in this project: https://github.com/apache/iceberg/projects/5

I got you, BTW, GenericDeleteFilter also need EncryptionManager

thanks for reminding, i'll have a look and fix it later, maybe tomorrow.

[shidayang]

Can you add some unit tests to verify if it's working?

[liaoyt]

Can you add some unit tests to verify if it's working?

sure，i can try add some unit test

[shidayang]

LGTM

[shidayang]

Thanks for your contributions