Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Arctic-1476]: Support load and optimizing iceberg table on s3 #1481

Merged
merged 251 commits into from
Jun 12, 2023
Merged

[Arctic-1476]: Support load and optimizing iceberg table on s3 #1481

merged 251 commits into from
Jun 12, 2023

Conversation

baiyangtx
Copy link
Contributor

Why are the changes needed?

fix #1476 to support load and optimzing iceberg table with s3 or other non-hadoop storage system.

Brief change log

  • Add ArcticFileIOIcebergAdapter
  • Using ArcticFileIOIcebergAdapter instead of HadoopFileIO when it's not a hadoop file system.

How was this patch tested?

  • Run test locally before making a pull request

baiyangtx and others added 30 commits April 18, 2023 21:22
@baiyangtx
fileIo refactor in hive module
d76be32
replace ams module with refactored codes
3289065
remove optimizer module
a458aa1
fix compile errors
1beb715
@shidayang
Refactor core and hive
b8f80af
@baiyangtx
fix terminal catalog connector check
67004f6
@baiyangtx
check terminal session NPE before release.
2badd35
modify ams-server and dashboard module and package names
0778478
Merge remote-tracking branch 'origin/refactor-ams' into refactor-ams
a96d386
@HuangFru
[Arctic-1372] Resolve compile error (apache#1417)
84f67a3 
* resolve compile error.
@baiyangtx
[ARCTIC-1062][AMS]Terminal support config spark properties in the loc…
35c1d23 
…al model (apache#1094)
@baiyangtx
[ARCTIC-1090][AMS]Terminal support add hadoop conf into sparkConf (ap…
0e0dd91 
…ache#1099)
add some PRs since v0.4
d8c28c3
Merge remote-tracking branch 'origin/refactor-ams' into refactor-ams
74b296a
@hameizi
[AMS-Refactor] fix runtime bug (apache#1421)
2ca6a24 
fix runtime bug
return null when got thrift MISSING_RESULT error
a95cb98
@huiyuanz
[ARCTIC][AMS]:display login error msg (apache#1424)
61346dd 
feat: display login error msg

Co-authored-by: zhubeibei <[email protected]>
@shidayang
Commit the DeleteFile delete operation separately in Iceberg Format (a…
b740a2f 
…pache#1425)
add 0.5.0-init.sql
60ae6fa
rename optimize_group to optimizer_groups in config.xml
fc410a4
remove useless configs
3f13bf1
Reuse PropertyNames property definetion in OptimizerConfig
70bc96b
fix conflicts
86cb016
remove old optimizer module files
b6d8c39
fix some compile error in test codes
58f22f2
@hameizi
[AMS-Refactor] fix runtime bug (apache#1430)
c18ff3f 
* fix runtime bug

* fix runtime bug
add unit tests for optimizer
3e6c868
rename util class name from utils to util
dcf323e
remove resourceId from optimizerRegisterInfo
56794fe
support derby database
0044970
@zhoujinsong zhoujinsong closed this Jun 1, 2023
@baiyangtx baiyangtx reopened this Jun 1, 2023
@baiyangtx baiyangtx changed the base branch from refactor-ams to master June 1, 2023 07:49
@baiyangtx baiyangtx linked an issue Jun 7, 2023 that may be closed by this pull request
[Feature]: Support Iceberg on S3 or other non-hadoop storage system. #1476
Closed
2 tasks
@github-actions github-actions bot added the module:mixed-flink Flink moduel for Mixed Format label Jun 7, 2023
@codecov
Copy link

codecov bot commented Jun 7, 2023 •
edited
Loading

Codecov Report

Patch coverage: 8.64% and project coverage change: -0.15 ⚠️

Comparison is base (fc579b4) 47.39% compared to head (0f16325) 47.24%.

Additional details and impacted files 
@@             Coverage Diff              @@
##             master    #1481      +/-   ##
============================================
- Coverage     47.39%   47.24%   -0.15%     
  Complexity     3442     3442              
============================================
  Files           461      463       +2     
  Lines         24401    24476      +75     
  Branches       2500     2500              
============================================
- Hits          11564    11563       -1     
- Misses        11768    11842      +74     
- Partials       1069     1071       +2
Flag Coverage Δ
core 47.06% <8.64%> (-0.19%) ⬇️
trino 48.04% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ava/com/netease/arctic/io/ArcticFileIOAdapter.java 0.00% <0.00%> (ø)
.../src/main/java/com/netease/arctic/io/PathInfo.java 75.00% <ø> (ø)
...etease/arctic/io/SupportFileRecycleOperations.java 0.00% <ø> (ø)
...etease/arctic/io/SupportsFileSystemOperations.java 50.00% <ø> (ø)
...va/com/netease/arctic/table/BasicTableBuilder.java 0.00% <0.00%> (ø)
.../com/netease/arctic/hive/table/KeyedHiveTable.java 81.81% <0.00%> (-5.28%) ⬇️
.../netease/arctic/catalog/IcebergCatalogWrapper.java 45.79% <22.72%> (-7.02%) ⬇️
...netease/arctic/hive/catalog/ArcticHiveCatalog.java 47.52% <100.00%> (ø)

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@baiyangtx
Merge branch 'master' into support-iceberg-on-s3
7b77866 
# Conflicts:
#	ams/server/src/main/java/com/netease/arctic/server/table/executor/OrphanFilesCleaningExecutor.java
#	ams/server/src/test/java/com/netease/arctic/server/optimizing/TestMixedHiveOptimizing.java
#	core/src/test/java/com/netease/arctic/catalog/TableTestBase.java
@github-actions github-actions bot removed type:build module:ams-dashboard Ams dashboard module module:mixed-spark Spark module for Mixed Format module:mixed-flink Flink moduel for Mixed Format labels Jun 12, 2023
zhoujinsong
zhoujinsong approved these changes Jun 12, 2023
View reviewed changes
Copy link
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@baiyangtx baiyangtx merged commit 24a2804 into apache:master Jun 12, 2023
@baiyangtx baiyangtx deleted the support-iceberg-on-s3 branch July 13, 2023 07:21
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
@baiyangtx @ShawHee
[Arctic-1476]: Support load and optimizing iceberg table on s3 (apach…
be86a47 
…e#1481)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhoujinsong zhoujinsong zhoujinsong approved these changes

Assignees
No one assigned
Labels
module:core Core module module:mixed-hive Hive moduel for Mixed Format
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature]: Support Iceberg on S3 or other non-hadoop storage system.
8 participants
@baiyangtx @zhoujinsong @shidayang @HuangFru @hameizi @huiyuanz @wangtaohz @hellojinsilei