Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: When using Flink api to write data into mix-iceberg table, it will update the table metadata to incorrect value #2525

Closed
2 tasks done
Tracked by #2176
GavinH1984 opened this issue Feb 4, 2024 · 0 comments · Fixed by #2529
Labels
type:bug Something isn't working

Comments

@GavinH1984
Copy link
Contributor

What happened?

using the flink Streaming API or Batch Api to write data into mix-iceberge format table( with HiveMetastore), it will update hive metadata storage_handler and ROW FORMAT SERDE to incorrect value:

com.netease.arctic.shaded.org.apache.iceberg.mr.hive.HiveIcebergStorageHandler/com.netease.arctic.shaded.org.apache.iceberg.mr.hive.HiveIcebergSerDe

this will cause when using client tool like DBeaver show error like class not found.

Affects Versions

0.6.0

What engines are you seeing the problem on?

Flink

How to reproduce

using the flink api to write data into mix-iceberge table, you will find the hive metadata have incorrect value:

select d.NAME , t.TBL_NAME, s.SLIB, t.CREATE_TIME, t.TBL_ID  
from tbls t 
join dbs d on t.DB_ID  = d.DB_ID 
join sds sd on sd.SD_ID = t.SD_ID 
join serdes s on s.SERDE_ID = sd.SERDE_ID 
where TBL_ID  in (
select TBL_ID  from table_params where PARAM_KEY = 'storage_handler' 
and PARAM_VALUE  in('com.netease.arctic.shaded.org.apache.iceberg.mr.hive.HiveIcebergStorageHandler')
)

Relevant log output

No response

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

  • I agree to follow this project's Code of Conduct
@GavinH1984 GavinH1984 added the type:bug Something isn't working label Feb 4, 2024
GavinH1984 added a commit to GavinH1984/arctic that referenced this issue Feb 4, 2024
…berg table, it will update the table metadata to incorrect value
zhoujinsong pushed a commit that referenced this issue Feb 5, 2024
…mix-iceberg table, it will update the table metadata to incorrect value (#2529)

Fix bug #2525 - When using Flink api to write data into mix-iceberg table, it will update the table metadata to incorrect value
zhoujinsong pushed a commit that referenced this issue Feb 6, 2024
…mix-iceberg table, it will update the table metadata to incorrect value (#2529)

Fix bug #2525 - When using Flink api to write data into mix-iceberg table, it will update the table metadata to incorrect value

(cherry picked from commit 9703f15)
Signed-off-by: zhoujinsong <[email protected]>
@zhoujinsong zhoujinsong mentioned this issue Jun 25, 2024
66 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:bug Something isn't working
Projects
None yet
1 participant