[Bug] Trino failed to query Amoro Mixed Format Table after restarting AMS

[Aitongong]

What happened?

重启amoro ams后，发现trino无法查询amoro表了

Affects Versions

0.5.0

What engines are you seeing the problem on?

No response

How to reproduce

重启amoro ams，执行trino查询amoro表

Relevant log output

No response

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[shidayang]

Thank you for reporting this bug. May I ask if you are using Mixed-format? What is the error log? BTW, could you please pay attention to the format of the issue report?

[Aitongong]

Thank you for reporting this bug. May I ask if you are using Mixed-format? What is the error log? BTW, could you please pay attention to the format of the issue report?

trino did not find the error log; Mixed-format

[shidayang]

I couldn't reproduce this issue, and restarting AMS had no impact on Trino in my environment. We may need some additional error information, which should be available in Trino's server.log.

[Aitongong]

I couldn't reproduce this issue, and restarting AMS had no impact on Trino in my environment. We may need some additional error information, which should be available in Trino's server.log.

重启执行查询之后 trino server.log 只有一条这样的日志
2023-09-18T08:42:43.535+0800 INFO dispatcher-query-6074 io.trino.event.QueryMonitor TIMELINE: Query 20230918_004030_20040_ade4d :: FAILED (USER_CANCELED) :: elapsed 132860ms :: planning 132860ms :: waiting 0ms :: scheduling 0ms :: running 0ms :: finishing 0ms :: begin 2023-09-18T08:40:30.672+08:00 :: end 2023-09-18T08:42:43.532+08:00

[Aitongong]

I couldn't reproduce this issue, and restarting AMS had no impact on Trino in my environment. We may need some additional error information, which should be available in Trino's server.log.

如果当我通过trino有查询一直在查amoro的话 这个时候停止amoro ams 后续就会连接不上amoro了

trino里面为什么不会打印amoro相关报错日志呢 我在log.properties 配置io.trino.plugin.arctic=WARN 好像不生效

[HuangFru]

I couldn't reproduce this issue, and restarting AMS had no impact on Trino in my environment. We may need some additional error information, which should be available in Trino's server.log.

如果当我通过trino有查询一直在查amoro的话 这个时候停止amoro ams 后续就会连接不上amoro了

trino里面为什么不会打印amoro相关报错日志呢 我在log.properties 配置io.trino.plugin.arctic=WARN 好像不生效

try this: io.trino=WARN.

[Aitongong]

I couldn't reproduce this issue, and restarting AMS had no impact on Trino in my environment. We may need some additional error information, which should be available in Trino's server.log.

如果当我通过trino有查询一直在查amoro的话 这个时候停止amoro ams 后续就会连接不上amoro了
trino里面为什么不会打印amoro相关报错日志呢 我在log.properties 配置io.trino.plugin.arctic=WARN 好像不生效

try this: io.trino=WARN.

这个我使用默认的 io.trino=INFO 这个级别配置也是不行 打印不出来amoro相关报错日志

[wangtaohz]

@Aitongong Thank you for posting your issue. 😄

To ensure that as many developers can participate in discussions and receive helpful responses, the Amoro community encourages the use of English as the main language for communication.

My suggestion is to first change the title and description to English.

[wangtaohz]

Has AMS been configured for high availability? I think it would be helpful if you could provide the Amoro-related configuration in Trino.

[HuangFru]

The reason for this issue is a leak in accessing the Thrift Client pool of AMS. If there are ongoing Trino queries During the period when AMS is inaccessible (due to restart or service stop), they will enter the retry logic of the client pool, where the inaccessible objects are not being destroyed in a timely manner. There will be a PR to fix this issue later.

[wangtaohz]

The reason for this issue is a leak in accessing the Thrift Client pool of AMS. If there are ongoing Trino queries During the period when AMS is inaccessible (due to restart or service stop), they will enter the retry logic of the client pool, where the inaccessible objects are not being destroyed in a timely manner. There will be a PR to fix this issue later.

Does this issue only affect Trino, or does it also affect other engines?

[HuangFru]

The reason for this issue is a leak in accessing the Thrift Client pool of AMS. If there are ongoing Trino queries During the period when AMS is inaccessible (due to restart or service stop), they will enter the retry logic of the client pool, where the inaccessible objects are not being destroyed in a timely manner. There will be a PR to fix this issue later.

Does this issue only affect Trino, or does it also affect other engines?

I think other engines have the same problem.

[Aitongong]

The reason for this issue is a leak in accessing the Thrift Client pool of AMS. If there are ongoing Trino queries During the period when AMS is inaccessible (due to restart or service stop), they will enter the retry logic of the client pool, where the inaccessible objects are not being destroyed in a timely manner. There will be a PR to fix this issue later.

How should I configure trino's amoro logs if I want to see them

[HuangFru]

The reason for this issue is a leak in accessing the Thrift Client pool of AMS. If there are ongoing Trino queries During the period when AMS is inaccessible (due to restart or service stop), they will enter the retry logic of the client pool, where the inaccessible objects are not being destroyed in a timely manner. There will be a PR to fix this issue later.

How should I configure trino's amoro logs if I want to see them

Could you show me the content of your node.properties and log.properties in ${TRINO_HOME}/etc?

[Aitongong]

The reason for this issue is a leak in accessing the Thrift Client pool of AMS. If there are ongoing Trino queries During the period when AMS is inaccessible (due to restart or service stop), they will enter the retry logic of the client pool, where the inaccessible objects are not being destroyed in a timely manner. There will be a PR to fix this issue later.

How should I configure trino's amoro logs if I want to see them

Could you show me the content of your node.properties and log.properties in ${TRINO_HOME}/etc?

Yes , io.trino=INFO

[HuangFru]

node.properties

What about node.properties? Just need the coordinator.

[Aitongong]

coordinator node , node.properties : io.trino=INFO

[HuangFru]

coordinator node , node.properties : io.trino=INFO

I think the io.trino=INFO should be set in log.properties instead of node.properties.
You can refer to this : https://trinodb.github.io/docs.trino.io/406/installation/deployment.html#node-properties