Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Planning table failed, java.lang.StackOverflowError: null #3239

Closed
1 of 2 tasks
Tracked by #3123
lintingbin opened this issue Oct 10, 2024 · 2 comments · Fixed by #3240
Closed
1 of 2 tasks
Tracked by #3123

[Bug]: Planning table failed, java.lang.StackOverflowError: null #3239

lintingbin opened this issue Oct 10, 2024 · 2 comments · Fixed by #3240
Labels
type:bug Something isn't working

Comments

@lintingbin
Copy link
Contributor

What happened?

When amoro executes planning with many partitions (over 20,000 in my case) and needs compression, a "StackOverflowError" occurs. Using the official iceberg rewrite_data_files does not have this issue.

Affects Versions

0.7.0

What table formats are you seeing the problem on?

Iceberg

What engines are you seeing the problem on?

AMS

How to reproduce

No response

Relevant log output

2024-10-09 07:39:05,651 ERROR [plan-executor-thread-37916] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Planning table hive.test.mytable(tableId=672) failed
java.lang.StackOverflowError: null
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:356) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]
        at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:369) ~[iceberg-api-1.4.3.jar:?]

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

  • I agree to follow this project's Code of Conduct
@lintingbin lintingbin added the type:bug Something isn't working label Oct 10, 2024
@7hong
Copy link
Contributor

7hong commented Oct 11, 2024

I had the same problem, and I resolved it by limiting the number of partitions read in the filter。
I will submit a pr later

@7hong 7hong mentioned this issue Oct 11, 2024
Merged
3 tasks
@majin1102
Copy link
Contributor

I had the same problem, and I resolved it by limiting the number of partitions read in the filter。 I will submit a pr later

I left comments in your PR, PTAL

@klion26 klion26 mentioned this issue Oct 24, 2024
Closed
28 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[AMORO-3239] Fix stack overflow caused by reading too many partitions in the filter
3 participants
@lintingbin @majin1102 @7hong