Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MergeRollupTaskGenerator incorrectly calculates earliest start time of in_progress segments #15128

Open
sajjad-moradi opened this issue Feb 26, 2025 · 1 comment

Comments

@sajjad-moradi
Copy link
Contributor

For realtime table, MergeRollup task is intended to not process segments with start time later than the earliest start time of all in progress segments, but the code is not calculating it correctly. The if condition in the following part of the code is alway false because totalDocs and startTimeMs are both -1 for consuming segments. This leads to incorrectly calculate earliestStartTime to Long.MAX_VALUE:

long earliestStartTimeMsOfInProgressSegments = Long.MAX_VALUE;
for (SegmentZKMetadata segmentZKMetadata : allSegments) {
if (!segmentZKMetadata.getStatus().isCompleted()
&& segmentZKMetadata.getTotalDocs() > 0
&& segmentZKMetadata.getStartTimeMs() < earliestStartTimeMsOfInProgressSegments) {
earliestStartTimeMsOfInProgressSegments = segmentZKMetadata.getStartTimeMs();
}
}

@kate-yq
Copy link
Contributor

kate-yq commented Feb 26, 2025

I'll work on this. Thx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants