Improved retention management for hybrid tables

[rajagopr]

Problem

Previously, RetentionManager would delete segments past retention period from a REALTIME table when the REALTIME table is part of a hybrid table setup. At times this resulted in prematurely deleting segments before the RTO task got a chance to move the segments to the OFFLINE table which resulted in data loss.

Solution

The changes in this PR makes the RetentionManager hybrid table aware. RetentionManager behavior is enhanced such that segments from a hybrid REALTIME table are eligible for deletion when the segment is older than the hybrid table time boundary.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 54.23729% with 27 lines in your changes missing coverage. Please review.

Project coverage is 63.62%. Comparing base (59551e4) to head (3ea4a8a).
Report is 1837 commits behind head on master.

Files with missing lines	Patch %	Lines
...troller/helix/core/retention/RetentionManager.java	52.63%	13 Missing and 14 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #15257      +/-   ##
============================================
+ Coverage     61.75%   63.62%   +1.87%     
- Complexity      207     1459    +1252     
============================================
  Files          2436     2772     +336     
  Lines        133233   156305   +23072     
  Branches      20636    23999    +3363     
============================================
+ Hits          82274    99447   +17173     
- Misses        44911    49366    +4455     
- Partials       6048     7492    +1444     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (+99.99%)	⬆️
integration	100.00% <ø> (+99.99%)	⬆️
integration1	100.00% <ø> (+99.99%)	⬆️
integration2	0.00% <ø> (ø)
java-11	63.59% <54.23%> (+1.88%)	⬆️
java-21	63.52% <54.23%> (+1.89%)	⬆️
skip-bytebuffers-false	63.60% <54.23%> (+1.86%)	⬆️
skip-bytebuffers-true	63.49% <54.23%> (+35.76%)	⬆️
temurin	63.62% <54.23%> (+1.87%)	⬆️
unittests	63.61% <54.23%> (+1.87%)	⬆️
unittests1	56.17% <100.00%> (+9.28%)	⬆️
unittests2	34.19% <54.23%> (+6.46%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.