Sometimes Some OFFLINE Table Entries Do Not Appear When Querying the Hybrid Table

[jburgoos]

Description

We are experiencing an issue where queries on a hybrid table in Apache Pinot do not always return all the expected data from the OFFLINE segments. However, when querying the OFFLINE table directly, all expected records are present. This suggests that Pinot is not correctly merging the results from OFFLINE and REALTIME segments in some cases.

Observed Behaviour

This query counts the number of entries grouped into 1-hour buckets. As shown in the screenshots, when executing the query on the OFFLINE table directly, more data is displayed compared to querying the hybrid table.

Expected Behavior

Queries on the hybrid table should return all relevant records from both OFFLINE and REALTIME segments, ensuring no data loss from OFFLINE.

Environment

• Pinot Version: 1.2.0
• Deployment Type: Running on physical machines

[Jackie-Jiang]

You can read more about how Pinot manages data merge of hybrid table here: https://docs.pinot.apache.org/basics/concepts/components/table/time-boundary