High Memory Usage in Mutable Dictionary Index When Cardinality Increases Significantly

[jtao15]

We've observed excessive heap memory allocation when new columns with a dictionary forward index are added first, and Kafka values are populated later. This issue arises due to static buffer size allocation based on column cardinality from StatsHistory (e.g., 1, due to the default value).

When Kafka ingestion begins and the actual cardinality increases significantly, small buffers are repeatedly allocated, leading to a long list of bufferReader. Since the implementation uses a CopyOnWriteArrayList, adding new bufferReader entries is expensive, leading to increased memory usage, excessive garbage collection, and potentially causing out-of-memory (OOM) issues.

Ideally, buffer size allocation should be dynamically adjusted at consumption time rather than relying only on StatsHistory.

[jtao15]

cc @vvivekiyer @sajjad-moradi