Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Statistics: Migrate to Distribution from Precision #14896

Open
ozankabak opened this issue Feb 26, 2025 · 1 comment
Open

Statistics: Migrate to Distribution from Precision #14896

ozankabak opened this issue Feb 26, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@ozankabak
Copy link
Contributor

Is your feature request related to a problem or challenge?

DataFusion doesn't have a great statistics infrastructure, which will be long project to fix. Luckily, we have begun the process to implement an extensible statistics infrastructure, and we have already merged a new low-level framework to evaluate expressions in a probabilistic context (i.e. the evaluate_statistics and propagate_statistics API introduced in #14699). The next step in this project is to switch from the old Precision object to the new Distribution object across the DataFusion codebase. This way, we will be able to use the existing mechanism around the ColumnStatistics and Statistics (maybe we should rename this to TableStatistics?) objects to handle statistics.

Describe the solution you'd like

Deprecating/removing Precision, updating its usages with Distribution.

Describe alternatives you've considered

N/A.

Additional context

No response

@ozankabak ozankabak added the enhancement New feature or request label Feb 26, 2025
@ozankabak ozankabak changed the title Statistics: Migrate to Distribution from Precision Statistics: Migrate to Distribution from Precision Feb 26, 2025
@berkaysynnada
Copy link
Contributor

Not just Precision, in some parts of the code, Interval objects could(should) be also converted to Distribution objects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants