Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No pushdown filters applied if temporal type was optimized into timestamp type by datafusion #942

Open
omerhadari opened this issue Feb 3, 2025 · 0 comments

Comments

@omerhadari
Copy link
Contributor

When running queries such as SELECT * FROM table WHERE ts_col = CAST('2024-01-01 00:12:34' AS TIMESTAMP), Datafusion converts the Cast directly to a TimestampNanosecond type. This is not handled by the code in here.

Test for example that shows the issue:

    #[test]
    fn test_comparison_with_nanos_work() {
        let left = Expr::Column(Column::new_unqualified("ts"));
        let right = Expr::Literal(ScalarValue::TimestampNanosecond(Some(1000), None));
        let op = Operator::GtEq;
        let binary_expr = Expr::BinaryExpr(BinaryExpr::new(Box::new(left), op, Box::new(right)));
        let predicate = convert_filters_to_predicate(&vec![binary_expr]).unwrap();
        let expected_predicate =
            Reference::new("ts").greater_than_or_equal_to(Datum::timestamp_nanos(1000));
        assert_eq!(predicate, expected_predicate);
    }

Note:
The correct type of Datum should be handled in values.rs, otherwise the issue will manifest as a failure to cast timestamp to whatever it's compared to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant