Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schema_force_view_type configuration not working for CREATE EXTERNAL TABLE #14909

Open
2010YOUY01 opened this issue Feb 27, 2025 · 1 comment · May be fixed by #14922
Open

schema_force_view_type configuration not working for CREATE EXTERNAL TABLE #14909

2010YOUY01 opened this issue Feb 27, 2025 · 1 comment · May be fixed by #14922
Assignees
Labels
bug Something isn't working

Comments

@2010YOUY01
Copy link
Contributor

Describe the bug

schema_force_view_type configuration said:

datafusion.execution.parquet.schema_force_view_types | true | (reading) If true, parquet reader will read columns of Utf8/Utf8Large with Utf8View, and Binary/BinaryLarge with BinaryView.

Reference: https://datafusion.apache.org/user-guide/configs.html

I tried the following script within datafusion-cli

set datafusion.execution.parquet.schema_force_view_types = true;

CREATE EXTERNAL TABLE IF NOT EXISTS lineitem (
        l_orderkey BIGINT,
        l_partkey BIGINT,
        l_suppkey BIGINT,
        l_linenumber INTEGER,
        l_quantity DECIMAL(15, 2),
        l_extendedprice DECIMAL(15, 2),
        l_discount DECIMAL(15, 2),
        l_tax DECIMAL(15, 2),
        l_returnflag VARCHAR,
        l_linestatus VARCHAR,
        l_shipdate DATE,
        l_commitdate DATE,
        l_receiptdate DATE,
        l_shipinstruct VARCHAR,
        l_shipmode VARCHAR,
        l_comment VARCHAR
) STORED AS parquet
LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf10/lineitem';

select arrow_typeof(l_comment) from lineitem limit 1;

The data type is Utf8 instead Utf8View

+----------------------------------+
| arrow_typeof(lineitem.l_comment) |
+----------------------------------+
| Utf8                             |
+----------------------------------+

To Reproduce

No response

Expected behavior

No response

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants