Describe the bug
A Parquet file with exactly zero rows can prevent DataFusion from using the ordering of the non-empty files in the same listing table.
An empty file contributes no rows, so it cannot violate any ordering. However, an empty Parquet file normally has no sorting_columns ordering and no column min/max values. This currently has two user-visible effects:
derive_common_ordering_from_files treats the empty file's None ordering as incompatible with the ordering inferred from non-empty files. The scan loses its output ordering and can retain an unnecessary SortExec.
- With
datafusion.execution.split_file_groups_by_statistics = true, MinMaxStatistics::new_from_files attempts to read min/max values from the empty file and planning fails with statistics not found.
To reproduce
Create two or more non-overlapping Parquet files using COPY (... ORDER BY key), then add a schema-only file to the same directory:
SET datafusion.execution.collect_statistics = true;
SET datafusion.execution.split_file_groups_by_statistics = true;
CREATE TABLE source(key INT) AS VALUES (1), (2), (3), (4);
COPY (SELECT * FROM source WHERE key <= 2 ORDER BY key)
TO 'sorted/data1.parquet' STORED AS PARQUET;
COPY (SELECT * FROM source WHERE key > 2 ORDER BY key)
TO 'sorted/data2.parquet' STORED AS PARQUET;
COPY (SELECT * FROM source WHERE FALSE)
TO 'sorted/empty.parquet' STORED AS PARQUET;
CREATE EXTERNAL TABLE sorted(key INT)
STORED AS PARQUET LOCATION 'sorted/';
EXPLAIN SELECT * FROM sorted ORDER BY key;
The non-empty files have Parquet ordering metadata and min/max statistics. The empty file has num_rows = Precision::Exact(0), but no ordering or column min/max values.
Expected behavior
Files with num_rows == Precision::Exact(0) should be ignored when deriving and validating file ordering and when constructing min/max statistics for file grouping. The scan should retain the ordering proven by the non-empty files, and statistics-based grouping should not fail.
Files whose row count is absent or inexact should remain conservative; only exact zero-row statistics are sufficient to ignore a file. The row-count statistics are already collected from the Parquet footer during listing, so this requires no additional I/O.
Additional context
This affects both explicitly declared file ordering and ordering inferred from Parquet sorting_columns metadata. A regression test can cover a directory containing ordered non-empty Parquet files plus a zero-row file without ordering metadata, with statistics-based file-group splitting enabled.
Describe the bug
A Parquet file with exactly zero rows can prevent DataFusion from using the ordering of the non-empty files in the same listing table.
An empty file contributes no rows, so it cannot violate any ordering. However, an empty Parquet file normally has no
sorting_columnsordering and no column min/max values. This currently has two user-visible effects:derive_common_ordering_from_filestreats the empty file'sNoneordering as incompatible with the ordering inferred from non-empty files. The scan loses its output ordering and can retain an unnecessarySortExec.datafusion.execution.split_file_groups_by_statistics = true,MinMaxStatistics::new_from_filesattempts to read min/max values from the empty file and planning fails withstatistics not found.To reproduce
Create two or more non-overlapping Parquet files using
COPY (... ORDER BY key), then add a schema-only file to the same directory:The non-empty files have Parquet ordering metadata and min/max statistics. The empty file has
num_rows = Precision::Exact(0), but no ordering or column min/max values.Expected behavior
Files with
num_rows == Precision::Exact(0)should be ignored when deriving and validating file ordering and when constructing min/max statistics for file grouping. The scan should retain the ordering proven by the non-empty files, and statistics-based grouping should not fail.Files whose row count is absent or inexact should remain conservative; only exact zero-row statistics are sufficient to ignore a file. The row-count statistics are already collected from the Parquet footer during listing, so this requires no additional I/O.
Additional context
This affects both explicitly declared file ordering and ordering inferred from Parquet
sorting_columnsmetadata. A regression test can cover a directory containing ordered non-empty Parquet files plus a zero-row file without ordering metadata, with statistics-based file-group splitting enabled.