Skip to content

Empty Parquet files invalidate file ordering and statistics-based grouping #24647

Description

@xudong963

Describe the bug

A Parquet file with exactly zero rows can prevent DataFusion from using the ordering of the non-empty files in the same listing table.

An empty file contributes no rows, so it cannot violate any ordering. However, an empty Parquet file normally has no sorting_columns ordering and no column min/max values. This currently has two user-visible effects:

  1. derive_common_ordering_from_files treats the empty file's None ordering as incompatible with the ordering inferred from non-empty files. The scan loses its output ordering and can retain an unnecessary SortExec.
  2. With datafusion.execution.split_file_groups_by_statistics = true, MinMaxStatistics::new_from_files attempts to read min/max values from the empty file and planning fails with statistics not found.

To reproduce

Create two or more non-overlapping Parquet files using COPY (... ORDER BY key), then add a schema-only file to the same directory:

SET datafusion.execution.collect_statistics = true;
SET datafusion.execution.split_file_groups_by_statistics = true;

CREATE TABLE source(key INT) AS VALUES (1), (2), (3), (4);

COPY (SELECT * FROM source WHERE key <= 2 ORDER BY key)
TO 'sorted/data1.parquet' STORED AS PARQUET;

COPY (SELECT * FROM source WHERE key > 2 ORDER BY key)
TO 'sorted/data2.parquet' STORED AS PARQUET;

COPY (SELECT * FROM source WHERE FALSE)
TO 'sorted/empty.parquet' STORED AS PARQUET;

CREATE EXTERNAL TABLE sorted(key INT)
STORED AS PARQUET LOCATION 'sorted/';

EXPLAIN SELECT * FROM sorted ORDER BY key;

The non-empty files have Parquet ordering metadata and min/max statistics. The empty file has num_rows = Precision::Exact(0), but no ordering or column min/max values.

Expected behavior

Files with num_rows == Precision::Exact(0) should be ignored when deriving and validating file ordering and when constructing min/max statistics for file grouping. The scan should retain the ordering proven by the non-empty files, and statistics-based grouping should not fail.

Files whose row count is absent or inexact should remain conservative; only exact zero-row statistics are sufficient to ignore a file. The row-count statistics are already collected from the Parquet footer during listing, so this requires no additional I/O.

Additional context

This affects both explicitly declared file ordering and ordering inferred from Parquet sorting_columns metadata. A regression test can cover a directory containing ordered non-empty Parquet files plus a zero-row file without ordering metadata, with statistics-based file-group splitting enabled.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions