Skip to content

[Story]: Rework how cudf-polars resolves executor/engine configuration #23740

Description

@Matt711

cudf-polars resolves executor/engine config (explicit value, then env var, then default)
in three separate places today: StreamingOptions, StreamingExecutor/
ConfigOptions.from_polars_engine, and a handful of ad hoc reads inside
SPMDEngine/DaskEngine/RayEngine.__init__. The same env var name and default end up
written two or three times for the same field, with nothing keeping them in sync. Some
fields end up reading a hardcoded fallback instead of the actual env var because one of
the duplicated copies never checks it, and _reset() doesn't consistently preserve a
field's previous value across an unrelated reset.

We want one shared way to resolve a field's value regardless of which class declares it,
only deferring resolution for fields that truly need it, and resolving an engine's config
once at construction instead of on every query.

Sub-issues:

Metadata

Metadata

Assignees

No one assigned

    Labels

    cudf-polarsIssues specific to cudf-polarsimprovementImprovement / enhancement to an existing function

    Type

    No type

    Projects

    Status
    Todo
    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions