cudf-polars resolves executor/engine config (explicit value, then env var, then default)
in three separate places today: StreamingOptions, StreamingExecutor/
ConfigOptions.from_polars_engine, and a handful of ad hoc reads inside
SPMDEngine/DaskEngine/RayEngine.__init__. The same env var name and default end up
written two or three times for the same field, with nothing keeping them in sync. Some
fields end up reading a hardcoded fallback instead of the actual env var because one of
the duplicated copies never checks it, and _reset() doesn't consistently preserve a
field's previous value across an unrelated reset.
We want one shared way to resolve a field's value regardless of which class declares it,
only deferring resolution for fields that truly need it, and resolving an engine's config
once at construction instead of on every query.
Sub-issues:
cudf-polars resolves executor/engine config (explicit value, then env var, then default)
in three separate places today:
StreamingOptions,StreamingExecutor/ConfigOptions.from_polars_engine, and a handful of ad hoc reads insideSPMDEngine/DaskEngine/RayEngine.__init__. The same env var name and default end upwritten two or three times for the same field, with nothing keeping them in sync. Some
fields end up reading a hardcoded fallback instead of the actual env var because one of
the duplicated copies never checks it, and
_reset()doesn't consistently preserve afield's previous value across an unrelated reset.
We want one shared way to resolve a field's value regardless of which class declares it,
only deferring resolution for fields that truly need it, and resolving an engine's config
once at construction instead of on every query.
Sub-issues:
StreamingOptions._from_argparsesilently drops env vars #23741StreamingOptions._from_argparsesilently drops env vars_reset()reverts unrelatedexecutor_optionsfields to their default #23742_reset()reverts unrelated fields to their defaultUNSPECIFIEDto fields that actually need to defer #23744 NarrowUNSPECIFIEDto fields that actually need to deferdynamic_planning/join_filter_pushdownenv var resolution lives outside_opt()Optionsdoesn't respect aRAPIDSMPF_*env var unless the caller forwards it