Skip to content

[WIP][ML] Add vector posexplode function - #58194

Draft
zhengruifeng wants to merge 1 commit into
apache:masterfrom
zhengruifeng:vector-posexplode-dev5
Draft

[WIP][ML] Add vector posexplode function#58194
zhengruifeng wants to merge 1 commit into
apache:masterfrom
zhengruifeng:vector-posexplode-dev5

Conversation

@zhengruifeng

@zhengruifeng zhengruifeng commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR adds an internal ML-specific vector_posexplode generator expression for MLlib vector
columns.

The new expression explodes org.apache.spark.ml.linalg.VectorUDT and
org.apache.spark.mllib.linalg.VectorUDT columns into (index, value) rows using the shared
four-field VectorUDT SQL representation:

  • sparse vectors emit the stored active indices/values pairs
  • dense vectors emit all (position, value) pairs

Why are the changes needed?

Spark SQL generator functions such as posexplode only accept arrays and maps. ML vector columns
are UDTs, so users currently need to densify vectors with vector_to_array or manually unwrap the
UDT struct and zip the internal sparse arrays.

This provides Spark ML internals with a direct vector-aware generator that preserves sparse active
elements without densifying first.

Does this PR introduce any user-facing change?

No. The Scala helper is private[ml], and the expression is intended only for Spark ML internals.

How was this patch tested?

Added coverage in FunctionsSuite for both spark.ml and spark.mllib dense and sparse vectors.

build/sbt mllib/Test/compile was started and the new mllib main sources compiled, but the task was
canceled during mllib test compilation so this draft PR could be pushed first.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)

@zhengruifeng
zhengruifeng force-pushed the vector-posexplode-dev5 branch from 4cd4ef9 to 32bf34b Compare August 21, 2026 13:24
@zhengruifeng
zhengruifeng force-pushed the vector-posexplode-dev5 branch from c0c7999 to c49167c Compare August 21, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant