Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-48910][SQL] Use HashSet/HashMap to avoid linear searches in Pr…
…eprocessTableCreation ### What changes were proposed in this pull request? Use `HashSet`/`HashMap` instead of doing linear searches over the `Seq`. In case of 1000s of partitions this significantly improves the performance. ### Why are the changes needed? To avoid the O(n*m) passes in the `PreprocessTableCreation` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing UTs ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#47484 from vladimirg-db/vladimirg-db/get-rid-of-linear-searches-preprocess-table-creation. Authored-by: Vladimir Golubev <vladimir.golubev@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
- Loading branch information