Skip to content

Commit

Permalink
Update migration doc for SPARK-48931
Browse files Browse the repository at this point in the history
  • Loading branch information
riyaverm-db committed Jul 26, 2024
1 parent 7e678a0 commit 7d0b610
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions docs/ss-migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Please refer [Migration Guide: SQL, Datasets and DataFrame](sql-migration-guide.
## Upgrading from Structured Streaming 3.5 to 4.0

- Since Spark 4.0, Spark falls back to single batch execution if any source in the query does not support `Trigger.AvailableNow`. This is to avoid any possible correctness, duplication, and dataloss issue due to incompatibility between source and wrapper implementation. (See [SPARK-45178](https://issues.apache.org/jira/browse/SPARK-45178) for more details.)
- Since Spark 4.0, new configuration `spark.sql.streaming.ratioExtraSpaceAllowedInCheckpoint` (default: `0.3`) controls the amount of additional space allowed in the checkpoint directory to store stale version files for batch deletion inside maintenance task. This is to amortize the cost of listing in cloud store. Setting this to `0` defaults to the old behavior. (See [SPARK-48931](https://issues.apache.org/jira/browse/SPARK-48931) for more details.)

## Upgrading from Structured Streaming 3.3 to 3.4

Expand Down

0 comments on commit 7d0b610

Please sign in to comment.