diff --git a/docs/ss-migration-guide.md b/docs/ss-migration-guide.md index 3247866206eef..2a34e91c87a88 100644 --- a/docs/ss-migration-guide.md +++ b/docs/ss-migration-guide.md @@ -29,6 +29,7 @@ Please refer [Migration Guide: SQL, Datasets and DataFrame](sql-migration-guide. ## Upgrading from Structured Streaming 3.5 to 4.0 - Since Spark 4.0, Spark falls back to single batch execution if any source in the query does not support `Trigger.AvailableNow`. This is to avoid any possible correctness, duplication, and dataloss issue due to incompatibility between source and wrapper implementation. (See [SPARK-45178](https://issues.apache.org/jira/browse/SPARK-45178) for more details.) +- Since Spark 4.0, new configuration `spark.sql.streaming.ratioExtraSpaceAllowedInCheckpoint` (default: `0.3`) controls the amount of additional space allowed in the checkpoint directory to store stale version files for batch deletion inside maintenance task. This is to amortize the cost of listing in cloud store. Setting this to `0` defaults to the old behavior. (See [SPARK-48931](https://issues.apache.org/jira/browse/SPARK-48931) for more details.) ## Upgrading from Structured Streaming 3.3 to 3.4