Release v0.9.0

Apache Uniffle (Incubating) Release v0.9.0

Highlight

• Introduce dashboard.
• Introduce rust-based shuffle server.
• Add support for Spark 3.5.
• The data transportation Netty mode is production available.
• Reduce block id layout limitations and simplify layout configuration for Spark.

ChangeLog

• [#1751][0.9] improvement: support gluten (#1753)
• [#1764] fix(client): Fix timeout time unit for unregister requests (#1766)
• [#1149] fix: GC logs in JDK 11 do not include date and time stamps. (#1240)
• [#1675][FOLLOWUP] fix(test): Fix various flaky tests (#1730)
• [MINOR] fix: Update outdated config: rss.writer.send.check.timeout -> rss.client.send.check.timeout.ms (#1734)
• [#1721] fix(coordinator): classCastExpection of boolean->String with yaml style remote client conf (#1722)
• [#1673] fix(K8S): Fix the deployment of stable version K8S cluster (#1694)
• [#1675][FOLLOWUP] fix(test): Fix flaky tests which may cause port conflicts (#1696)
• [MINOR] fix(typo): Correct the removeShuffle method name (#1697)
• [MINOR] docs: modify the default value of rss.coordinator.select.partition.strategy in docs (#1692)
• [#1680] improvement(server): Remove partial HDFS files that written by server self for expired apps (#1681)
• [#1675] fix(test): Fix tests which may be flaky on different machines (#1676)
• [#1684] fix(server): use the diskSize obtained from periodic check to determine whether is writable (#1685)
• [#1678] fix(server): disk size leak on removing resources by AppPurgeEvent (#1679) (#1689)
• [#1657] build: Add license information after version 0.9.0 (#1671)
• [MINOR] chore(rust): disable flaky test of local_store_test (#1674)
• [#1459][FOLLOWUP] fix(server): Fix the issue of log variable printing (#1672)
• [#1459][FOLLOWUP] improvement(server): Print an error log when an event is dropped (#1643)
• [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. (#1666)
• [#378][FOLLOWUP] fix(server): Fix huge_partition_num metric (#1669)
• [#1662] fix(test): Fix Netty related flaky tests (#1663)
• [#1629] fix(operator): Support parsing NaN float value in metrics (#1630)
• [#1634] fix(server): remove app folder if app is expired (#1635)
• [MINOR] chore(rust): disable flaky test of test_ticket_manager (#1637)
• [#1596][FOLLOWUP] fix(netty): Send failed responses only when the channel is writable (#1641)
• [#1626] fix(server): Remove the meaningless eventOfUnderStorageManagers cache (#1627)
• [#1631] fix(server): ShuffleTaskInfo may leak when app is removed. (#1632)
• [#1373][FOLLOWUP] fix(spark): register with incorrect partitionRanges after reassign (#1612)
• [#1608][part-2] fix(spark): avoid releasing block in advance when enable block resend (#1610)
• [#1606] feat(client): Add client retry mechanism for NO_BUFFER when reading data(memory/local/index) (#1616)
• [#1608][part-1] fix(spark): Only share the replacement servers for faulty servers in one stage (#1609)
• [#1373][FOLLOWUP] fix(spark): shuffle manager rpc service invalid when partition data reassign is enabled (#1583)
• [#1596] fix(netty): Use a ChannelFutureListener callback mechanism to release readMemory (#1605)
• [#1598] fix(server) Fix inaccurate used_direct_memory_size metric (#1599)
• [#1472][FOLLOWUP] improvement(server): Release memory more accurately when failing to cache shuffle data (#1597)
• [MINOR] refactor: Calling lock() method outside try block to avoid unnecessary errors (#1590)
• [#1591] feat(spark): Support Spark 3.5.1 (#1592)
• [#1586] improvement(netty): Allow Netty Worker thread pool size to dynamically adapt to the number of processor cores (#1587)
• [#1588] improvement(server): Add exception handling for the thread pool when flushing events (#1589)
• [#1576] feat(doc): server deploy guide without hadoop-home env (#1577)
• [#1571] fix(server): Memory may leak when EventInvalidException occurs (#1574)
• [#1373][FOLLOWUP] fix(spark): incorrect partition id type (#1582)
• [#1373][FOLLOWUP] fix(spark3):Add client type when request shuffle assignment (#1580)
• build(deps): bump google.golang.org/protobuf from 1.28.0 to 1.33.0 (#1575)
• [#1554] feat(spark): Fetch dynamic client conf as early as possible (#1557)
• [#1572] fix(spark): Exceptions might be discarded when spilling buffers (#1573)
• [#1564] fix(server): disk health check invalid when hang (#1568)
• [#731][FOLLOWUP] feat(Spark): Configure blockIdLayout for Spark based on max partitions (#1566)
• [#1567] fix(spark): Let Spark use its own NettyUtils (#1565)
• [#1569] fix(rust): flaky test for test_ticket_manager (#1570)
• [MINOR] improvement(test): A better computation logic for WriteAndReadMetricsTest without using reflection (#1563)
• [#731] feat(spark): Make blockid layout configurable for Spark clients (#1528)
• [#808] improvement(spark): Verify the number of written records to ensure data correctness (#1558)
• [MINOR] improvement(client): Override getClientInfo method in ShuffleServerGrpcNettyClient and remove unused getDesc method (#1559)
• [#1552] improvement: Migrate from log4j1 to log4j2 (#1553)
• [#1472][part-6] FOLLOWUP: Fix Netty transport time when sending shuffle data requests (#1551)
• [#134][FOLLOWUP] improvement(spark2): Use taskId and attemptNo as taskAttemptId (#1544)
• [#1549] fix(common): Uniformly throw RssException for external callers (#1550)
• [MINOR] test: Use sensible partition ids in ShuffleReadClientImplTest (#1545)
• [#1546] fix(spark): NPE could happen before uncompressing after #1360 (#1547)
• feat(docker): Add example docker compose Uniffle/Spark cluster (#1532)
• [#1472][part-6] fix(netty): Make UTs truly test Netty mode (#1540)
• [MINOR] improvement(tez): Only invoking LOG.debug when LOG.isDebugEnabled is true (#1541)
• [#1459] fix(server): Memory leak for exceptional scenarios when flushing events (#1537)
• [#1472] fix(client): IlegalReferenceCountException for clientReadHandler.readShuffleData (#1536)
• [#1472][part-5] Use UnpooledByteBufAllocator to fix inaccurate usedMemory issue causing OOM (#1534)
• [MINOR] refactor(common): Move blockId bit logic into common class (#1527)
• [#1373][part-1] feat(spark): partition write to multi servers leveraging from reassignment mechanism (#1445)
• [MINOR] Update dashboard pom.xml to take arguments for node and npm download locations (#1530)
• [#1316] improvement(spark): detect OutputTracker API version via Spark version (#1317)
• [#134] improvement(spark3): Use taskId and attemptNo as taskAttemptId (#1529)
• [MINOR] feat(build): Allow to build distribution without some modules (#1525)
• [#1407] fix(rust): use grpc runtime worker threads and adjust default runtime config (#1517)
• [#1407] feat(rust): fix + add total grpc request metrics (#1516)
• [#1407] chore(rust): add cpu profile doc (#1515)
• [#1472][part-2] fix(server): Reuse ByteBuf when decoding shuffle blocks instead of reallocating it (#1521)
• [MINOR] fix(CI): Improve dashboard across the CI (#1526)
• [#1472][part-3] fix(client): Fix occasional IllegalReferenceCountException issues in extremely rare scenarios (#1522)
• [MINOR] fix(pom): Add missing shuffle-server dependencies to work with -Ptez
• [#1472][part-4] feature(server): Add metrics for Netty's pinnedDirectMemory and usedDirectMemory (#1524)
• [#1472][part-1] fix(server): Upgrade Netty and GRPC (#1520)
• [MINOR] fix(deploy): Fix invocation of kubernetes bash scripts (#1513)
• [#1476] feat(rust): Provide dedicated unregister app rpc interface (#1511)
• [#1476] feat(spark): Provide dedicated unregister app rpc interface (#1510)
• [MINOR] improvement(CI): Rework build and rust workflow events (#1508)
• [#1407] fix(rust): drop events and release memory when errors happened (#1509)
• [#1267][FOLLOWUP] improvement(client): INFO log level should be used in RetryUtils (#1500)
• [MINOR] feat(CI): Report test results in github comments (#1506)
• [#1407] fix(rust): return error when getting data from hdfs by client (#1507)
• [#1501] fix(server): storage selection cache accidentally deleted when clearing stage level data. (#1505)
• [#1407] fix(rust): dont panic when no available local disks (#1504)
• fix(rust): avoid checking storage type in runtime (#1503)
• [MINOR] build: Move dashboard module into profile and disable it by default (#1498)
• [#1497] improvement(spark): flushing buffer if the memoryUsed of the first record of WriterBuffer larger than bufferSize (#1485)
• [MINOR] improvement(test): Identify duplicate blocks in TestUtils.validateResult (#1495)
• [MINOR] fix: Get and increment ATOMIC_LONG in that order everywhere (#1496)
• [MINOR] docs: Improve comment on blockId structure (#1492)
• [MINOR] fix(server): Assert actual number of bitmaps matches bitNum (#1493)
• [#1490] improvement(spark3): Disable dynamic allocation shuffle tracking by default (#1491)
• [#1407] feat(rust): support more metrics about disk and topN data size (#1488)
• [#1407] feat(rust): support multiple spill policies and simplify hdfs config (#1487)
• [#1356] feat(server): improve expired buffers metric and log (#1469)
• [#1464][FOLLOWUP] improvement(spark): print abnormal shuffle servers that blocks fail to send (#1473)
• [#1467] feat(server): introduce total hdfs write data size for huge partition (#1468)
• [#1355] fix(client): Netty client will leak when decoding responses (#1455)
• [#1462] fix(server): Memory may leak when flushQueue is full (#1463)
• [#1466] feat(server): introduce the JvmPauseMonitor to detect the gc pause (#1470)
• [#1459] improvement(server): refactor DefaultFlushEventHandler and support event retry into pending queue (#1461)
• [#1464] improvement(spark): print abnormal shuffle servers that blocks fail to send (#1465)
• [#1456] improvement(client): Better exception handling when calling requireBuffer using GRPC (#1457)
• [#1428] fix(server): fallback invalid when local storage can't write (#1429)
• [#1453] improvement: Force to use the UNIX line...

Release v0.8.0

Apache Uniffle (Incubating) Release v0.8.0

Highlight

• Support TEZ
• Introduce Netty for shuffle data transmission
• Use off heap memory to store shuffle data.
• Introduce REST API for cluster management.
• Introduce command line for cluster management.

ChangeLog

• Change license owner to ASF by @kaijchen in #5
• Trivial code improvements by @wForget in #7
• [Minor] Store shuffleId int to be consistent with other data structure by @zuston in #10
• Introduce the asList method in ConfigOptions by @zuston in #9
• Rename package by @jerqi in #6
• Minimize apache-rat excluded files by @kaijchen in #11
• Update module names by @kaijchen in #12
• Covert PartitionAssignmentInfo to static inner class by @pan3793 in #15
• [Followup] Migrate to Junit5 by @zuston in #14
• [Bug] Fix NPE problem when process the event if application was cleared already by @colinmjj in #16
• [CI] Enable codecov report by @kaijchen in #17
• Correct the config description and fix typo by @zuston in #19
• Add CI and Codecov badges in README by @kaijchen in #20
• [Followup] Use asList method in some existing configOptions by @zuston in #18
• Move rss-integration-spark-common-test module package by @wForget in #23
• [INFRA] Improve asf.yaml to reduce the notifications by @jerryshao in #25
• [TEST] Improve code coverage in rss-common by @kaijchen in #26
• Remove redundant package by @wForget in #27
• [CI] Switch to temurin JDK by @kaijchen in #24
• [INFRA] Improve asf.yaml to reduce the notifications (another-try) by @jerryshao in #33
• Bump commons-lang3 from 3.5 to 3.10 by @wForget in #28
• Fix the log of incorrectly bound class by @wForget in #35
• [TYPO] Fix misspelled word "integration" by @kaijchen in #34
• Fix some hyperlink in README.md by @daugraph in #32
• Upgrade gRPC to support Apple Silicon by @pan3793 in #13
• Allow to specify custom tags to decide the assignment of servers by @zuston in #30
• Optimize the bash script by @zuston in #29
• [Improvement] reduce compiler warnings by @advancedxy in #46
• [Chore]: document update and build time optimize by @advancedxy in #45
• Supplement doc about assignment tags by @zuston in #47
• [Bug] Fix skip() api maybe skip unexpected bytes which makes inconsistent data by @colinmjj in #40
• [improvement] Remove experimental feature with ShuffleUploader by @colinmjj in #51
• [Improvement] Provides utility classes for creating thread factories by @smallzhongfeng in #49
• Enable spotbugs and fix high priority bugs by @kaijchen in #38
• [CI] Change default checkstyle severity to error by @kaijchen in #57
• [Style] Check indentation by @kaijchen in #56
• [Experimental Feature] MR Supports Remote Spill by @frankliee in #55
• [Improvement] Log indicate the shuffle server host:port when doing re… by @zuston in #58
• Send commit concurrently in client side by @zuston in #59
• Explicitly set the constructor with AccessManager when extending AccessChecker by @zuston in #43
• [DOC] Replace Firestorm with Uniffle by @jerqi in #60
• Introduce the extraProperties to support user-defined pluggable accessCheckers by @zuston in #42
• Log enhancement: Merge multiple logs into oneline and add more description by @zuston in #62
• [TEST] Add more unit tests in rss-common by @kaijchen in #63
• [MINOR] Comments of PartitionBalanceAssignmentStrategy miss byte units by @smallzhongfeng in #68
• [Minor] Make config keys and default values finalized by @kaijchen in #70
• [Log Improvment] Add more detailed debug info for MR client by @frankliee in #84
• [Improvement] Shutdown the grpc executors pool when closing by @zuston in #83
• Log enhancement: return error message when getting assignment servers and log exception when initializing by @zuston in #64
• [ISSUE-48] [Feature] Init Kubernetes operator directory by @jerqi in #75
• [Improvement] No need to use synchronized lock of the method scope when getting client by @zuston in #82
• [DOC] Remove Wechat group in README by @jerqi in #88
• [Performance Optimization] Improve the speed of writing index file in shuffle server by @zuston in #91
• [DOC] Update title and description in README by @kaijchen in #94
• [Improvement] ShuffleBlock should be release when finished reading by @xianjingfeng in #74
• [IMPROVEMENT][COMMON] Fix common module code style by @jerqi in #99
• [Improvement]LocalStorage init use multi thread #71 by @xianjingfeng in #72
• [Improvement] Use OR operation instead of serialization for cloning BitMaps by @kaijchen in #103
• [Improvement] Ignore partial failure on initializing local storage in shuffle server side by @zuston in #102
• [CI] Test compile in Java 11 and Java 17 by @kaijchen in #105
• Sleep less time but try more times when stopping by @xianjingfeng in #112
• [Improvement] Use ConfigBuilder to rewrite the class RssSparkConfig by @smallzhongfeng in #104
• [Improvement] Introduce config to customize assignment server numbers in client side by @zuston in #100
• Assign partition again if registerShuffleServers failed by @xianjingfeng in #115
• [ISSUE-106][IMPROVEMENT] Set rpc timeout for all rpc interface by @xianjingfeng in #113
• [MINOR][IMPROVEMENT] Avoid CoordinatorServer#initialization multiple new Configuration() by @zwangsheng in #118
• [Improve] Remove useless server id from StorageManagerFactory#createStorageManager by @zwangsheng in #119
• [MINOR][IMPROVEMENT][COORD] Fix coordinator module code style by @jerqi in #122
• [Improvement] Set heartBeatExecutorService as daemon thread by @smallzhongfeng in #121
• [JUnit] Introduce the property of trimStackTrace to show error stacktrace in mvn-test by @zuston in #126
• Make the conf of rss.storage.basePath as list by @zuston in #130
• [MINOR][IMPROVEMENT][STORAGE] Fix storage module code style by @jerqi in #131
• [Improvement] Add timeout reconnection when DelegationRssShuffleManager send the request of AccessCluster by @smallzhongfeng in #139
• [MINOR] Fix flaky test testGetHostIp by @izchen in #141
• [Improvement] Add the number of unhealthy nodes in CoordinatorMetrics by @smallzhongfeng in #147
• [ISSUE-48][FEATURE] Add Uniffle Dockerfile by @wangao1236 in #132
• [BUGFIX] Fix memory leak which cause oom by @summaryzb in #145
• [Log Improvement] Output the register...

Release v0.7.1

Apache Uniffle (Incubating) Release v0.7.1

Highlight

• Improvements

  • Refresh application when reading memory data to prevent application from being expired.

• Major bug fixes

  • Cache proxy user ugi to avoid memory leak.
  • Make metric reporter usable by fixing some logic errors.
  • Close ShuffleWriteClient after task was completed to exit container process.
  • Make sure 'finishShuffle' invoked after all shuffle data sent.
  • Update local storage metadata for all related events instead of just the first event.
  • Correct a wrong metric, grpc_server_connection_number.

• Minor bug fixes

  • Avoid returning null in defaultUserApps when quote file doesn't config user.
  • Fix LocalStorageManager divide by zero exception.

ChangeLog

• [ISSUE-669] improvement: refresh application when reading memory data (#741)
• Bump project version to 0.7.1-SNAPSHOT
• [ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• Revert "[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• [ISSUE-796][0.7] bug: Fix the issues of MetricReporter (#821)
• [MINOR][0.7] Avoid returning null in defaultUserApps when quota file doesn't config user (#822)
• [ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• Revert "[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• [ISSUE-772][0.7] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• [ISSUE-715] fix(mr): The container does not exit because shuffleclient is not closed (#882)
• [ISSUE-886] fix(mr): MR Client may lose data or throw exception when rss.storage.type without MEMORY (#887)
• Revert "[ISSUE-886] fix(mr): MR Client may lose data or throw exception when rss.storage.type without MEMORY (#887)
• [MINOR] fix: Fix LocalStorageManager divide by zero exception (#900)
• [ISSUE-881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0 (#914)
• [ISSUE-933][0.7] fix: incorrect metric grpc_server_connection_number (#941)
• change version to 0.7.1 release

Release v0.7.0

Apache Uniffle (Incubating) Release v0.7.0

Highlight

• Better support for Spark AQE
  • Leveraging the LOCAL_ORDER data distribution to improve performance
  • Estimating assignment number of shuffle servers to improve user experience
  • Better assignment policy of assigning adjacent partitions to the same shuffle server to improve performance
• Optimization of huge partition to improve stability of shuffle servers
• More bug fixes and usability improvements of K8S operator
• Add support of user quota management and more compression algorithms
• Add support of spark data eviction mechanism of stage level
• Some improvement of stability and performance

Changelog

• Add more badges in README by @kaijchen in #219
• Fix incorrect log format strings by @kaijchen in #220
• Change total lines badge url to sloc.xyz in README by @kaijchen in #222
• [MINOR] Fix warnings reported by lgtm by @kaijchen in #223
• [MINOR] Simplify creating buffer logic by @zuston in #227
• Support cancelling previous ci actions by @zuston in #225
• Use the conf of shuffleNodesNumber from jobs to be as checking factor by @zuston in #208
• Output the stderr and stdout to output file in startup script by @zuston in #226
• [ISSUE-48][FEATURE][FOLLOW UP] Add controller component by @wangao1236 in #214
• Add more metrics about requiring read memory by @zuston in #231
• Adjust the memory required times to match grpc max deadline conf by @zuston in #218
• [MINOR] Fix flaky test by @jerqi in #238
• [ISSUE-48][FEATURE][FOLLOW UP] Add yaml of components and crd exampes by @wangao1236 in #236
• Fix Flaky test GetShuffleReportForMultiPartTest by @leixm in #241
• Set the default disk capacity to the total space by @zuston in #237
• Add issue template by @jerqi in #8
• [MINOR] Fix inefficient map iteration by @kaijchen in #245
• Support deploy multiple shuffle servers in a single node by @xianjingfeng in #166
• Fast fail when reading failed in ComposedClientReadHandler by @zuston in #213
• Fix startup shell problem by @jerqi in #251
• New version 0.7.0-snapshot by @jerqi in #252
• [ISSUE-196] Fix flaky test about kerberos by @zuston in #250
• [ISSUE-48][FEATURE][FOLLOW UP] add unit test for validating rss objects by @wangao1236 in #248
• [Improvement] Add hdfs path health check to AppBalanceSelectStorageStrategy by @smallzhongfeng in #210
• [TYPO] Replace Chinese colon by ASCII colon by @kaijchen in #255
• Introduce startup-silent-period mechanism to avoid partial assignments by @zuston in #247
• Replace DISCLAIMER with DISCLAIMER-WIP by @jerqi in #258
• [ISSUE-244] Fix flaky test of CoordinatorGrpcTest.rpcMetricsTest by @zuston in #256
• Fix flaky test of ClientConfManagerTest by @smallzhongfeng in #260
• [Refactor] Optimize creating shuffle handlers by @zuston in #259
• Introduce data cleanup mechanism on stage level by @zuston in #249
• [ISSUE-48][FEATURE][FOLLOW UP] add docs for operator by @wangao1236 in #261
• Fix potenial missing reads of exclude nodes by @zuston in #269
• [ISSUE-257] RssMRUtils#getBlockId change the partitionId of int type to long by @fpkgithub in #266
• [ISSUE-273][BUG] Get shuffle result failed caused by concurrent calls to registerShuffle by @leixm in #274
• Add enum type test about case insensitive by @zuston in #280
• Support ZSTD by @zuston in #254
• [ISSUE-239][BUG] RssUtils#transIndexDataToSegments should consider the length of the data file by @leixm in #275
• Remove code quality badge and add release badge by @kaijchen in #284
• [ISSUE-163][FEATURE] Write to hdfs when local disk can't be write by @xianjingfeng in #235
• Upgrade Github actions for Node.js 16 by @kaijchen in #292
• Fix NPE in WriteBufferManager.addRecord by @wForget in #296
• Fix AbstractStorage#containsWriteHandler by @xianjingfeng in #281
• Add more test cases on LocalStorageManager.selectStorage by @zuston in #298
• [ISSUE-137][Improvement][AQE] Sort MapId before the data are flushed by @zuston in #293
• [ISSUE-283][FEATURE] Support snappy compression/decompression by @amaliujia in #304
• [ISSUE-290] Make RpcNodePort and HttpNodePort optional by @amaliujia in #305
• [ISSUE-301][Subtask][Improvement][AQE] Merge continuous ShuffleDataSegment into single one by @zuston in #303
• Cleanup RuntimeException and fetchRemoteStorage logic in ClientUtils by @kaijchen in #295
• [ISSUE-135][FOLLOWUP][Improvement][AQE] Assign adjacent partitions to the same ShuffleServer by @leixm in #307
• Correct the contributing guide link in pull-request template by @zuston in #314
• Fix bug of "Comparison method violates its general contract" by @zuston in #315
• [AQE][LocalOrder] Fix potenial bug when merging continuous segments by @zuston in #318
• [AQE][LocalOrder] Fix wrong param of expectedTaskIds in LocalOrderSegmentSplit by @zuston in #319
• [Feature] Support the estimated number of ShuffleServers required. by @leixm in #322
• [Bug] Fix potenial bug when the index reading offset is greater than data length by @zuston in #320
• [ISSUE-154][Improvement] Support Empty assignment to Shuffle Server by @rhh777 in #325
• [Bug] Fix invalid owner of host path volumes by @wangao1236 in #330
• [ISSUE-309][FEATURE] Support ShuffleServer latency metrics. by @leixm in #327
• [ISSUE-329]Catch NPE in ShuffleTaskManager#addFinishedBlockIds by @xianjingfeng in #331
• [BUG] Fix wrong method name by @leixm in #335
• [ISSUE-328] Cleanup unused shuffle servers after stage completed by @xianjingfeng in #334
• [MINOR] Migrate RankValue to the package of the common class by @smallzhongfeng in #265
• [BUG] Fix incorrect spark metrics by @zuston in #324
• [Improvement][LocalOrder] Add tests about keeping consistent with FixedSize when no skew optimization by @zuston in #336
• [INFRA] Add k8s pipeline by @jerqi in #340
• Remove unused class of RssShuffleUtils by @zuston in #345
• [ISSUE-342][Improvement] Check Spark Serializer type by @chong0929 in #344
• [Feature] Support user's app quota level limit by @smallzhongfeng in #311
• [BUG][AQE][LocalOrder] Fix the bug of missed data due to block sorting by @zuston in #347
• [ISSUE-364] Fix indexWriter don't close if exception thrown when close dataWriter by @xianjingfeng in #349
• [BUG] Fix flaky test of AQESkewedJoinWithLocalOrderTest by @zuston in #350
• Add collaborators by @jerqi in #351
• [BUG][FOLLOWUP] Fix flaky test of AQESkewedJoinWithLocalOrderTest by @zuston in https://github.com/...

Release v0.6.1

Apache Uniffle (Incubating) Release v0.6.1

Highlight

• Major bug fixes

  • Partition cannot be accessed in MapReduce when the reduce task number exceeds 1024.
  • Get shuffle result failure caused by concurrent calls to registerShuffle.
  • Inconsistent blocks caused by missing length in RssUtils#transIndexDataToSegments.
  • Handle NPE in WriteBufferManager#addRecord in the same way as Spark.
  • AbstractStorage#containsWriteHandler is checking the wrong Map.
  • indexWriter isn't closed if exception is thrown when closing dataWriter.
  • Incorrect dependency of protobuf-java at compile time.
  • Potential memory leak when encountering disk unhealthy.

• Minor bug fixes

  • Potenial missing reads of exclude nodes.
  • Incorrect contributing link in pull-request template.
  • Incorrect spark metrics.

ChangeLog

• [ISSUE-257] RssMRUtils#getBlockId change the partitionId of int type to long (#266)
• [ISSUE-273][BUG] Get shuffle result failed caused by concurrent calls to registerShuffle (#274)
• Fix potenial missing reads of exclude nodes (#269)
• [ISSUE-239][BUG] RssUtils#transIndexDataToSegments should consider the length of the data file (#275)
• Fix NPE in WriteBufferManager.addRecord (#296)
• Fix AbstractStorage#containsWriteHandler (#281)
• Correct the pull-request contributing link in template (#314)
• [BUG] Fix incorrect spark metrics (#324)
• [ISSUE-364] Fix indexWriter don't close if exception thrown when close dataWriter (#349)
• [ISSUE-228] Fix the problem of protobuf-java incorrect dependency at compile time (#362)
• Bump project version to 0.6.1
• [BUG] Potenial memory leak when encountering disk unhealthy (#370)

Release v0.6.0

Apache Uniffle (Incubating) Release v0.6.0

Highlight

• Optimize the assignment strategy

• Some improvement of stability and performance

• Add a plugin mechanism of SelectStorageStrategy

• Add LowestIOSampleCostSelectStorageStrategy

• Support Kerberos HDFS

ChangeLog

• Change license owner to ASF by @kaijchen in #5
• Trivial code improvements by @wForget in #7
• [Minor] Store shuffleId int to be consistent with other data structure by @zuston in #10
• Introduce the asList method in ConfigOptions by @zuston in #9
• Rename package by @jerqi in #6
• Minimize apache-rat excluded files by @kaijchen in #11
• Update module names by @kaijchen in #12
• Covert PartitionAssignmentInfo to static inner class by @pan3793 in #15
• [Followup] Migrate to Junit5 by @zuston in #14
• [Bug] Fix NPE problem when process the event if application was cleared already by @colinmjj in #16
• [CI] Enable codecov report by @kaijchen in #17
• Correct the config description and fix typo by @zuston in #19
• Add CI and Codecov badges in README by @kaijchen in #20
• [Followup] Use asList method in some existing configOptions by @zuston in #18
• Move rss-integration-spark-common-test module package by @wForget in #23
• [INFRA] Improve asf.yaml to reduce the notifications by @jerryshao in #25
• [TEST] Improve code coverage in rss-common by @kaijchen in #26
• Remove redundant package by @wForget in #27
• [CI] Switch to temurin JDK by @kaijchen in #24
• [INFRA] Improve asf.yaml to reduce the notifications (another-try) by @jerryshao in #33
• Bump commons-lang3 from 3.5 to 3.10 by @wForget in #28
• Fix the log of incorrectly bound class by @wForget in #35
• [TYPO] Fix misspelled word "integration" by @kaijchen in #34
• Fix some hyperlink in README.md by @daugraph in #32
• Upgrade gRPC to support Apple Silicon by @pan3793 in #13
• Allow to specify custom tags to decide the assignment of servers by @zuston in #30
• Optimize the bash script by @zuston in #29
• [Improvement] reduce compiler warnings by @advancedxy in #46
• [Chore]: document update and build time optimize by @advancedxy in #45
• Supplement doc about assignment tags by @zuston in #47
• [Bug] Fix skip() api maybe skip unexpected bytes which makes inconsistent data by @colinmjj in #40
• [improvement] Remove experimental feature with ShuffleUploader by @colinmjj in #51
• [Improvement] Provides utility classes for creating thread factories by @smallzhongfeng in #49
• Enable spotbugs and fix high priority bugs by @kaijchen in #38
• [CI] Change default checkstyle severity to error by @kaijchen in #57
• [Style] Check indentation by @kaijchen in #56
• [Experimental Feature] MR Supports Remote Spill by @frankliee in #55
• [Improvement] Log indicate the shuffle server host:port when doing re… by @zuston in #58
• Send commit concurrently in client side by @zuston in #59
• Explicitly set the constructor with AccessManager when extending AccessChecker by @zuston in #43
• [DOC] Replace Firestorm with Uniffle by @jerqi in #60
• Introduce the extraProperties to support user-defined pluggable accessCheckers by @zuston in #42
• Log enhancement: Merge multiple logs into oneline and add more description by @zuston in #62
• [TEST] Add more unit tests in rss-common by @kaijchen in #63
• [MINOR] Comments of PartitionBalanceAssignmentStrategy miss byte units by @smallzhongfeng in #68
• [Minor] Make config keys and default values finalized by @kaijchen in #70
• [Log Improvment] Add more detailed debug info for MR client by @frankliee in #84
• [Improvement] Shutdown the grpc executors pool when closing by @zuston in #83
• Log enhancement: return error message when getting assignment servers and log exception when initializing by @zuston in #64
• [ISSUE-48] [Feature] Init Kubernetes operator directory by @jerqi in #75
• [Improvement] No need to use synchronized lock of the method scope when getting client by @zuston in #82
• [DOC] Remove Wechat group in README by @jerqi in #88
• [Performance Optimization] Improve the speed of writing index file in shuffle server by @zuston in #91
• [DOC] Update title and description in README by @kaijchen in #94
• [Improvement] ShuffleBlock should be release when finished reading by @xianjingfeng in #74
• [IMPROVEMENT][COMMON] Fix common module code style by @jerqi in #99
• [Improvement]LocalStorage init use multi thread #71 by @xianjingfeng in #72
• [Improvement] Use OR operation instead of serialization for cloning BitMaps by @kaijchen in #103
• [Improvement] Ignore partial failure on initializing local storage in shuffle server side by @zuston in #102
• [CI] Test compile in Java 11 and Java 17 by @kaijchen in #105
• Sleep less time but try more times when stopping by @xianjingfeng in #112
• [Improvement] Use ConfigBuilder to rewrite the class RssSparkConfig by @smallzhongfeng in #104
• [Improvement] Introduce config to customize assignment server numbers in client side by @zuston in #100
• Assign partition again if registerShuffleServers failed by @xianjingfeng in #115
• [ISSUE-106][IMPROVEMENT] Set rpc timeout for all rpc interface by @xianjingfeng in #113
• [MINOR][IMPROVEMENT] Avoid CoordinatorServer#initialization multiple new Configuration() by @zwangsheng in #118
• [Improve] Remove useless server id from StorageManagerFactory#createStorageManager by @zwangsheng in #119
• [MINOR][IMPROVEMENT][COORD] Fix coordinator module code style by @jerqi in #122
• [Improvement] Set heartBeatExecutorService as daemon thread by @smallzhongfeng in #121
• [JUnit] Introduce the property of trimStackTrace to show error stacktrace in mvn-test by @zuston in #126
• Make the conf of rss.storage.basePath as list by @zuston in #130
• [MINOR][IMPROVEMENT][STORAGE] Fix storage module code style by @jerqi in #131
• [Improvement] Add timeout reconnection when DelegationRssShuffleManager send the request of AccessCluster by @smallzhongfeng in #139
• [MINOR] Fix flaky test testGetHostIp by @izchen in #141
• [Improvement] Add the number of unhealthy nodes in CoordinatorMetrics by @smallzhongfeng in #147
• [ISSUE-48][FEATURE] Add Uniffle Dockerfile by @wangao1236 in #132
• [BUGFIX] Fix memory leak which cause oom by @summaryzb in #145
• [Log Improvement] Output the re...