[SPARK-56511][CORE][TESTS][FOLLOWUP] Disable off-heap memory in ShuffleExternalSorter cleanupResources test#55409
Closed
viirya wants to merge 1 commit intoapache:masterfrom
Closed
[SPARK-56511][CORE][TESTS][FOLLOWUP] Disable off-heap memory in ShuffleExternalSorter cleanupResources test#55409viirya wants to merge 1 commit intoapache:masterfrom
viirya wants to merge 1 commit intoapache:masterfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Followup to #55373.
Explicitly disable off-heap memory (spark.memory.offHeap.enabled=false) in the SparkConf used by the new "cleanupResources should not NPE when reset fails to reallocate array" test in ShuffleExternalSorterSuite.
Why are the changes needed?
The test's TaskMemoryManager override uses MemoryMode.ON_HEAP when stealing memory via acquireExecutionMemory, and the test tunes spark.testing.memory / spark.memory.fraction which only govern on-heap memory. If off-heap memory is enabled in the environment where the test runs (e.g., via a global default or driver config), the on-heap memory-stealing path may not reliably trigger the intended allocateArray() OOM, and the test can become flaky or fail to cover the regression it was introduced for.
Explicitly disabling off-heap memory in the test's SparkConf makes the setup hermetic and independent of the caller's environment, keeping the test focused on the on-heap scenario it is designed to reproduce.
Does this PR introduce any user-facing change?
No - test-only change.
How was this patch tested?
Existing test ("cleanupResources should not NPE when reset fails to reallocate array") continues to pass.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code