Fix excessive stack usage when calling vorbis_analysis_wrote
with lots of samples
#104
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
vorbis_analysis_wrote
incrementsv->pcm_current
byvals
, and this incremented value can be used by_preextrapolate_helper
right after to allocate a float array in the stackv->pcm_current
positions large. Clearly, sincealloca
does not check that there is enough stack space available to satisfy the allocation request, this can lead to a stack overflow and memory corruption, which at best have no effect, more likely cause segmentation faults, and at worst introduce security risks.The documentation for
vorbis_analysis_buffer
andvorbis_analysis_wrote
does not specify a maximum value forvals
. It states that "1024 is a reasonable choice", but callers are free to use larger or smaller counts as they wish. Therefore,libvorbis
not handling this case is undesirable behavior.To better handle this case without throwing the performance benefits of
alloca
out the window, let's check whether the allocation would exceed 256 KiB (an estimate for the minimum stack space available is 1 MiB, which is the default on Windows platforms), and if so fall back to a heap allocated array. The heap array that may be allocated for this purpose is freed whenvorbis_dsp_clear
is called._preextrapolate_helper
takes neglible execution time compared to the encoding process for usual sample block sizes, though.