Force merge with only_expunge_deletes should honor max segment size

[msfroh]

Is your feature request related to a problem? Please describe.
There is longstanding logic that ignores the max segment size when doing any kind of forced merge, here https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/OpenSearchTieredMergePolicy.java#L59.

This makes sense when you have an index that stops receiving updates, has been switched to read-only, and you want to merge down to a single segment.

There are times, though, when someone just wants to periodically expunge deletes, because soft deletes mean that they have accumulated way too many deleted docs. As mentioned in e.g. the answer to this Stackoverflow question, this may result in segments that are larger than the max segment size, so they will never merge away deletes going forward (unless there are so many deletes that the segment could drop back below the max segment size).

Describe the solution you'd like
I would like a force merge with only_expunge_deletes to honor the max segment size.

The simple way to do that is to change this line to use regularMergePolicy instead of forcedMergePolicy.

Describe alternatives you've considered
I guess we could add another option to the force merge API to say "merge away deletes, but still honor the max segment size" (or even just add a general "honor the max segment size" argument).

In my opinion, only_expunge_deletes already implies that I just want to get rid of deletes and am not trying to merge to one big segment.

Additional context
N/A

[andrross]

This makes sense when you have an index that stops receiving updates, has been switched to read-only, and you want to merge down to a single segment.

Does supplying max_num_segments=1 accomplish this explicitly?

Is there some other reason why the current behavior of only_expunge_deletes might be desirable? I agree with your analysis but it seems odd it wasn't implement the way you've proposed here from the beginning.

[msfroh]

Does supplying max_num_segments=1 accomplish this explicitly?

Yeah, specifying max_num_segments will really merge down to a single segment, ignoring the configured max segment size.

When Lucene made the changes to ensure that any force-merge would honor the max segment size constraint, it broke that behavior. That's why this composite merge policy was added -- to get back that old "merge down to a single segment regardless of max segment size" behavior.

Is there some other reason why the current behavior of only_expunge_deletes might be desirable? I agree with your analysis but it seems odd it wasn't implement the way you've proposed here from the beginning.

I could be mistaken, but I think there was a desire to just get back the old force-merge behavior, essentially undoing the Lucene change from https://issues.apache.org/jira/browse/LUCENE-7976.

In my opinion, this was wrong. I love the singleton-merging of large segments (to expunge deletes) that https://issues.apache.org/jira/browse/LUCENE-7976 introduced.