Allow callers to define the scaling strategy of the FLAME pool

[nickdichev-firework]

I've been evaluating FLAME to replace the lambda architecture we are using for one of our services. We are currently using Kubernetes jobs as the runner. We have explicitly chosen to make the job/lambda call one-to-one due to the constraints of our application's requirements.

As such, in my first few experiments with FLAME I set the :max_concurrency option to 1 and :single_use to true. This works okay, however, due to the logic of the existing "max concurrency per runner" strategy, it causes us to get penalized waiting for a new k8s pod to spin up more often than we can tolerate.

The strategy we are trying to implement is "constant number of overprovisioned workers". I think the behavior I've extracted is flexible enough for all callers to be able to define their own strategies, however, I am definitely looking for feedback there.

There are still some rough edges in the PR, but I was hoping to see what you all think about this general direction.

For reference, here's the strategy module I've implemented in my app: https://gist.github.com/nickdichev-firework/e24530a6a36833c9c5aff8fb9ed8f970

[nickdichev-firework]

One issue I've realized with my approach is that the initial booted runners don't take into account our desired "overprovisioned" count, this one can be worked around by taking that into account in our :min config.

Similarly, I there's a problem where the min workers can be shutdown due since we use single_use: true and as such the pool can be scaled down past the :min config if there is no waiting callers due to the has_unmet_servicable_demand? check in the :DOWN handler. I'm still trying to write a test to exhibit the behavior, but I observed it in an actual deployment.

I thought about also moving that callback to the behavior as well, since I think my implementation would like to do it slightly differently than is implemented in the existing Pool implementation.

[nickdichev-firework]

Hi @chrismccord I'm curious if you had a chance to look at this one.

I saw some of the code I touched here has changed in main, so hoping to catch you while that's fresh in your mind.