Add per-service mutex to prevent concurrent reconciliation

[mweibel]

Prevents duplicate LB creation when multiple goroutines process the same service concurrently (e.g. EnsureLoadBalancer + UpdateLoadBalancer triggered by node sync).

Uses sync.Map keyed by service UID. Locks are not cleaned up on service deletion to avoid issues with late-arriving goroutines.

Includes a failing unit test that reproduces the concurrent creation race.

[alakae]

I compared it with: https://github.com/cloudscale-ch/terraform-provider-cloudscale/blob/master/cloudscale/mutex_kv.go

• any reason why you did not use the channel pattern you suggested in that repo?
• I just now realized that both implementations do never actually delete map entries. In Terraform this should not be a problem, as the whole process is short-lived, in the CCM the process could live for weeks/months. Do you have an idea how we can address it? I don't, at least not a trivial one :)

[mweibel]

any reason why you did not use the channel pattern you suggested in that repo?

the mutex_kv.go is a slightly different case: it used a loop with timer for waiting until the lock is there. It also supports cancellation with context to support terraform-native timeouts (and ctrl+c). It used TryLock for this and we changed it to a channel-based approach to get rid of the loop and timer and instead use "native" Go polling.

This case here is different and much simpler: we just acquire the lock and nobody else waits on it. We don't really need context cancellation here since it's running as a service. While it may happen that the parent context could be cancelled for some reason, the lock/unlock should not be the point where this gets respected (and hasn't been so far).

I just now realized that both implementations do never actually delete map entries. In Terraform this should not be a problem, as the whole process is short-lived, in the CCM the process could live for weeks/months. Do you have an idea how we can address it? I don't, at least not a trivial one :)

That's true and I tried to address this in the commit message:

Locks are not cleaned up on service deletion to avoid issues with late-arriving goroutines.

How often this happens is questionable but it avoids potential issues without much cost: Even large clusters won't have more than a couple loadbalancers. Keeping them around would be a couple 100 bytes (one entry is ~150B). Of course if somebody creates and deletes services 1000s of times it eventually could crash the CCM, but that would rather speak for a misuse of the system than anything else TBH.

Still, We could remove the entry for a service in two ways:

1. once EnsureLoadBalancerDeleted is successfully done
2. Add a goroutine which periodically removes unused mutexes. This would need to track which services still exist.

point 1 is rather trivial, I just wonder if there are any edge cases. I don't think so, but I didn't add this change because the benefit is not 100% clear to me.

Let me know what you think - happy to add the deletion logic if you feel it's valuable.

[mweibel]

As an aside: we could also use a plain map with an accompanying single mutex in this case. I think either approach would work well.