TODO: unclean shutdown of envtest e2e tests

[jiridanek]

Harmless, but annoying

2024-10-22T13:38:47+02:00	INFO	Stopping and waiting for webhooks
2024-10-22T13:38:47+02:00	INFO	controller-runtime.webhook	Shutting down webhook server with timeout of 1 minute
2024-10-22T13:38:47+02:00	INFO	Stopping and waiting for HTTP servers
2024-10-22T13:38:47+02:00	INFO	Wait completed, proceeding to shutdown the manager
2024-10-22T13:38:47+02:00	DEBUG	controller-runtime.certwatcher	certificate event	{"event": "REMOVE        \"/var/folders/f1/3m518k5d34l72v_9nqyjzqm80000gn/T/envtest-serving-certs-2153001584/tls.key\""}
2024-10-22T13:38:47+02:00	ERROR	controller-runtime.certwatcher	error re-watching file	{"error": "fsnotify: watcher already closed"}
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
	/Users/jdanek/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:185
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
	/Users/jdanek/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:133
2024-10-22T13:38:47+02:00	ERROR	controller-runtime.certwatcher	error re-reading certificate	{"error": "open /var/folders/f1/3m518k5d34l72v_9nqyjzqm80000gn/T/envtest-serving-certs-2153001584/tls.crt: no such file or directory"}
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
	/Users/jdanek/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:190
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
	/Users/jdanek/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:133


Ran 2 of 2 Specs in 10.003 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 0 Skipped

This is something that people in general struggled with

• envtest - Failed to Watch error on every test run kubernetes-sigs/controller-runtime#904
• Kubebuilder v2 improvements + ideas gocardless/theatre#210

and we have a TODO comment for it

https://github.com/opendatahub-io/kubeflow/blob/0743e0e444f236b75d90eb2053ac0e79ead5950f/components/odh-notebook-controller/controllers/suite_test.go#L189-L195

The solution seems to be to create a nested context for

subctx, cancel := context.WithCancel(ctx)

...

	// Start the manager
	go func() {
		defer GinkgoRecover()
		err := mgr.Start(ctx)
		Expect(err).ToNot(HaveOccurred(), "Failed to run manager")
	}()

...

// when we're done testing
cancel()

but it needs to be checked.

[jiridanek]

^^ that nested context does not seem to be good enough, I still see unclean shutdowns