[v4.y] Test VERIFY_PEER / VERIFY_NONE work against real cluster

[cben]

Followup to #540. (It fixed #525 which was only broken on master, not v4.y, but good to have these tests here too.)

Adding tests that Client honor VERIFY_PEER/VERIFY_NONE by connecting to the temporary cluster during test/config/update_certs_k0s.rb script.
They are skipped during a regular rake test so they will NOT run in CI.
(Can we run docker --priviledged in github actions?)

[cben]

Ooh, exciting! Yes, it can run in CI. (IIUC service containers can't be --priviledged, but directly executing docker run --priviledged on Linux works 💡)

• it failed with SSL errors differing from what I expected 🎉
  https://github.com/ManageIQ/kubeclient/runs/5346969200?check_suite_focus=true#step:6:60

  Some runs pass the full tests, some fail. Mostly 3.0 was the first to fail but saw a time it was 2.5 that failed. I'm starting to think it's simple race condition with k0s starting up — kubeconfig existing doesn't guarantee apiserver is up yet?
  => Yep, saw curl also failing:
  curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:6443
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.2:6443
  => added polling until apiserver is serving.

• Should switch to docker.pkg.github.com/k0sproject/k0s/k0s so GitHub actions don't hammer docker hub.

• Can it run on macos / windows? NO.

[cben]

OK pin-pointed the timeout:

KubeclientRealClusterTest#test_real_cluster_verify_none = 1.15 s = .
KubeclientRealClusterTest#test_real_cluster_verify_peer = 
Error: Process completed with exit code 124.

Probably exercise_watcher_with_timeout, which suggests .finish on watches is generally not as reliable as I believed :-(

[cben]

@russell @grosser @agrare I still need to tackle the runaway test ^ but please review the general direction.

I want real end-to-end tests of when TLS verification actually verifies, and the setup here of starting a real cluster works surprisingly fast and opens the door to many more future real-cluster tests...

[grosser]

❤️ nice approach

[cben]

Ignoring truffleruby, which always fails anyway (opened #551).
So far I failed to reproduce locally the new cases sometimes getting stuck, only happens on CI, at random.

Same CI situation adding these on master branch (#550).

But these provide real value so I'm going to merge and we'll see how flaky it is in future... (Worst-case, can always fall back to not running real-cluster tests in CI.)