Investigate and document correlation between Rancher cacerts and server-url settings

[anmazzotti]

The behavior of server-url and cacerts need better documentation to inform the user how Elemental uses these settings.

These settings are used for the elemental-register, and elemental-system-agent (and rancher-system-agent?) configs.
While the elemental-register can cope with a mismatching cacerts value, the elemental-system-agent is going to fail if the cacerts is not matching the server-url's certificate.

This is because the server-url is used to create the kubeconfig used by the elemental-system-agent, and by default we use the cacerts value to populate the certificate-authority-data in the config.

If I understand this correctly, the kubeconfig strictly uses the certificate-authority-data value to validate the k8s API. So this has a large impact when the server-url is pointing to a secondary ingress for example.

I think this is the relevant doc: https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/resources/update-rancher-certificate

Also relevant, the tls-rancher-ingress CA can be easily checked on https://my.rancher.domain/cacerts

[anmazzotti]

Trying to successfully bring up a private CA Rancher environment, but hardcoding the cacerts.pem path does not really work well with cert-manager and tls secrets. This would need to be reopened: rancher/rancher#36994

[anmazzotti]

After playing with private CA setup and switching CAs, I have a few questions still to answer:

1. Does the process to update the Rancher certificate cover the rancher-system-agent that Elemental installs? More in specific is there anything already updating /var/lib/rancher/agent/rancher2_connection_info.json?

2. When using an ACME provider, like LetsEncrypt, does the cacerts Rancher setting get updated with the new CA or is that going to be empty?

3. There seem to be no process to update /var/lib/elemental/agent/elemental_connection.json, which is driven by the yip config: /oem/elemental-system-agent.yaml. This is going to be a problem, since we have no way to update anything on the machine once the elemental-system-agent fails due to the CA renewal.

[anmazzotti]

Good news, the TLS enforcement mode can be configured on agents: https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-references/tls-settings#agent-tls-enforcement
This is useful for the LetsEncrypt/empty cacerts scenario.

[anmazzotti]

Spike is done, closing as a new set of issues was created as result: #1612