Skip to content

Latest commit

 

History

History
189 lines (136 loc) · 5.41 KB

cilium-image.md

File metadata and controls

189 lines (136 loc) · 5.41 KB
Raw

Instructions for installing Cilium on OpenShift local

This instructions have been used on a machine with Fedora 38.

Pre-requisites for OpenShift local build

Have KVM/libvirt enabled

Fedora virtualization getting started

Configuration

DNS

dnsmasq

Configured by the installer (nothing to do)

cat etc/NetworkManager/dnsmasq.d/crc-snc.conf
server=/crc.testing/192.168.126.1
address=/apps-crc.testing/192.168.126.11

systemd-resolved

/etc/systemd/resolved.conf

DNS=127.0.0.1
Domains=crc.testing,apps-crc.testing

sytemctl restart systemd-resolved.service

Script modifications

--- a/snc.sh
+++ b/snc.sh
@@ -133,10 +133,17 @@ ${YQ} eval --inplace ".metadata.name = \"${SNC_PRODUCT_NAME}\"" ${INSTALL_DIR}/i
 ${YQ} eval --inplace '.compute[0].replicas = 0' ${INSTALL_DIR}/install-config.yaml
 replace_pull_secret ${INSTALL_DIR}/install-config.yaml
 ${YQ} eval ".sshKey = \"$(cat id_ecdsa_crc.pub)\"" --inplace ${INSTALL_DIR}/install-config.yaml
+${YQ} eval --inplace ".networking.networkType = \"Cilium\"" ${INSTALL_DIR}/install-config.yaml
 
 # Create the manifests using the INSTALL_DIR
 OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=$OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE ${OPENSHIFT_INSTALL} --dir ${INSTALL_DIR} create manifests
 
+# Add Cilium manifests
+cilium_version="1.14.3"
+git_dir="$(mktemp -d)"
+git clone https://github.com/isovalent/olm-for-cilium ${git_dir}
+cp ${git_dir}/manifests/cilium.v${cilium_version}/* "${INSTALL_DIR}/manifests"
+
 # Add CVO overrides before first start of the cluster. Objects declared in this file won't be created.
 ${YQ} eval-all --inplace 'select(fileIndex == 0) * select(filename == "cvo-overrides.yaml")' ${INSTALL_DIR}/manifests/cvo-overrides.yaml cvo-overrides.yaml

libvirt access

The installer needs to communicate with libvirt. Therefore it uses libvirt default network. If a different network range is configured for the default network the URI is install-config.yaml needs to be amended to point to the default gateway of the configured default network.

platform:
  libvirt:
    URI: qemu+tcp://192.168.124.1/system

Building OpenShift local

A pull secret can be downloaded from Red Hat portal.

export OPENSHIFT_PULL_SECRET_PATH=<path-to-pull-secret>
./snc.sh

Time synchronization

For whatever reason the script deactivates time sync and I had to reactivate it afterwards to avoid issues with TLS communication.

timedatectl set-ntp true

SSH

a key pair is generated by the script (instead of using the user configuration). It is however possible to ssh the bootstrap and master using the key in the current directory.

# bootstrap
ssh -i id_ecdsa_crc core@192.168.126.10
# master
ssh -i id_ecdsa_crc core@192.168.126.11

Accessing the cluster

To access the cluster as the system:admin user when using oc, run export KUBECONFIG=<path to crc-snc repo>/crc-tmp-install-data/auth/kubeconfig

Creating a disk image

./createdisk.sh crc-tmp-install-data

Issue

operator olm deployments

cilium-olm and cilium-operator are started twice and the installer is somehow waiting for them to get ready.

cilium          cilium-olm-555d455d68-w6w5h         1/1     Running     14 (14m ago)   25h
cilium          cilium-olm-6865b55f76-bws8q         0/1     Pending     0              25h
cilium          cilium-operator-75c959989b-7kvpx    0/1     Pending     0              25h
cilium          cilium-operator-75c959989b-dn9lt    1/1     Running     8 (14m ago)    25h

This won't happen on a single node cluster:

message: '0/1 nodes are available: 1 node(s) didn''t have free ports for the requested
pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found

cilium-olm can be fixed by amending the following files cluster-network-06-cilium-00014-cilium.v1.14.3-x041dc10-clusterserviceversion.yaml cluster-network-06-cilium-00002-cilium-olm-deployment.yaml

spec:
  strategy
    type: Recreate
  resources:
    requests:
      cpu: 100m
      memory: 512Mi

cilium-operator is supposed to be configured through ciliumconfig /cluster-network-07-cilium-ciliumconfig.yaml

operator.replicas: 1
operator.resources: {"requests":{"cpu": "10m", "memory": "64Mi"}}
operator.updateStrategy: `{"type":"Recreate"}`

In yaml:

operator:
  replicas: 1
  resources: 
    requests:
      cpu: "10m"
      memory: "64Mi"
  updateStrategy: 
    type: "Recreate"

This could be scripted in snc.sh. The updateStrategy cannot however be changed as it induces conflict with the helm value file backed in the operator image.

Disk image generation

For the createdisk.sh script to be successful on Fedora it was necessary to deactivate Windows image generation:

export SNC_GENERATE_WINDOWS_BUNDLE=0
./createdisk.sh

CRC

It is needed to build from source a CRC version without a bundle: https://github.com/crc-org/crc/pull/1535/files

The crc repository needs to be cloned and simply run the comand make.

Running CRC with Cilium

The bundle previously created can then be used with the following command.

 ~/go/bin/crc --bundle=../crc-snc/crc_libvirt_4.14.1_amd64.crcbundle setup
 ~/go/bin/crc --bundle=../crc-snc/crc_libvirt_4.14.1_amd64.crcbundle start