The Future of Kubernetes: Rancher RKE2 and Cilium CNI

Deploying Rancher RKE2 with Cilium CNI and kube-proxyless

Published in

Level Up Coding

7 min readOct 19, 2023

Cilium’s ascent in the CNCF community is undeniable. Major cloud providers have already recognized the potential of Cilium and have incorporated it into their offerings. As Saiyam Pathak, CNCF ambassador and CTO at managed Kubernetes service provider Civo, notes,

“Kubernetes has established itself as the de facto platform for container orchestration, and Cilium is rapidly becoming the de facto CNI solution of choice for Kubernetes.”

In this post, we’ll dive deeper into the growing importance of Cilium by focusing primarily on its CNI implementation and kube-proxyless deployment. As an underlying orchestrator, we’re going to set up the Rancher RKE2 Kubernetes distribution.

First off, why RKE2?

When it comes to selecting the ideal Kubernetes distribution, your decision hinges on the specific use case at hand. K3s excels in catering to edge deployments on resource-constrained devices, while Microk8s and Kind prioritize ease of use. In my view, RKE2 distinguishes itself by offering a robust foundation for meticulously configuring your clusters, especially when stringent security and compliance standards are non-negotiable. Regardless of the level of security your environment demands, RKE2 empowers you to configure and adhere to the latest security best practices.

While it may not be the most suitable choice for the demonstration application we’ll be deploying later, I personally believe that experimenting with different distribution setups is invaluable for gaining insights into the deployment and maintenance challenges.

The Power of Cilium CNI

Cilium isn’t just another Container Networking Interface (CNI). It’s a versatile solution that offers a wide range of features, from load balancing to providing egress gateways. However, our focus today is on deploying RKE2 with Cilium CNI and going kube-proxyless. Traditional kube-proxy, which uses iptables, has often been seen as a bottleneck for larger deployments due to its unscalable design. Cilium steps in by using eBPF to forward packets directly to their destination pods as soon as they reach the host, bypassing the need for time-consuming iptables rule evaluations. In essence, eBPF programs are the heroes attached in the kernel, ensuring speedy and efficient packet routing.

What’s also important to realize is that by leveraging eBPF, not only does Cilium enhance network performance, but it also reduces the load on the host, enabling it to handle a higher volume of packets than ever before.

Enough about the theory, let’s look at how we can setup a cluster and a demo application on top of it.

Cluster Setup

First, we’ll initiate the deployment of the RKE2 cluster. The Rancher documentation already offers a comprehensive guide on this process, but in essence, it involves executing just three commands to set up RKE2 as a Systemd service.

curl -sfL https://get.rke2.io | sh -
systemctl enable rke2-server.service
systemctl start rke2-server.service

Since RKE2 comes preinstalled with the Canal CNI by default, we need to reconfigure the deploymeny, for our purpose. In order to install it with Cilium CNI and configure it using kube-proxyless, we’re gonna need to apply two changes:

/etc/rancher/rke2/config.yaml

cni: "cilium"
disable-kube-proxy: true

/var/lib/rancher/rke2/manifests/rke2-cilium-config.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    kubeProxyReplacement: true
    k8sServiceHost: "<server-ip>"
    k8sServicePort: "<server-port"
    ingressController:
      enabled: false
    gatewayAPI:
      enabled: true

In order to apply the changes, you need to restart the rke2-server.service using:

systemctl restart rke2-server.service

As one might have already noticed we’ve also configured the gatewayAPI in the latter config, since we’re gonna need that for our demo application.

Deployment

As a test deployment, that we will expose to the external traffic, we’re gonna use the following demo application by Istio authors:

# Copyright Istio Authors
#
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

##################################################################################################
# This file defines the services, service accounts, and deployments for the Bookinfo sample.
#
# To apply all 4 Bookinfo services, their corresponding service accounts, and deployments:
#
#   kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
#
# Alternatively, you can deploy any resource separately:
#
#   kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml -l service=reviews # reviews Service
#   kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml -l account=reviews # reviews ServiceAccount
#   kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml -l app=reviews,version=v3 # reviews-v3 Deployment
##################################################################################################

##################################################################################################
# Details service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: details
  namespace: default
  labels:
    app: details
    service: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-details
  namespace: default
  labels:
    account: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  namespace: default
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
    spec:
      serviceAccountName: bookinfo-details
      containers:
      - name: details
        image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        securityContext:
          runAsUser: 1000
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: ratings
  namespace: default
  labels:
    app: ratings
    service: ratings
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-ratings
  namespace: default
  labels:
    account: ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratings-v1
  namespace: default
  labels:
    app: ratings
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratings
      version: v1
  template:
    metadata:
      labels:
        app: ratings
        version: v1
    spec:
      serviceAccountName: bookinfo-ratings
      containers:
      - name: ratings
        image: docker.io/istio/examples-bookinfo-ratings-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        securityContext:
          runAsUser: 1000
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: reviews
  namespace: default
  labels:
    app: reviews
    service: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-reviews
  namespace: default
  labels:
    account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v1
  namespace: default
  labels:
    app: reviews
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v1
  template:
    metadata:
      labels:
        app: reviews
        version: v1
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: docker.io/istio/examples-bookinfo-reviews-v1:1.16.2
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
        securityContext:
          runAsUser: 1000
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v2
  namespace: default
  labels:
    app: reviews
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v2
  template:
    metadata:
      labels:
        app: reviews
        version: v2
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: docker.io/istio/examples-bookinfo-reviews-v2:1.16.2
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
        securityContext:
          runAsUser: 1000
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v3
  namespace: default
  labels:
    app: reviews
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v3
  template:
    metadata:
      labels:
        app: reviews
        version: v3
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: docker.io/istio/examples-bookinfo-reviews-v3:1.16.2
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
        securityContext:
          runAsUser: 1000
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: productpage
  namespace: default
  labels:
    app: productpage
    service: productpage
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: productpage
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-productpage
  namespace: default
  labels:
    account: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  namespace: default
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      labels:
        app: productpage
        version: v1
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
      - name: productpage
        image: docker.io/istio/examples-bookinfo-productpage-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        securityContext:
          runAsUser: 1000
      volumes:
      - name: tmp
        emptyDir: {}
---

Apply it using kubectl apply -f deployment.yaml. There after we’re also gonna need the gateway deployment which you can deploy in a similar fashion using kubectl apply -f gateway.yaml:

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: my-gateway
  namespace: default
spec:
  gatewayClassName: cilium
  listeners:
  - protocol: HTTP
    port: 80
    name: web-gw
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-app-1
  namespace: default
spec:
  parentRefs:
  - name: my-gateway
    namespace: default
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /details
    backendRefs:
    - name: details
      port: 9080
  - matches:
    - headers:
      - type: Exact
        name: magic
        value: foo
      queryParams:
      - type: Exact
        name: great
        value: example
      path:
        type: PathPrefix
        value: /
      method: GET
    backendRefs:
    - name: productpage
      port: 9080

Since the gatewayAPI itself required non-standard CRD, we’re gonna need to install them to:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.8.0/experimental-install.yaml

Make sure, you update the version of the installed CRDs in the URL if you prefer to work with the latest release.

And this is it — essentially this should enable you to have your demo application running on your RKE2 cluster, utilizing Cilium that is accessible through the server public IP.

Conclusion

The combination of Rancher RKE2 and Cilium CNI presents a compelling vision for the future of Kubernetes. As you explore these technologies further, you’ll witness a transformation in the way we manage and optimize network performance in our clusters.

Stay tuned for more exciting topics, including IPAM and Ingress, where Cilium’s capabilities can significantly enhance cluster performance. The journey to harnessing the full potential of Kubernetes is far from over.

To stay current with the latest cloud technologies, make sure to subscribe to my weekly newsletter, Cloud Chirp. 🚀