Kubernetes: Change base YAML config for different environments prod/test using Kustomize

We would like to have a base set of Kubernetes config YAML files which we then want to alter depending on the environment. We will use Kustomize as the tool to accomplish this.

Content

• Part1: Merge multiple YAML files into one
• Part 2 (this article): Change base YAML config for different environments

Install Kustomize

Make sure to have the latest kubectl installed because Kustomize is already included (kubectl kustomize). Otherwise install it manually.

What does Kustomize need server side?

Nothing. All Kustomize does is merge various YAML files into one. You can then kubectl apply -f merged.yaml this file. It doesn’t matter if you created the merged.yaml manually or using Kustomize.

Today’s Scenario

We will have three environments:

• base: contains a set of YAML configuration on which other environments can be based on
• test: extends base, will change the image of the base deployment
• prod: extends base, will change image, service type, and adds a completely new resource

Github project

The whole example is finished in this repo: https://github.com/wuestkamp/kubernetes-kustomize-example

Base Env

Create a folder base containing three files:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      id: app
  template:
    metadata:
      labels:
        id: app
    spec:
      containers:
      - image: XXX # needs to be overridden by environments
        name: nginx

service.yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: service
  name: service
spec:
  ports:
  - name: 80-80
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30080
  selector:
    id: app
  type: NodePort

kustomize.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

Now when we run kubectl kustomize build base, it will output a Multi Resource File containing both the Service and Deployment. Deploying this would horribly fail though, because there is a wrong image name set:

compiled base env yaml

Test Env

Create a new folder overlays. In it create a new subfolder test and two files:

It seems to be common to use a folder named overlays, but you can name and change the directory structure as you like.

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app # necessary for Kustomize identification
spec:
  template:
    spec:
      containers:
      - image: nginx:1.17.3
        name: nginx # necessary for Kustomize identification

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base # relative path to the base folder

patchesStrategicMerge:
  - deployment.yaml

Running kubectl kustomize build overlays/test will now use all the base resources and replaces just the image name for the Deployment to nginx:1.17.3, which should result in this:

compiled test env yaml

Because we defined a patchesStrategicMerge entry for deployment, Kustomize will look for the file deployment.yaml and overrides specified values from the base deployment.yaml. The name attributes are important to specify for the to-be-overridden resource identification.

Prod Env

In production we would like to do even more. We would like to replace the image name, the replicas count, the service type and add a new resource. For this we create four files:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - image: nginx:stable
        name: nginx

service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  type: LoadBalancer

storage.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Mi

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

resources:
  - storage.yaml # a new resource

patchesStrategicMerge:
  - deployment.yaml
  - service.yaml

Now running kubectl kustomize build overlays/prod will compile the production yaml, which should look like this:

compiled prod env yaml

Deployment

For deploying an environment configuration you can run:

kubectl kustomize build overlays/prod | kubectl apply -f -

Or

kubectl kustomize build overlays/prod > tmp_prod.yaml
kubectl apply -f tmp_prod.yaml

There is always more

We only used one patching strategy (patchesStrategicMerge), but there is also a JSON Patch one.

(Helm) Templating vs Kustomize

When using Helm for its templating engine, YAML files can become very ugly very soon, check out this example.

Using Kustomize it can be much easier to keep configuration clean, sure always depending on the use case.

The end.

Are you using Kustomize in production?

Become Kubernetes Certified

https://killer.sh