Local CI/CD with Skaffold

Published in

Level Up Coding

5 min readMar 13, 2020

The world of software development and how apps are run in production environments has come a long way over the years. Starting with bare metal physical servers, we evolved to virtual machines, onward to LXC, Docker daemons, and now our current state of container orchestration via things like Kubernetes.

The other side of the world… that which defines how software developers locally develop, test, iterate, package, build and deploy those apps to their final execution environments likewise has varied wildly. Much of this is due to obvious things like the choice of language and frameworks, but another factor in it is the final execution environment by which the application will live. As target runtime environments have evolved from bare-metal to containers, much of the complexity of configuration and “installing an application” has now been pushed down to the developer’s plate, as the developer is now responsible for defining the context by which the application will execute in using container images.

With this comes more responsibility for the developer of not only defining and documenting an app's dependencies, but now also implementing all of it via Dockerfiles; building those Dockerfiles into images, then pushing them to an artifact repository. The containerization standards over the past few years have certainly offloaded more DevOps-like work on the developer's plate but with that extra work comes a big benefit: Like never before, developers can now test their apps locally in much more realistic execution environments as they will run in production (i.e. local Minkube, Docker, k3s etc).

However, in order to be able to test the artifacts locally, they still need to be built and deployed (locally or remotely) to a container execution engine. Typically this can just be a centralized CI/CD service which handles all of these extra steps in reaction to a developer just pushing a commit; but what if a developer wants to do all of this in a more real-time fashion and avoid pushing/deploying artifacts to remote environments on every change over numerous iterations? i.e. just iterate locally.

Well, over the past few years several tools have evolved which bring powerful CI/CD capabilities right to the developer’s laptop, enabling them to harness the power of container automation using standard CI/CD tooling to build, package, test and deploy both remotely OR locally… even in real-time as local files are being changed.

Let’s take a brief look at one of these tools and please keep in mind that my coverage here is based primarily on my personal experience using it which was very specific to certain use-cases. This article is not an exhaustive overview of all the capabilities.

When I was prototyping various tooling for local automation I came across Skaffold, a Google project. Skaffold is “continuous development” tool for developers on their desktop to deploy to local or remote k8s clusters on demand or as watched files change locally as they iterate without the need to necessarily have everything pushed to Git origin.

Skaffold executes entirely locally and there is no server side component. The developer installs Skaffold locally, then does a “skaffold init” your project and then crafts a skaffold.yaml (yes! more YAML!). This serves as a mini pipeline definition that can be executed on demand via “ skaffold run” or the skaffold daemon (via “skaffold dev”) which can monitor the project contents and automatically reprocess your skaffold.yaml as changes occur. It can even automatically setup port-forwarding to the cluster (local or remote) that your app is provisioned to as well as a beta feature for actual debugging instrumentation (limited language support).

Skaffold at its heart is a mini CI/CD engine with various well defined stages, and each stage supports numerous plugins that work with common third party tooling to implement the actual building (Kaniko, Docker, Bazel, custom script etc), testing, manifest generation (Helm, Kustomize), deploying (vanilla kubectl, Helm) etc. Which “plugins” will be used for each stage, and their configuration options are all configured via YAML in the skaffold.yaml

*Skaffold’s high level architecture showing the different “stages” of local execution defined in a skaffold.yaml and the various ways they can be implemented:* *https://skaffold.dev/docs/design/*

Skaffold’s YAML schema has a ton of options all of which you can read here. Most of the values are interpreted as statically defined constants, with the exception of a few very limited areas where you can utilize Golang template syntax to reference contextual or runtime environment variables. My biggest problem with this is the lack of consistency with regards to the use of variables in values. Why isn’t this principle of variables in values consistently available across the entire document, or even be able to declare new variables not supported? There are also various bugs with it and a bit annoying. This is covered in various outstanding issues here and here.

Here is an example skaffold.yaml that only implements the build and deploy stages. The build stage invokes a custom script. Skaffold’s lack of hooks and consistent template variable interopolation can lead to annoying workarounds.

For the use case I was working on, the build needed to generate a unique Dockerfile prior to building. Well unfortunately Skaffold did not provide a “pre-build” hook where I could generate that file, but instead provided only the custom “build” stage command; which I subsequently overloaded to both generate and build myself…. rather than being able to leverage Skaffold’s “docker” native build stage capability. This was annoying. This worked decently … but really needs improved support for more custom hooks that can invoke custom commands before/after all stages, as well as a contract for exchanging data/variables with those hooks. Skaffold’s very inconsistent implementation of variable parsing within the skaffold.yaml is also a drawback. It would be great to be able to define your own setup of variables via another script, instead you might just have to write your own Skaffold wrapper script to setup your ENV vars the way you want them.

Overall, Skaffold is cool, but nothing magical. Its a nice self-contained CI/CD automation engine that can be run anywhere, but more specifically is targeted to run on your local desktop. With automatic file change detection and automated rebuilding of your image, testing and deploying to a k8s cluster, its pretty powerful. For most use-cases I think it would be a great fit for local development. You should definitely give Skaffold a look.

Some issues that if addressed would make Skaffold ever better!

Originally published at http://bitsofinfo.wordpress.com on March 13, 2020.

Local CI/CD with Skaffold

Written by bitsofinfo