Introducing kube-applier: Declarative Configuration for Kubernetes

At Box, we're always looking to improve our engineers' ability to deliver high-quality software as easily as possible. As part of our ongoing migration to microservices running on Kubernetes, we've built an array of custom tools and services to improve our deployment process. We're happy to announce that we are open-sourcing one of those projects: kube-applier, a service that automatically keeps your Kubernetes objects up-to-date with their declarative configuration files.

Bringing Declarative Configuration to Kubernetes

In a declarative model, you can check your configuration files into version control. This allows you to do code review for your config (to avoid breaking things) and provides an audit trail (if things end up breaking anyway). To help bring declarative configuration to Kubernetes, we contributed much of the code powering "kubectl apply."

With the apply command at your disposal, management of Kubernetes objects becomes much easier. You can maintain a set of config files (each containing one or more objects), and use the apply command to reconcile the actual objects on the cluster with the state described by the files.

However, the apply command alone isn't typically enough. Do we want our engineers to connect to each of our clusters and run "kubectl apply" every time they modify their config files? Version control will track changes to the config files, but how can we easily find info about when the changes were applied to the cluster?

Automating the Apply Process

kube-applier solves these problems for developers by abstracting away the apply process. It watches a remote Git repository for new commits then runs kubectl apply on each file against your Kubernetes cluster, making deployment as simple as committing your file changes.

Development teams at Box run an armada of services across multiple clusters and environments. However, we centralize all of the config files in a single repo, divided into subdirectories for each cluster. kube-applier lets you choose its "root" to be any directory within a Git repo, so we run a kube-applier instance on each cluster and point it at the correct subdirectory.

Each new commit triggers a "run" of successive apply commands for all config files in the targeted directory. kube-applier also performs periodic runs (every 5 minutes by default) even if there are no new commits, to ensure that the cluster remains synchronized with the contents of the repo.

The basic functionality isn't too technically daunting - the first version of kube-applier was just a shell script. (The official Kubernetes addon-manager is still just a shell script, so we're not too embarrassed.) However, we wanted to implement more robust monitoring features, and we eventually rebuilt kube-applier as a full-fledged Go app.

Who Watches the Watchmen?

kube-applier does extensive logging to maintain a history of apply runs. Information about the most recent apply run is served at a web status page, which our development teams visit to make sure their changes were applied correctly.

We use Prometheus to emit metrics that can be used for monitoring. For example, we've set up alerts that fire if it's been a while since the last successful apply run.

Summary

kube-applier is well-suited for any individual or organization that maintains a Git repo of config files for Kubernetes objects. At Box, we've found that it allows us to worry less about deployment and focus more on writing and maintaining code.

We're excited to continue building on Kubernetes, and we're hoping that developers everywhere can take advantage of kube-applier and help us continue improving it.

To learn more, check out the project repository on Github.

Greg Lyons worked on kube-applier as an intern from Duke University. He will start full-time at Box this summer.