Local Kubernetes Development With Pulumi

How to develop in containers without messing around with YAML

Kubernetes is almost 8 years old now. I like Kubernetes a lot! But one of the big surprises is that users are still messing around with YAML files. This has surprised a few other people in the community too:

As @bryanl says: YAML is for computers. When we started with YAML we never intended it to be the user facing solution. We saw it as "assembly code". I'm horrified that we are still interacting with it directly. That is a failure.

— Joe Beda (@jbeda) May 10, 2018

And Joe’s tweet above is from 4 years ago!

In the past, we’ve written about the ways you can use Helm or Kustomize. Both these tools are great for organizing YAML into packages and adapting them to multiple environments. Helm is usually the first thing that I personally reach for when my YAML is getting unwieldy. But they are the “buy a USB hub” solution to infrastructure.

Recently, I’ve been playing around with Pulumi as an alternative.

Pulumi generates real programming language APIs for infrastructure, so you can set up your infrastructure with code rather than with YAML. Then you can organize the common bits in the way that works best for your services.

It’s still declarative! It’s just defined in code.¹ And this lets Pulumi provide more of the building blocks of a nice, pleasant-to-use deploy system:

• A dashboard with a historical record of your deploys.

• CI/CD integration so you can link deploys to particular CI runs.

• Conflict checks that warn you if two people are trying to deploy at the same time.

This blog post is a quick guide to how to set up Pulumi for local development, so that you can iterate quickly without messing around with YAML.

What Converting to Pulumi Looks Like

This is the Tilt blog. So this is eventually going to lead up to the Tilt Pulumi extension.

But first let’s look at an app that uses Pulumi to define Deployments.

Here’s what a normal Deployment looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld
spec:
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: helloworld-image

Pulumi’s APIs try to closely mirror what this YAML looks like, providing exactly the same arguments. Here’s an example that uses Javascript (though Pulumi also has APIs for Python, Typescript, and Go):

"use strict";
const k8s = require("@pulumi/kubernetes");

const deployment = new k8s.apps.v1.Deployment("pulumi-helloworld", {
  spec: {
    selector: { 
      matchLabels: { app: "pulumi-helloworld" } 
    },
    replicas: 1,
    template: {
      metadata: { labels: { app: "pulumi-helloworld" } },
      spec: { 
        containers: [{ 
          name: "pulumi-helloworld", 
          image: "pulumi-helloworld-image" 
        }] 
      }
    }
  }
});
exports.name = deployment.metadata.name;

And because this is a real programming language, it’s easy to factor out the common constants.

"use strict";
const k8s = require("@pulumi/kubernetes");

const appLabels = { app: "pulumi-helloworld" };
const deployment = new k8s.apps.v1.Deployment("pulumi-helloworld", {
  spec: {
    selector: { 
      matchLabels: appLabels 
    },
    replicas: 1,
    template: {
      metadata: { labels: appLabels },
      spec: { 
        containers: [{ 
          name: "pulumi-helloworld", 
          image: "pulumi-helloworld-image" 
        }] 
      }
    }
  }
});
exports.name = deployment.metadata.name;

Setting up a Fast Local Dev Loop

I’ve built a lot of developer tools! And I started to notice how every tool would gradually expand in scope to add the same set of features to make iterating on them easy:

• File watches and dependency tracking for auto-builds.

• Terminal & UI dashboards for real-time status.

• Diagnostic CLIs for inspecting when things went wrong.

We built Tilt to be a dev environment that integrates with any build/deploy tool. That’s why Tilt has a pluggable system for deploying to Kubernetes, such that we can layer on our own image builds and live updates for local development.

Let’s take a look at how it works!

First, we need a way to pass a Tilt-built image to our Pulumi script. We use the pulumi.Config API to read in an image from an outside config:

"use strict";
const pulumi = require("@pulumi/pulumi");
const k8s = require("@pulumi/kubernetes");

let config = new pulumi.Config();
let image = config.require("image");

const appLabels = { app: "pulumi-helloworld" };
const deployment = new k8s.apps.v1.Deployment("pulumi-helloworld", {
    spec: {
        selector: { matchLabels: appLabels },
        replicas: 1,
        template: {
            metadata: { labels: appLabels },
            spec: { containers: [{ name: "pulumi-helloworld", image: image }] }
        }
    }
});
exports.name = deployment.metadata.name;

Now, we use Tilt to build the image, live update the running Pod, and connect a port-forward from localhost:8000 to the container deployed with Pulumi.

Tilt’s pulumi extension invokes pulumi up to deploy to the cluster.

load('ext://pulumi', 'pulumi_resource')

docker_build(
  'pulumi-helloworld-image',
  './helloworld',
  live_update=[
    sync('./helloworld', '/app'),
  ])

pulumi_resource(
  'helloworld',
  stack='dev',
  dir='./',
  deps=['./index.js'],
  image_deps=['pulumi-helloworld-image'],
  image_configs=['image'],
  labels=['helloworld'],
  port_forwards=['8000:8000'])

The pulumi_resource function has arguments to define:

• The name of your resource in the Tilt UI.

• The name of the stack in Pulumi (this is usually implicit when you use the Pulumi CLI).

• Which Pulumi files the deploy depends on. (If either the docker_build or pulumi_resource dependencies change, tilt will re-deploy.)

• How to inject the image into your script.

• Any other live updates or port forwards to attach!

A Pulumi Tiltfile is a good way to set up a fast feedback loop so you can make small changes – a little bit at a time – and verify that they work.

To try this example yourself, check out the Tilt Pulumi extension docs or the accompanying example.

But how do we make sure it all works?

Verifying Your Pulumi Scripts in CI

The Pulumi engine is built so that you can write unit tests against a “fake” backend. The Pulumi docs have more detail on how to use Pulumi’s testing libraries.

But we can also use Tilt’s blackbox testing to test our Pulumi script against a real, one-time-use cluster with tilt ci. This is how the Tilt team tests our own Pulumi extension!

#!/bin/bash

cd "$(dirname "$0")"
set -ex

# install pulumi deps
yarn install

tilt ci

The tilt ci command:

• Builds all the images in the tiltfile.

• Runs pulumi up to deploy the images to Kubernetes.

• Tracks the rollout of pods.

• Exits when all the pods become ready.

For more on how to set up a one-time-use cluster for testing your infrastructure, see the Tilt CI Guide.

Future Work

We hope this guide helped you to understand how Pulumi might fit into your infrastucture stack, and how to use it for local Kubernetes development.

The Pulumi extension uses a much more full-featured plugin API that lets Tilt use any arbitrary Bash script for Kubernetes deployments. To learn more how to adapt Tilt’s dev environment to other similar tools, check out Milas’ post on k8s_custom_deploy. And if it’s a widely used tool, you can share it with other Tilt users by submitting it to the extensions repo.

1. Tilt has a similar philosophy to Pulumi. Tiltfiles are dev environments as code. The language may be imperative, but we’re using it to define our env declaratively. ↩