How to Confidently Not Break Server Startup
Introducing Tilt CI
Does this story sound familiar?
You commit a change. You push it to CI. All tests pass. You merge. The change deploys to staging.
Then your Ops team notifies you that all the staging servers are down. But don’t worry, they’ve already reverted your change!
But after years of failure, I’m optimistic.
Kubernetes is becoming a shared standard for running multi-service apps in prod. Lots of teams are getting Kubernetes into lots of different environments, including CI.
Does that mean we have the tools to test, in CI, that all the servers in our app can start?
To test our Kubernetes app in CI, we need two big pieces.
We need a new kind of build system that knows how to watch runtime and verify your server is healthy.
We need a way to spin up short-lived clusters from scratch.
We’re going to briefly walk you through each piece.
tilt ci Command
Tilt now has a new command:
tilt up is for an interactive multi-service development environment,
tilt ci is for testing that same environment in CI.
When you run
tilt ci, Tilt will:
- Read your Tiltfile
- Build and push all the container images
- Deploy the servers
- Wait until the servers are healthy
- And exit on success!
If it encounters any unrecoverable errors, Tilt will quit immediately.
If you want to customize how Tilt defines “healthy”, Kubernetes lets you define readiness probes that let you provide your own definition.
And when you need checks beyond what you can express in readiness probes, use a
local_resource with arbitrary
commands to run against your server.
Ephemeral CI clusters
You can run
tilt ci against any cluster, even a production one like AKS or
But for testing on CI, we recommend Kind.
When you start Kind with a local registry, you have an entirely self-contained cluster environment. No secret keys for a private container registry. No service accounts. No hosted inscrutable magic. Just you and your cluster.
For specific details on how to get a Kind cluster set up on your own CI environment, see:
See it in Action
We’re not just advocates for the local Kubernetes dev club. We’re also clients! 👨🦲
Every change in our repo creates a new Kind cluster and runs the latest build of Tilt against it.
We recently developed guides for servers written in:
All of them use
tilt ci with a Kind cluster to test that the servers start
up. With each Tilt release, we create a Tilt Docker
contains Tilt and Kind for use in CI. Check out the guides if you need a working