Cutting Kubernetes From Dev Isn't Enough

Replace start.sh for Great Usability

Using Kubernetes in development is painful. Many teams find ways to cut Kubernetes out of their development inner loop so they can skip slow image builds and the conceptual overhead of containers/clusters. It’s easy to write a custom script that starts each server in your app, perhaps using foreman, docker-compose or vagrant. Teams should prioritize productivity over purity.

Cutting Kubernetes out of dev removes overhead, but it isn’t enough. Kubernetes was the source of many, but not all, problems. What’s left are problems that are inherent to multi-service development:

• What should you rebuild/restart after you’ve edited files? Just as manual/error-prone/ad hoc running bare processes on your laptop.
• Finding a problem across many services? Playing 20 questions is as clumsy with tail -f or ps as kubectl.
• Onboarding new users to this clusterf…un? You want them to be productive on day 1 without having to learn the whole stack, regardless of where it’s running.

We’re expanding Tilt to handle multi-service development outside Kubernetes. Tilt can add automation, usability and shareability to your dev workflow, no migration to Kubernetes required. This post describes how you can use local_resource and serve_cmd to see your existing setup in Tilt with an hour of experimenting.

If you have Kubernetes in Prod, you need Tilt for Dev. Even if you don’t use Kubernetes in Dev.

What You Get

After writing a Tiltfile, you’ll have a better local development experience:

• Start your whole app just by running tilt up
• Update microservices automatically as you edit
• Web UI that lets you see problems across services and zoom in on issues
• Easily run workflows like resetting database state

From this UI, you can see:

• The list of servers and workflows (“Resources”) in the sidebar on the right
• All logs, multiplexed together
• The status of each Resource (e.g. if a server died, its status will be red instead of green)
• Logs for just one resource, by clicking on it in the sidebar

local_resource with serve_cmd

Let’s look at a sample Tiltfile for a project (simplified from our sample project abc123) that uses Tilt for two different cases:

• a Go frontend that requires a build on each change
• a Python backend (“numbers”) that watches files and updates itself, except for a change to requirements.txt.

local_resource(
  'fe',
  'go install github.com/windmilleng/abc123/fe',
  serve_cmd='cd fe && fe --port=8000 --backend=localhost:8002',
  deps=['fe'])

local_resource(
  'numbers',
  'cd numbers && pip install -r requirements.txt',
  serve_cmd='cd numbers && python app.py --port=%d' % (numbers_port),
  # numbers is a flask app that handles most changes via hot reloading, but
  # not requirements.txt changes
  deps=['numbers/requirements.txt'])

This example project has clean separation between services. Tilt can handle more complex setups using options to local_resource that you can learn about in the API reference.

Check Environments With local_resource

If you’re not using containers for development, you lose the benefit of consistency a Dockerfile provides. You can get some of it back with a local_resource that checks local software installations. Tilt will highlight incompatibilities before you or a teammate spend hours debugging a build failure. Write a script called check_compat.sh that does the checks and add this to your Tiltfile:

local_resource('check_compat', 'check_compat.sh')

Tilt will run this on startup and display an error if the command exits with an error.

Try Tilt With Your Project

Download Tilt and start it. When you change the Tiltfile, Tilt will automatically pick up the changes. If you want to pair program with a member of the Tilt team to configure Tilt for your local dev, you can schedule a session or check out our contact info.

Where You Can Go Next

If you deploy to Kubernetes for production, you have bugs that only show up in Kubernetes. That kind of development, like modifying YAML configuration or fixing networking-specific bugs requires a Kubernetes development workflow. Your team has this workflow as well, but it’s probably less understood and much slower.

Fixing those pains is Tilt’s sweet spot. Live Update can sync and build into running containers as quickly as running processes locally. Tilt’s UI natively includes Kubernetes events and statuses.

Using Tilt for both of these workflows is easy: devs can switch a command-line flag and the same UI that was displaying status of uncontained processes on your laptop is suddenly connected to pods in a dev Kubernetes cluster.

Conclusion

Moving your non-Kubernetes local dev workflow into Tilt can free your team from maintaining custom tooling, manually updating microservices or hunting for error messages in logs. You can see Tilt in action on your project quickly by writing a Titlfile based on your existing setup. Your team can have a more consistent dev experience this sprint, on a foundation that you can improve over time.