Deploy All The Things Even If They Aren't YAML

If you’ve used Tilt, chances are you are well acquainted with the k8s_yaml Tiltfile function, which deploys objects to Kubernetes from either a file or string. (Think kubectl apply -f ...)

Most of the time, this is all you need - after all, YAML is the go-to choice for Kubernetes tooling. Many tools, such as kustomize, always output YAML, while others, like Helm (which manages the entire deployment lifecycle), offer a template subcommand.

In some instances, however, pre-templated YAML alone is not sufficient or might not even exist! For example, Helm charts can include “hooks” to run at a specific part of the deployment lifecycle but aren’t part of the templated YAML. Other tools, such as OpenFaaS, provide a streamlined build & deploy experience managed entirely by their CLI, so there’s no YAML at all. It’s also common for teams to have their own custom deployment shell scripts (e.g. for use with CI/CD).

While it’s possible to use Tilt’s local_resource function as a workaround in these instances, that means giving up much of Tilt’s deep Kubernetes integration, such as Pod log streaming and port forwards, for the resource as a result.

In Tilt v0.23.0, we’ve introduced a built-in function to enable the use of an external Kubernetes deployment tool or script. With the new k8s_custom_deploy function, Tilt will delegate to a command or shell script to perform the deployment instead of directly calling the Kubernetes API with YAML. Tilt features like status reporting and log streaming work automatically. Even the more advanced customizations provided by k8s_resource such as port forwards are supported! This is possible as a result of lots of under the hood changes to Tilt over the past months to expose individual Tilt features via composable APIs.

Example

Let’s use k8s_custom_deploy to deploy an OpenFaaS function without writing a Dockerfile or Kubernetes YAML.

For this example, I’ve installed the OpenFaaS components in my cluster manually; in a real Tiltfile, you could include this as well to ensure the local environment is set up properly. The OpenFaaS blog has a guide that might be useful.

I’m using a Golang handler bootstrapped by running faas-cli new go-fn --lang go. This created a go-fn.yml file (this is an OpenFaaS config, NOT a Kubernetes manifest.) It also made a go-fn/ directory with “Hello world” example handler code.

# on `tilt up` and file changes -> run the OpenFaaS build + deploy and then query for what it deployed
apply_cmd = """
faas-cli up -f go-fn.yml 1>&2
kubectl get -oyaml --namespace=openfaas-fn all -l "faas_function=go-fn"
"""

# on `tilt down` -> delete the OpenFaaS function from the cluster
delete_cmd = 'faas-cli delete -f go-fn.yml'

k8s_custom_deploy(
    'go-fn',
    apply_cmd=apply_cmd,
    delete_cmd=delete_cmd,
    # apply_cmd will be re-executed whenever these files change
    deps=['./go-fn.yml', './go-fn/']
)

# add a port forward so we can debug the function from our host machine directly
k8s_resource('go-fn', port_forwards=['9372:8080'])

When we run tilt up, Tilt will execute the apply_cmd, which in our case invokes faas-cli up to build and deploy an image and then returns the result as YAML so Tilt can track the new or updated Kubernetes objects. If go-fn.yml or any file in the go-fn/ directory tree changes, the apply_cmd will be automatically re-run.

On tilt down, our delete_cmd will be invoked, which allows faas-cli to delete any objects from Kubernetes it created as well as clean up any extra OpenFaaS state.

What if Something Goes Wrong?

Integrating your workflow with an external tool can be tricky! Luckily, the Tilt API allows us to quickly introspect what’s happening behind the scenes.

If something goes wrong, the error field in the KubernetesApply object status will have more information:

$ tilt get -ojsonpath='{.status.error}' kapp go-fn | head -n 5
apply command returned malformed YAML: error converting YAML to JSON: yaml: control characters are not allowed
stdout:
[0] > Building go-fn.
Clearing temporary build folder: ./build/go-fn/
Preparing: ./go-fn/ build/go-fn/function

Or, we can see which objects were deployed:

$ tilt get -ojsonpath='{.status.resultYAML}' kubernetesapply go-fn | yq '.kind + "/" + .metadata.name'
"Pod/go-fn-fd78fc4d8-d558b"
"Deployment/go-fn"
"ReplicaSet/go-fn-fd78fc4d8"

💡 Use tilt describe kapp to get a human readable version!

Important information will always be shown in the Tilt web UI, but the API lets you see the exact same data that Tilt is using internally. We’ve begun to rely on it ourselves for debugging heavily and hope you find it as useful as we do!

What’s the Catch?

At the moment, it is not practical to use images built with Tilt (e.g. via docker_build) in conjunction with k8s_custom_deploy. We are still exploring the best semantics for passing Tilt-built image references to external tools. In its initial state, k8s_custom_deploy is best suited for use with pre-built images or tools that handle both image build + deploy.

Additionally, if your tool is capable of templating YAML, and you don’t need other functionality provided by it while developing, k8s_yaml is often simpler and faster.

Looking for a More Complete Example?

The knative Tilt extension uses k8s_custom_deploy to deploy and monitor the Knative serving components. Be sure to check it out if you’re looking to use Knative serving with Tilt or a real world example.