Big News!  Tilt is joining Docker

How to Compile Protobufs When You Have Friends Who Want to Use Them Too

I love GRPC, but hate compiling protobufs.

Protobufs make me feel productive! I can write a small .proto file that describes my interface. The protobuf compiler will generate all the client/server code that I need.

When I want to introduce protobufs into a codebase, I start with a Make rule:

  protoc --go_out=plugins=grpc:../../../ -I. proto/*.proto

Whenever I change the proto file, I run make proto and commit the generated code into git.

This works when you have no friends! When I do bring friends onto my project, that’s when the complaining begins:

  • How do I install protoc?

  • Ugh, it doesn’t work, I also need to install the Go protoc plugin!

  • Wait, why did all the generated code change when I ran it on my machine?

  • Oops, it’s because I installed the wrong version of protoc! Do we all need to be on the same version?

  • Agh, the Go protoc plugin instructions only tell me how to install HEAD, how do I install a fixed version?

  • Why are we on such on old version? How do I upgrade without breaking everyone?

It adds work to each new team member onboarding. And coordinating upgrades is A Big Event.

How do we compile protobufs consistently?

One way to fix this is to switch to a hermetic build system with protobuf support, like Bazel. But you’d have to rewrite your build system, which is a drag.

Instead, we wrote a little shell script to compile the protobufs inside a Docker container. The Dockerfile looked something like this:

# Dockerfile.protogen
FROM golang:1.11


RUN something something install protoc
RUN something something install protoc-gen-go

WORKDIR /go/src/

COPY my.proto my.proto
RUN protoc --go_out=plugins=grpc:../../../ -I. my.proto

The Makefile rule turned into this:

  docker build -t protogen -f Dockerfile.protogen
  docker run --name protogen protogen
  docker cp protogen:/go/src/ ./
  docker rm protogen

That worked fiiiine…for a while. It was finicky when adding new .proto files. It wasn’t easy to share across projects.

Making this pattern reusable

We wrote a small python script to help us apply this pattern in multiple projects.

We thought about calling it GRIND (“GRpc IN Docker”). But that would be silly. Instead we call it protocc.

windmilleng/protocc - Compile protobufs (protoc) inside a container (protocc)!

Just copy and run:

python --out go

to run protocc inside a container in your own repo. No need to install protoc or manage protoc plugins yourself!

It currently only generates Go code. If this sounds like something you want for your target language, let us know and we’d be happy to add it!

Originally posted on the Windmill Engineering blog on Medium


Already have a Dockerfile and a Kubernetes config?

You’ll be able to setup Tilt in no time and start getting things done. Check out the docs! 

Having trouble developing your servers in Kubernetes?