How to Compile Protobufs When You Have Friends Who Want to Use Them Too
I love GRPC, but hate compiling protobufs.
Protobufs make me feel productive! I can write a small
.proto file that describes my interface. The protobuf compiler will generate all the client/server code that I need.
When I want to introduce protobufs into a codebase, I start with a Make rule:
proto: protoc --go_out=plugins=grpc:../../../ -I. proto/*.proto
Whenever I change the proto file, I run
make proto and commit the generated code into git.
This works when you have no friends! When I do bring friends onto my project, that’s when the complaining begins:
How do I install
Ugh, it doesn’t work, I also need to install the Go
Wait, why did all the generated code change when I ran it on my machine?
Oops, it’s because I installed the wrong version of
protoc! Do we all need to be on the same version?
Agh, the Go
protocplugin instructions only tell me how to install HEAD, how do I install a fixed version?
Why are we on such on old version? How do I upgrade without breaking everyone?
It adds work to each new team member onboarding. And coordinating upgrades is A Big Event.
How do we compile protobufs consistently?
One way to fix this is to switch to a hermetic build system with protobuf support, like Bazel. But you’d have to rewrite your build system, which is a drag.
Instead, we wrote a little shell script to compile the protobufs inside a Docker container. The Dockerfile looked something like this:
# Dockerfile.protogen FROM golang:1.11 ENV PROTOC_VERSION 3.6.1 ENV PROTOC_GEN_GO_VERSION v1.2.0 RUN something something install protoc RUN something something install protoc-gen-go WORKDIR /go/src/github.com/windmilleng/tilt COPY my.proto my.proto RUN protoc --go_out=plugins=grpc:../../../ -I. my.proto
The Makefile rule turned into this:
proto: docker build -t protogen -f Dockerfile.protogen docker run --name protogen protogen docker cp protogen:/go/src/github.com/windmilleng/tilt/my.pb.go ./ docker rm protogen
That worked fiiiine…for a while. It was finicky when adding new
.proto files. It wasn’t easy to share across projects.
Making this pattern reusable
We wrote a small python script to help us apply this pattern in multiple projects.
We thought about calling it GRIND (“GRpc IN Docker”). But that would be silly. Instead we call it
windmilleng/protocc - Compile protobufs (protoc) inside a container (protocc)!
protocc.py and run:
python protocc.py --out go
protocc inside a container in your own repo. No need to install
protoc or manage
protoc plugins yourself!
It currently only generates Go code. If this sounds like something you want for your target language, let us know and we’d be happy to add it!
Originally posted on the Windmill Engineering blog on Medium