Stories by Nick Santos on Medium

Stories by Nick Santos on Medium

Go Rant: Buffered Channels Should be Tossed in a Fire

Empirical and Theoretical Arguments Why This Feature Sucks

Channels are Go’s fundamental tool for concurrent programming.

Buffered channels are a small, innocent-looking feature on top of channels. I’m going to try to convince you that they’re a monstrous abomination.

Quick Recap on What I’m Ranting About

First, a quick recap of how channels work if you haven’t used Go in a while.

A channel lets you send data on one goroutine, and receive it on another, concurrent goroutine.

In a normal unbuffered channel, sending data will block until the data is received.

This blocks forever:

ch := make(chan int, 0)
ch <- 1

This prints “Received 1\nPossible”

ch := make(chan int, 0)
go func() {
fmt.Printf("Received: %d\n", <-ch)
ch <- 1

But you can make buffered channels of arbitrary size N. A buffer of size N will block if you send more than N times before receiving.

This prints “Possible” then deadlocks.

ch := make(chan int, 1)
ch <- 1
ch <- 1

My Assertion

There are only 3 reasonable values of N:

  • 0 — the channel always blocks.
  • 1 — the channel lets you push one element at a time.
  • Infinite — the channel lets you keep pushing elements until you run out of memory.

All other values of N will inevitably lead to bugs.


There are two ways to make this case: the empirical argument (why real projects blow up when they use buffered channels) and the theoretical one (why even hypothetical imagined projects will blow up when they use buffered channels).

From the Empirical Argument

Every opensource project I’ve ever seen that used an arbitrary-N buffered channel was using that channel incorrectly. And ended up having bugs.

Someone would inevitably send a list of elements to that channel. But the problem is that you can only do this safely if:

  • You can guarantee that the list is less than N.
  • You can guarantee that no one else is also sending on the channel.
  • OR you can guarantee that another goroutine is available to receive on the channel.

These are often hard guarantees to make! They’re often about other actors that you don’t control (particularly if you got your list of elements somewhere else.) And they’re guarantees that the programming language doesn’t help you enforce at all.

That hints at:

From the Theoretical Argument

“I can only call this function N times synchronously” is an abnormal API constraint. And it’s not a constraint that programming languages do anything to help you enforce.

Programming languages have lots of constraints they do help you to enforce!

  • They help you ensure that a function isn’t called (think: private functions).
  • They help you ensure that a function is called exactly once (think: constructors, sync.Once).
  • They help you ensure that a function is called only once at a time (think: locks/mutexes and all the tooling around them).

Now, I don’t want to harsh your buzz for programming language constraint checks. We live in 2021! Rust’s borrow checker can do compile-time checks that I never would have imagined!

But let’s be honest: the Go-team does not have the appetite for the kind of programming language features that would help you use buffered channels well.


I liked this proposal for Go2 for an unlimited-capacity buffered channel.

proposal: spec: add support for unlimited capacity channels · Issue #20352 · golang/go

But I also liked Ian Lance Taylor’s response!

If we had generic types, unlimited channels could be implemented in a library with full type safety. A library would also make it possible to improve the implementation easily over time as we learn more.

So I’m hoping for a one-two punch:

  1. Go gets generics!
  2. We can add a channel wrapper that makes more sense!!
  3. They can remove buffered channels from the core language!!!
  4. Profit!!!!

Three Ways to Run Kubernetes on CI and Which One is Right for You!
A tarot deck, suitable for choosing Kubernetes topologies in CI. Via Wikipedia.

When we first started developing Tilt, we broke ALL THE TIME.

Either Kubernetes changed. Or we had a subtle misunderstanding in how the API works. Our changes would pass unit tests, but fail with a real Kubernetes cluster.

I built out an integration test suite that used the latest version of Tilt to deploy real sample projects against a real cluster.

At the start, it was slow and flakey. But the tooling around running Kubernetes in CI has come a long way, especially in the last 1–2 years. Now it’s less flakey than our normal unit tests 😬. Every new example repo we set up uses a one-time Kubernetes cluster to run tests against.

A few of our friends have been asking us how we set it up and how to run their own clusters in CI. I’ve now explained it enough times that I should probably write down what we learned.

Here are three ways to set it up, with the pros and cons of each!

Strategy #1: Local Cluster, Remote Registry

Here’s how I set up our first integration test framework.

  1. I created a dedicated bucket for us to store images, and a GCP service account with permission to write to it.
  2. I added the GCP service account credentials as a secret in our CI build.
  3. I forked kubeadm-dind-cluster, a set of Bash scripts to set up Kubernetes with Docker-in-Docker techniques.

All our test projects had Tilt build images, push them to the bucket, then deploy servers that used these images.

I barely got this working. A huge breakthrough! It caught so many subtle bugs and race conditions.

I wouldn’t call the Bash scripts readable. But they are hackable, cut-and-pasteable. There were examples of how to run it on CircleCI and TravisCI. kubeadm-dind-cluster has been deprecated in favor of more modern approaches like . But I learned a lot from its Bash scripts. We still use a lot of the techniques in this project today.

There were other downsides though:

  • When drive-by contributors sent us PRs, the integration tests failed. They didn’t have access to to write to the bucket. This made me so sad. Contributors felt unwelcome. I never figured out a way to make this secure.
  • We didn’t reset the bucket between test runs. So it was hard to guarantee that images weren’t leaking between tests. For example, if image pushing failed, we wanted to be sure we weren’t picking up a cached image from a previous test.

Strategy #2: Local Registry On a VM

When I revisited this, I wanted to make sure:

By this time, kind was taking off as the default choice for testing Kubernetes itself. kind also comes with the ability to run a local registry, so you can push images to the registry on localhost:5000 and pull them from inside kind.

I set up a new CI pipeline that:

  1. Creates a VM.
  2. Installs all our dependencies, including Docker.
  3. Creates a kind cluster with a local registry, using their script.

This worked well! And because the registry was local, it was faster than pushing to a remote registry. We still use this approach to test ctlptl with both minikube and kind. Here's the CI config.

But I wasn’t totally happy! Most of our team is more comfortable managing containers than managing VMs. VMs are slower. Upgrading dependencies is more heavyweight. We wondered: can we make this work in containers?

Strategy #3: Local Registry On Remote Docker

The last approach (and the one we use in most of our projects) uses some of the tricks that kubeadm-dind-cluster uses.

The CI pipeline:

  1. Creates a container with our code.
  2. Sets up a remote Docker environment outside the container. (This avoids the pitfalls of running Docker inside Docker.)
  3. Starts a kind cluster with a local registry inside the remote Docker environment.
  4. Uses socat networking jujitsu to expose the remote registry and Kubernetes cluster inside the local container.

The socat element makes this a bit tricky. But if you want to fork and hack it, check out this Bash script.

But once it’s set up: it’s fast, robust, and easy to upgrade dependencies.

Putting it Together

Hacking together this with Bash was the hard part.

Tilt-team maintains , a CLI for declaratively setting up local Kubernetes clusters.

I eventually folded all the logic in the Bash script into ctlptl. As of ctlptl 0.5.0, it will try to detect when you have a remote docker environment and set up the socat forwarding.

The Go code in ctlptl is far more verbose than the Bash script, comparing number of lines. But it includes error handling, cleanup logic, and idempotency, which makes it more suitable for local dev. (CI environments don't need any of this because we tear them down at the end anyway.)

We use image-management tools that auto-detect the registry location from the cluster, which helps with the configuration burden. I like the general trend of Kubernetes as a general-purpose config-sharing system so that tools can interoperate, rather than having to configure each tool individually.

We currently use ctlptl to set up clusters and test the services on real Kube clusters in all of our example projects.

It’s been a long journey! But I hope the examples here will make that journey a lot shorter for the next person 🙈.


Originally published at on April 2, 2021.

Three Ways to Run Kubernetes on CI and Which One is Right for You! was originally published in Tilt Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Kubernetes is so Simple You Can Explore it with Curl
“Curling; — a Scottish Game, at Central Park” by John George Brown. Via Wikipedia.

A common take on Kubernetes is that it’s very complicated.

… and because it’s complicated, the configuration is very verbose.

… and because there’s so much config YAML, we need big toolchains just to handle that config.

I want to convince you that the arrow of blame points in the opposite direction!

Kubernetes has a simple, genius idea about how to manage configuration.

Because it’s straightforward and consistent, we can manage more config than we ever could before! And now that we can manage oodles more config, we can build overcomplicated systems. Hooray!

The configs themselves may be complicated. So in this post, I’m going to skip the configs. I’ll focus purely on the API machinery and how to explore that API.

Building APIs this way could benefit a lot of tools.

What is the Idea?

To explain the simple, genius idea, let’s start with the simple, genius idea of Unix:

Everything is a file.

Or to be more precise, everything is a text stream. Unix programs read and write text streams. The filesystem is an API for finding text streams to read. Not all of these text streams are files!

  • ~/hello-world.txt is a text file
  • /dev/null is an empty text stream
  • /proc is a set of text streams for reading about processes

Let’s take a closer look at /proc. Here's a Julia Evans comic about it.

You can learn about what’s running on your system by looking at /proc, like:

  • How many processes are running (ls /proc - List the processes)
  • What command line started process PID (cat /proc/PID/cmdline - Get the process specification)
  • How much memory process PID is using (cat /proc/PID/status - Get the process status)

What is the Kubernetes API?

The Kubernetes API is /proc for distributed systems.

Everything is a resource over HTTP. We can explore every Kubernetes resource with a few HTTP GET commands.

To follow along, you’ll need:

  • - or any small, throwaway Kubernetes cluster
  • curl - or any CLI tool for sending HTTP requests
  • jq - or any CLI tool for exploring JSON
  • kubectl - to help curl authenticate

Let’s start by creating a cluster:

$ kind create cluster
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.19.1) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

$ kubectl proxy &
Starting to serve on

kubectl proxy is a server that handles certificates for us, so that we don't need to worry about auth tokens with curl.

The Kubernetes API has more hierarchy than /proc. It's split into folders by version and namespace and resource type. The API path format looks like:


On a fresh kind cluster, there should be some pods already running in the kube-system namespace we can look at. Let's list all the system processes in our cluster:

$ curl -s http://localhost:8001/api/v1/namespaces/kube-system/pods | head -n 20
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/kube-system/pods",
"resourceVersion": "1233"
"items": [
"metadata": {
"name": "coredns-f9fd979d6-5zxtx",
"generateName": "coredns-f9fd979d6-",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/pods/coredns-f9fd979d6-5zxtx",
"uid": "a30e70cc-2b53-4511-a5de-57c80e5b68ad",
"resourceVersion": "549",
"creationTimestamp": "2021-03-04T15:51:21Z",
"labels": {
"k8s-app": "kube-dns",
"pod-template-hash": "f9fd979d6"

That’s a lot of text! We can use jq to pull out the names of objects.

$ curl -s http://localhost:8001/api/v1/namespaces/kube-system/pods | jq '.items[]'

The /pods endpoint lists out all the processes, like ls /proc. If we want to look at a particular process, we can query/pods/POD_NAME.

$ curl -s http://localhost:8001/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane | head -n 10
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "kube-apiserver-kind-control-plane",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane",
"uid": "a8f893b7-1cdb-48fd-9505-87d71c81adcb",
"resourceVersion": "458",
"creationTimestamp": "2021-03-04T15:51:17Z",

Or, again, we can use jq to fetch a particular field.

$ curl -s http://localhost:8001/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane | jq '.status.phase'

How to unpack what kubectl is doing

All of the things above can be done with kubectl. kubectl provides a more friendly interface. But if you're ever wondering what APIs kubectl is calling, you can run it with-v 6:

$ kubectl get -v 6 -n kube-system pods kube-apiserver-kind-control-plane
I0304 12:47:59.687088 3573879 loader.go:375] Config loaded from file: /home/nick/.kube/config
I0304 12:47:59.697325 3573879 round_trippers.go:443] GET 200 OK in 5 milliseconds
kube-apiserver-kind-control-plane 1/1 Running 0 116m

For more advanced debugging, use -v 8 to see the complete response body.

The point isn’t that you should throw away kubectl in favor of curl to interact with Kubernetes. Just like you shouldn't throw away ps in favor of ls /proc.

But I’ve found disecting Kubernetes like this is helpful to think of it as a process-management system built on a couple straightforward principles:

  • Everything is a resource over HTTP.
  • Every object is read and written the same way.
  • All object state is readable.

These are powerful ideas. They help us build tools that fit together well.

In the same way that we can pipe Unix tools together (like jq), we can define new Kubernetes objects and combine them with existing ones.

Sometimes they’re silly! Like in this Ellen Körbes talk on how to build a Useless Machine.

In future posts, I want to talk about how to write code that uses these APIs effectively. And how we’re leaning into these ideas in Tilt. Stay tuned!

Originally published at on March 18, 2021.

Kubernetes is so Simple You Can Explore it with Curl was originally published in Tilt Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Reading List 2020

My reading productivity took a big hit year, because

  1. The New York Public Library closed until late summer
  2. My daily reading routine revolved around the daily subway commute, which was no more
  3. The cats kept attacking my books

But I still managed to finish some great reads!

7 Books I Particularly Loved And Why


Perspective Check

(The joint authors throw the totals off)

By Only Men: 20 / 38
By Only Women: 15 / 38

By Only White People: 26 / 38
By Only Non-White People: 10 / 38

Fiction: 17 / 38
Non-Fiction: 21 / 38

Recent (2019 or 2020): 21 / 38
Earlier This Decade (2011–2018): 6 / 38
Classics (pre-2011): 11 / 38

Happy 2021!

The First Modern Mention of Kubernetes

And Why it Still Matters

I found the first mention of Kubernetes in computer science!!

It comes from a book. “Cybernetics: or Control and Communication in the Animal and Machine” by Norbert Wiener. Originally published in 1948. (Yes, even in 1948, non-fiction book titles abused the colon.)

The book has its own Wikipedia page. So many people read it that he published a sequel!

It’s surprisingly hard to find. The New York Public Library has two copies offsite, only available on advanced request. The Brooklyn Public Library has zero copies.

I like to imagine I have the only physical copy in pandemic-lockdown New York City! Because right before the lockdown, I borrowed the 1961 edition from a science historian. And she hasn’t asked for it back yet 😬.

In the book, Wiener tries to come up with a name for his new field. He writes: “All the existing teminology has too heavy a bias […] we have been forced to coin at least one artificial neo-Greek expression to fill the gap.”

He suggests “Cybernetics,” from the Greek word χυβερνήτης, also pronounced “Kubernetes”:

Why Cybernetics?

You may think that this is just a pointy-headed in-joke to make the rest of us feel dumb.

But it’s actually a very relevant pointy-headed in-joke that only incidentally makes the rest of us feel dumb!

When Wiener published “Cybernetics,” the dominant model of computing was finite automata and Turing machines. These are systems that take inputs at the start and produce outputs at the end.

Wiener points out that there are two problems with this model:

1) In any system with lots of closely coupled inputs, we need statistical models to handle the complexity. In chapter one, he compares astronomy versus meteorology. We can count how many stars there are, and can capture the interaction between stars with simple formulas. But in meteorology, there are simply too many particles and the interactions between them are too complex.

2) Information is distributed over time. We can use the time component to learn which inputs cause which outputs, and build feedback loops.

Wiener argues that if computer science does a better job embracing complex statistical systems and time-based feedback loops, we’ll be able to better understand lots of non-mechanical systems, like biology and sociology.

The book gets very galaxy-brained to be honest, from “How do we build better thermostats?” to “Could this help us find a cure for Parkinson’s disease?”

What Does That Mean For How We Build Tilt?

Once you understand this parallel, you see it everywhere!

Consider Kubernetes. Before Kubernetes, you may have had a deploy script. That deploy script worked like a finite automata: look at some inputs, then deploy a server.

One of the key insights of Kubernetes is that when you’re working with multiple servers with close coupling, this isn’t enough. You need a system with runtime feedback loops to handle the runtime dependencies between servers.

This is, after all, the Tilt blog, and we think this insight applies to developer tools as well. From Make in the 70s to Bazel today, build systems are still stuck in a world of finite automata, mapping inputs to outputs.

But the servers we’re building have closely coupled runtime dependencies! Multi-service developer tools need runtime feedback loops as well.

Originally published at on April 28, 2020.

The First Modern Mention of Kubernetes was originally published in Tilt Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Books I Read in 2019

Here are all the covers (thanks Goodreads!)

10 Books I Particularly Loved And Why

  • “Big Little Lies” by Liane Moriarty
    Why do we even need horror stories? Parenting is already a state of continuous, low-level dread.
  • “The Calculating Stars” by Mary Robinette Kowal
    Misogyny in the space program is so entrenched that even destroying the whole East Coast of the US with an asteroid can’t fix it.
  • “Change is the Only Constant” by Ben Orlin
    Calculus is so much fun when you don’t need to prove anything.
  • “Say Nothing” by Patrick Radden Keefe
    Twenty-somethings can be so self-righteous it’s terrifying.
  • “The New Jim Crow” by Michelle Alexander
    The history of civil rights makes me appreciate how things can change so much and so little at the same time.
  • “Record of a Spaceborn Few” by Becky Chambers
    US immigration policy is a mess, but maybe there’s still hope for interstellar immigration policy.
  • “Mind and Matter” by John Urschel and Louisa Thomas
    Getting a PhD in math can’t possibly be as fun as this book makes it sound.
  • “Becoming” by Michelle Obama
    Obama should write a sequel entirely of stories about her parents being cheeky.
  • “The Fifth Risk” by Michael Lewis
    My friends in middle school had multiple conversations about how much we loved The Weather Channel. This book felt like the grown-up version of that.

Perspective Check

(The collections and the John Urschel / Louisa Thomas book throw the totals off a bit)

By Only Men: 15 / 53
By Only Women: 33 / 53

By Only White People: 36 / 53
By Only Non-White People: 12 / 53

Fiction: 28 / 53
Non-Fiction: 25 / 53

Recent Releases (2018 or 2019): 23 / 53
Modern Releases (2010–2017): 16 / 53
Classic Releases (pre-2010): 14 / 53

Happy 2020!

The Kubectl with a Thousand Faces
Photo by Yuri Bodrikhin on Unsplash

Commit of the Month / October 2019

Welcome to the Commit of the Month, the blog post series where we highlight recent work on Tilt.

October’s commit is


Or for you humans:

facets: show the applied k8s yaml

What Does it Do?

When you’re watching a resource in Tilt, you used to see two tabs: Logs and Alerts.

The Logs tab displays the most recent output of building and running something on Kuberenetes. That includes image build logs, pod logs, events that popped up, etc. All of it is in recency order.

The Alerts tab displays messages that we think you should look at Right Now, like build failures or pod crashes.

This commit adds data to a third tab, the Facets tab.

Facets are diagnostic information about your dev environment. If your dev env is misbehaving, and you’re not sure why, you can dig into the facets tab to get more detail on what Tilt knows. This includes:

  • The Kubernetes objects that Tilt applied to your cluster
  • The most recent build log
  • The build history

Maybe these details will point to the problem. Maybe they’ll help rule out a few suspects. Maybe they won’t help at all!

Wait, Why Does it Do That?

Kubernetes exposes a ton of its internal state via its API. kubectl lets you systematically query most of that state.

That makes kubectl the monkey's paw of command-line interfaces. It will always give you what you ask for. But it may not give you what you need. A big part of learning Kubernetes is just learning how to navigate kubectl.

We want to make Kubernetes a pleasant development environment for everyone, not just kubectl pros.

On team Tilt, we spend a lot of time talking about how to make the right information available when you need it. We don’t want to hide information from you. But we also don’t want to overwhelm you with irrelevant dumps of Tilt’s internal state.

How does a vague idea to display some internal Kubernetes detail evolve into an essential “check engine” light?

The Facets tab is a laboratory where we try out new displays. When we’re hacking on our own projects and have a problem, we check the Facets tab to see if the data helps. As we put this to use and watch other teams play with it, we expect to get a better sense of when this information should “pop up” in other places in the interface.

So the next time you’re puzzling over why your app is broken, check the facets tab! We’d love to know if you found it useful, or even if you didn’t.

Thanks Matt!

Further Reading

Lessons from Building Static Analysis Tools at Google, a great overview of some of the trade-offs in tooling that tries to surface high-quality information to developers.

Originally published at on November 12, 2019.

The Kubectl with a Thousand Faces was originally published in Tilt Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tips on Moving your Dev Env from Docker Compose to Kubernetes
Photo by Olivier Miche on Unsplash

When I first started learning how to write Kubernetes configs, I would sometimes complain to people about it. “They’re so complicated!” they would complain back.

They would show me an example. Here’s a simple Docker Compose config:

- 8000:8000
command: sh -c 'node server.js'

And here’s the equivalent Kubernetes config:

apiVersion: apps/v1
kind: Deployment
app: simple-node-app
name: simple-node-app
app: simple-node-app
app: simple-node-app
- command: ["sh", "-c", "node server.js"]
name: app
- containerPort: 8000

“Why can’t Kubernetes configs be as simple as my Docker Compose configs?”

After two years of playing around with both of them, I think I’ve figured out the answer!!

Kubernetes configs force us to handle change.

When I write the Docker Compose file, I get to optimize for a world where:

  • Our servers only run on a single machine
  • One instance of each server runs at a time
  • When I reload, I stop all the servers and start them all up again

When I write a Kubernetes file, I have to think about how to upgrade that system:

  • Our servers could run on different machines
  • A new version of a server can gracefully replace an older version
  • Each component is independently replaceable. We can replace each server process without touching the others, and redirect traffic without touching the server processes

Most Docker Compose files we see are small. That makes them easy to start! But they also don’t give you the right tools for managing updates and complexity. Once a Docker Compose file grows beyond a certain size, it becomes a tangled mess that no one can touch without breaking. Teams end up replacing their big Docker Compose files with someting else. It’s survivorship bias in action.

I’ve met many teams whose Docker Compose files are in this state. They hope that migrating to Kubernetes configs will make it easier to maintain. But they’re not sure where to start!

Where Not to Start: Pods

A good strategy is to start with a minimum viable migration, verify that it works, then iterate. But a big mistake people make is how they define “minimum”:

  1. I have a Docker Compose file with several apps.
  2. The minimum Kubernetes primitive for a single running app is a pod.
  3. Therefore, I should convert all app configs to Pod configs and see if it works end-to-end.

Don’t do this!

A pod is a good building block but a terrible dev experience.

When you replace a Pod, Kubernetes will, in serial, signal the process to shut down, gracefully wait for it to exit, free all the pod resources, and then start your new Pod. If you’re iterating on your config, this process will be too slow.

But with a bit of planning, Kubernetes can parallelize these steps. It can run the new process and the old process side-by-side, directing traffic to the new server while the old one is shutting down.

Key Insight: Play to Kubernetes’ Strengths

Kubernetes is good at incremental updates, independent components, and reusability. How can we take advantage of that?

Step 1) Start with a docker-compose.yml file

You can look at the one in our sample repo

Step 2) Run kompose convert

Kompose is a tool that helps convert Docker Compose configs to Kubernetes configs.

Kompose will create a Deployment YAML and Service YAML for each app.

They won’t work. Don’t Panic! That’s OK.

Kompose has lots of bugs. It’s still using beta Kubernetes Deployments by default and seems unmaintained. For most projects, it won’t be a complete solution. But it’s still a great start to build from.

Step 3) Get a “base” service working

Find a base service that doesn’t depend on any other services. Make sure it has both a Deployment (for running the server process) and a Service (for directing traffic to that process).

Deploy it to a local Kubernetes cluster. Tweak the Kubernetes configs and keep kubectl apply-ing until it's running. Usekubectl port-forward to connect the service to a port on your machine, and test it manually with a browser.

If you get stuck, kubespy trace is a great tool for checking common misconfigurations.

Step 4) Get a service the depends on “base” working

Find a service that only depends on your base service. This is a good way to explore how networking works in Kubernetes.

Keep kubectl apply-ing it until it's running OK, just like you did in Step 3.

Repeat until all services work!

Step 5) Throw docker-compose.yml away

This is a good time to throw a party.

How Tilt Can Help

One of the big reasons we made Tilt is to make it easier to iteratively hack on YAML files. The workflow I use:

  • Create a Tiltfile with the new K8s configs behind an if branch, so that I can check them in one at a time without affecting teammates using Docker Compose.
  • Leave Tilt running while I hack on YAML files, so that Tilt continuously redeploys them
  • Add port-forwards to the Tiltfile, so that I can look at the services with a browser without manually invoking kubectl port-forward for each new pod.

Here’s an example Tiltfile I was noodling on while writing this blog post.

Building maintainable cloud-native dev envs is hard. We want to make it easier!

Originally published at on September 16, 2019.

Tips on Moving your Dev Env from Docker Compose to Kubernetes was originally published in Tilt Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Local Development at KubeCon EU

The Tilt engineering team is back from KubeCon in Barcelona. Thanks for everyone who visited our booth to press our button!

We went to KubeCon to learn more about real-world struggles for engineers building services on Kubernetes.

Folks, the struggle is real.

Here’s a quick view of some of the common trends we saw:

Bazel: A Better Container Builder

Three talks discussed Bazel:

Bazel is a language-agnostic build system based on over a decade of experience with large-scale build tools at Google. We’ve been tongue-in-cheek critical of Bazel in the past, but it’s a great tool.

Bazel can also create reproducible container images. Unlike Dockerfiles, Bazel ensures that you get fast, iterative builds every time. No need to carefully stack Dockerfile layers like a Jenga tower to maximize caching.

If you’re using Tilt to run your microservices, Tilt has a plugin system to have
Bazel build the images. Head on over to our Bazel & Tilt guide to learn how. Two of the speakers mentioned that they were already using Tilt! 😊

Connecting Local Dev Across Services

If you’re developing services in Kuberenetes, Tilt isn’t the only game in
town. We stopped by the booths of other teams in this space.

Telepresence tries to solve this problem by throwing networking at it. They give you a toolkit of network proxies that let you run a server locally, then connect it to the servers running in your cluster. The tools are flexible with many options, depending on how high fidelity you want the network to be.

Garden has built a graph-based visualization of the services you’re working
on. You codify which services in that graph update when you change a file. Then, when you make a change, you can watch your changes automatically propagate through the build-run-test graph.

For us, it was exciting to see multiple creative approaches to solving the same

Kubernetes Clusters on Your Laptop

For a long time, Minikube was the best way to create a local Kubernetes cluster for testing. But now there are so many more tools with much lower overhead!

The KIND (Kubernetes IN Docker) team gave two talks, one focused on using KIND to test Kubernetes itself, and one focused on using KIND to test your apps & controllers. They’ve put a lot of work into making the startup time fast, so that it’s cheap to throw away a broken cluster and start a new one.

Konstantinos Tsakalozos of the MicroK8s team was at the Ubuntu booth. MicroK8s makes your Linux desktop a single-node Kubernetes cluster. The overhead is so low that it’s been my go-to for local dev.

Rancher had a booth too. I haven’t tried out their k3s / k3d yet. But the idea
of a Kubernetes with non-essential features stripped out for local dev is super
appealing. If you’ve tried it, we’d love to hear what you think of it and how it
compares to the others.

Post-KubeCon Development

We hope you come away from this post with a better understanding of the problem space, and where the community is making progress. We’re hoping to post some more digested thoughts in the next couple weeks.

If you’re interesting in learning more about how Tilt approaches these problems, we’d love to hear from you.

We hang out in the #tilt channel in Kubernetes slack (get an invite at We write code in windmilleng/tilt.

Maybe we’ll see you at the next KubeCon?

Local Development at KubeCon EU was originally published in Tilt Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.