Forem: Eyal Bukchin

mirrord as an alternative to Telepresence

Eyal Bukchin — Fri, 11 Aug 2023 09:50:22 +0000

A question that comes up often from those already familiar with local Kubernetes development is how mirrord compares to Telepresence. The idea at the base of both products is indeed similar: instead of deploying your new code to the cloud and testing it there, connect it to the cloud from your local machine. By shifting left on cloud testing this way you utilize your cloud environment much more effectively and speed up your development process. The people at Ambassador Labs wrote about it at length.

However, the technical approach at the base of mirrord is very different from that of Telepresence, which translates into significant differences in usability, compatibility, and performance. What Telepresence does is install an operator in your cluster, then connect you to the cluster via VPN (either your entire development machine or a containerized subset). On the other hand, this is what happens when you run a local process with mirrord:

mirrord starts a pod (called the mirrord agent) on the same network namespace as your target. This pod is cleaned up automatically at the end of execution.
mirrord then injects itself into your process, overrides low-level functions and relays them to the mirrord agent, who then executes them on the target in the cluster and sends back the results. For example, when your process tries to read a file, mirrord intercepts that function call and instead asks the agent to read that file from the remote target. mirrord does this for everything - traffic, file operations, environment variables - so your process behaves as if it was executed within the Kubernetes cluster, and with the entire context of your target.

It’s pretty heavy lifting compared to just starting a VPN - mirrord has to hook and reimplement a lot of functions for everything to work smoothly across different frameworks. But there’s a payoff:

By providing your local process with the entire remote context of the pod you chose to target, mirrord natively supports complex flows. For example, if when your process receives a request it queries a database - since mirrord provides it with the pod’s environment variables and files, it’s going to have the necessary credentials, and since mirrord intercepts the outgoing requests and sends it out from the remote pod instead, it won’t be interrupted by any network rules blocking access from outside of the cluster.
mirrord doesn’t need root access on your machine. All it does is override the functions of a running process; the rest of your machine remains untouched
You don’t have to install anything on your cluster - mirrord uses the Kubernetes API, so all you need is a configured kubeconfig. mirrord does create a pod when it runs, but it cleans itself up at the end of execution.
As you might have guessed, mirrord supports traffic mirroring. It can still intercept traffic like Telepresence does, but if you want to leave the remote process completely untouched, mirrord can also just duplicate its traffic and send a copy to you. The original requests are handled by the remote service, so people accessing it are completely unaware that you’re using mirrord to debug it.
As opposed to Telepresence which works at the network level, mirrord works at the pod level. This means it also supports pods that aren’t exposed through a service, and gives your local code access to anything the pod can access, including components outside the cluster. It also natively supports environments with service mesh like Linkerd or Istio.
Because the mirrord client works at the process level, you can easily configure exactly what’s executed remotely and what stays local. For example, you could make all the components of the remote cluster accessible to your local process, but have it write to the database locally; or have it read files from the local file system, except a few specific files that you want to get from the remote pod.
mirrord is a tool for developers, and as such, in addition to a CLI tool, it also comes as an IDE plugin for both VS Code and the IntelliJ IDEs. This is the most popular way to use mirrord, where it naturally fits into the existing development workflow.

Most importantly though, all of these things are supported out of the box and with no required changes to your system. You don’t have to set up mounts, manually ingest environment variables, or create headless services. Just run your process with mirrord, and everything works. It’s this lack of friction that really makes the vision of local Kubernetes development possible - a true shift left of the cloud, where running your code in Kubernetes isn’t the thing you’re saving for last after all the unit tests and local setups, but rather the first thing you do when you write new code.

Usage is simple and intuitive. In the CLI:

And in the IDE - enable mirrord, run or debug your code, and select a target. For example, in VSCode:

Try it out!

If you want to take mirrord for a spin, check out the quick start guide. We’d love to hear about your experience or just general thoughts - chat us up on our Discord or open an issue or discussion on GitHub.

Approaches in Cloud Development Ergonomics

Eyal Bukchin — Tue, 15 Nov 2022 15:47:30 +0000

The advent of microservice architectures and cloud-native has taken some pretty severe tolls on developer ergonomics. It feels as if the tools cloud developers have at their disposal haven’t evolved fast enough to keep up with the rapid progress in infrastructure, and this tool debt is readily apparent in the day-to-day of the modern developer.

The clearest example of this is that it’s now really hard to just run your application. Not the one microservice you’re currently working on, but your actual entire application, which you ultimately deploy to the cloud and serves your customers. You used to be able to fire up your monolith right there on your laptop from within your IDE (crash, change some code, run it again), but architectures today have become resource-intensive, reliant on third-party services, and mostly just plain convoluted to the point where local execution is no longer a viable option.

When we talk about something as elementary as running your application, it naturally applies to many aspects of the developer’s day-to-day: debugging, testing, even just writing new code to see if it works. Therefore, solutions that make your application even a little easier to run provide significant value, just by affecting so many routine activities.
In this article, we’re going to discuss today’s dominant approaches to making microservice applications easier for developers to run, and we’re also going to make the case for our own thing - namely, a single shared environment in the cloud that the organization maintains, and that developers can plug in and out of non-intrusively as they develop their microservices.

Contemporary Cloud Development Workflows

In the frictionless microservice utopia, the development workflow goes something along these lines:

Microservices are independent and new functionality can be developed one microservice at a time
Microservices talk to each other via well-defined contracts which can be tested in isolation
Databases, queues, and third-party services consumed by a microservice also expose a consistent and well-defined contract, and as such can be mocked for testing purposes

In other words, a service can be tested on its own, and then deployed to the production environment where it would play nicely with all the other components.

However, let's assume you’re a microservice developer who doesn’t work in an organization where everyone designs perfect contracts and writes rigorous tests (or they do, but the third-party managed services you rely on don’t). When writing new code, you probably make as much progress as you can within the scope of the one or two microservices you’re working on at the moment, but then you do one or more of the following:

Wrangle your entire application - microservices, databases and all - to run on your local machine
Deploy your new code to a personal persistent/ephemeral development environment in the cloud
Deploy your new code to a shared staging environment and test it there

Run Locally

This approach works great when it’s feasible, which is usually at a very early stage in the life of the application where it’s still small and tenable. There’s some tooling that lets you extend this honeymoon phase by letting you do it more easily, like docker-compose, Skaffold, or Tilt. However, at a certain point, even if you’ve written whatever scriptage is needed to actually configure and run the latest stable version of all of your components together, you’re going to hit some sort of ceiling: if you’ve got a large database, or some CPU-heavy computations, or you’re relying on some managed service that can’t be containerized, this approach soon becomes untenable.

Pros:

Convenient development on your local machine
No reliance on network or external services

Cons:

Difficult/impossible for large applications (ones with e.g. many microservices, a large database, high bandwidth usage)
Doesn’t support hard-to-containerize components, like managed services

Personal Cloud Environment

With Infrastructure as Code at its current state of maturity, it’s now easier than ever to replicate microservice environments in the cloud. This unlocked a new approach of having a personal production-like cloud environment for every developer, which they can use freely and in isolation. It comes in two flavors - persistent environments, or ephemeral environments created on demand with products like Okteto or Bunnyshell (also sometimes called Environment as a Service)¹. This approach overcomes the resource limitations of the local environment but substitutes them for some new difficulties:

For persistent environments, cost (determined by the size of the environment times the number of developers) becomes a factor. As the company grows, so do its architecture and the number of developers, leading to soaring cloud bills
To generate state, database seed files are used. These need to be maintained so they don’t become stale
You have to work against a cloud environment, which isn’t always fun (though some tools make it easier, like GitHub Codespaces or Gitpod)
Same as with running your application locally, having hard-to-containerize components in your architecture like managed services can make this approach unfeasible.

Pros:

Not capped by the hardware limitations of your development machine
Better simulation of your production environment

Cons:

No production-like database state
Local changes have to be synced to the cloud
Doesn’t support hard-to-containerize components, like managed services

Shared Cloud Environment

A lot of organizations have already put in the effort of setting up a single, shared production-like environment for testing - the staging environment. It’s persistent and has already been set up manually to include all components, including third-party services, databases with full state (some companies even automatically stream data from production), and all the microservices. In terms of simulating production, it’s hard to do much better.

However, since this environment is shared by multiple developers, some measures are usually in place to keep it stable - for example, you can only actually deploy new code through the CI, often meaning you have to wait for a code review and your entire suite of automated tests to run (not to mention the deployment script itself) before it actually gets there. Even then, the tests aren’t perfect, and sometimes you end up deploying broken code and corrupting the environment, much to the chagrin of your fellow developers.

This is where shared environment tools like Telepresence and CodeZero can help. They assume you're only working on one or two microservices anyway, and running them locally is not an issue. These tools let you connect your local service to the staging environment, replacing the service currently running in the cluster, without deployment. The code you're working on runs locally, and its dependencies run in the cloud. Not only can you keep working on your local environment, you can iterate on testing against your cloud environment without waiting for lengthy CI and deployment processes.

Pros:

Leverage a single shared environment, making it feasible for you to manually set up database state and managed services (since you only have to do it once, for one environment)
You can still work locally within your IDE
Code changes happen locally, so iterations are fast (no need to deploy to the cloud)

Cons:

One developer can break the environment for the others
If the environment is corrupted, it’s hard to recover

The Sell

So local setups are easy to use but don’t support large deployments; personal cloud environments support large deployments but don’t really reflect production, don’t support components that can’t be containerized, and cost a lot; and shared environments reflect production for cheap, but don’t really support concurrency without sacrificing stability.

mirrord lets you have all of these cakes and eat them too. We essentially took the shared environment approach and tacked on isolation. We do this by:

Letting developers mirror incoming traffic rather than just intercept it. This lets multiple developers work on the same service in the cloud, while the service is still running and handling requests.
Giving developers fine-grained control over what happens locally and what happens remotely. This way, for example, incoming traffic mirroring and file reads can happen remotely, but outgoing traffic and file writes can be local. Future versions of mirrord will include more advanced modes of configuration, letting users filter by e.g. hostname or protocol.
In future versions, supporting advanced features like copy-on-write for databases.
In our upcoming cloud solution, providing RBAC, letting the organization control which teams have access to which parts of the staging environment.

By using mirrord, developers can run the one service they’re working on locally, without juggling configurations or straining their development machine; they can run it against a full environment with mature database state and managed services; and they can do it quickly, easily and without risk.

We’ve recently released our first stable version, and would love to hear your thoughts (on GitHub or Discord).

It can also be supplemented with a cloud development environment, as provided by products like Gitpod or Codespaces. For a comprehensive survey of the non-localhost space see this excellent article. ↩

(Re)Introducing mirrord (or: the War on the Long Dev Loop)

Eyal Bukchin — Tue, 31 May 2022 09:12:57 +0000

Originally posted on the MetalBear blog.

The Dev Loop (or: know your enemy)

Imagine you’re a backend developer at post series B SaaS soonicorn. You’ve spent half a sprint adding a new feature to your microservice. You’ve thoroughly researched possible inputs and database states and built an elaborate test suite. Your code was mercilessly reviewed by two of your teammates. Finally, the tests pass, the pull request is approved, and as a final verification, you deploy your new code to the staging environment…

Where it crashes almost immediately. Turns out you forgot to test some obscure configuration, or some complex database state that would only ever occur on a mature, complex environment - like staging, and unlike your local machine.
So you roll the service back on staging, fix the bug, write some more tests, get it reviewed again, build the code, deploy it to staging… and it crashes again. While you were fixing the bug, your teammate deployed a new version of some other microservice - except they're not as thorough a tester as you are, and the new version is broken, so you can’t test your own service. You wait for them to roll their service back to a working version, the sprint is long over, and you’ve started on your next feature during your downtime so now you’re dividing your focus between two things.

The worst part being that you know it’s going to happen again next sprint.

mirrord (or: how you win)

I’ve encountered some variation of the dev loop at every startup I’ve ever worked in, and it’s this pain exactly that we’re trying to solve with mirrord. mirrord runs in two places - on your local development machine, and in your staging environment. By wrapping your local process and hooking system calls, it lets you plug your process non-invasively into that service’s instance on the staging environment.
Simply put, mirrord lets you run a local process in the context of your cloud service. This means you can test your code on staging, without actually deploying it there. Not only does it save you the hassle of deployment every time you make a change, but it also keeps the staging environment free from untested versions, and constantly usable for everyone else in the organization.

Getting Started (and: other practical info)

Today we’re releasing mirrord 2.0, which supports incoming traffic. That means whatever traffic reaches your service in staging is duplicated by mirrord and sent to your local process. We’ll soon be adding support for outgoing traffic, file access, and environment variables - everything needed for your local process to “think” it’s running on the staging environment.
All you need to run mirrord is kubectl access to your staging environment¹. You can use it as either a CLI tool, or an extension for VS Code (IntelliJ support is on the way). Try it out here, and let us know what you think at hi@metalbear.co, or in our repo!

note that mirrord doesn’t install anything persistent in your Kubernetes cluster. It runs a job that self-deletes when mirrord terminates. ↩