Forem: Marcos Brizeno

Lessons learned from one year with Kubernetes and Istio

Marcos Brizeno — Wed, 25 Dec 2019 15:41:25 +0000

tl;dr;

We've been happy with all the features of Kubernetes and Istio to create a service mesh for APIs and web apps. It comes with a cost and require a team of people willing to sometimes dive deep into the unknown.

Kubernetes is pretty mature and if you are using a cloud provider you can get up and running pretty fast. Keeping it up to date is also fairly straightforward and, most of the time, upgrading components won't even require downtime.

Istio on the other hand requires a lot more time to get right and, although it adds a really wide array of capabilities, it has a considerable maintenance cost.

If you don't have a safe environment to learn and fail (be it due to time constraints or the criticality of projects) I recommend sticking with just Kubernetes, you'll get a lot from using it. Meanwhile try to look at service mesh capabilities since you'll probably find the need for them early on, maybe try implementing them one at a time.

If you want more details on why, keep reading :)

Before you begin

Let's just get the fact that Kubernetes is complex out of the way early on this article. There are many components for you to learn and figure out the right configuration in multiple yaml files until you get your application up and running.

Imagine if you were part of a team or company that has to run 100+ microservices. Even if all of them are extremely well designed and developed into a standard tech stack, following all the best practices and building quality into every step of the way, it is still a pretty hard and complex problem to solve.

Running a distributed service oriented architecture is really complex, there's no easy way to do it (and if anyone tells you there is they either have never done it or are straight up lying). In my opinion Kubernetes offers a programatic and pretty standardize way to deal with all requirements to run a distributed architecture. The API is well designed and after the initial learning curve you'll find to be quite powerful.

Kubernetes will require you to really understand your applications. How can you signal that a service is ready to receive traffic? How can you signal they are unhealthy and need to be restarted? How many instances be created? How to route traffic between different versions of services?

In my experience as a developer these questions were rarely asked, but to really benefit from Kubernetes and Istio you need to have answers sooner rather than later.

I want to highlight two main areas which I think are really important in a distributed environment: compute engine (where applications are running) and service mesh (how all the services integrate with each other).

Compute engine

The first big challenge of adopting Kubernetes and Istio is not the technology but the mindset. As mentioned in the previous sections if you are deploying your application in Kubernetes you need to have answers for a lot of questions that rarely get asked and looking at the bigger picture can be a daunting experience.

The benefit you gain from running an application is Kubernetes is pretty incredible. Having an elastic and highly available infrastructure frees you from worrying about a lot of common problems. Once you put all the pieces in place it is a pretty smooth ride. The same principle is valid for the infrastructure itself, creating a cluster that scales as needed is fairly straightforward if you use a cloud provider.

From the perspective of developing the infrastructure itself, deploying and running a cluster is not as hard as most people may think - but not as easy as shown in a presentation. Many cloud services have their own self-managed version of Kubernetes which takes care of a whole lot of concerns you'd have running a cluster on your own.

The second biggest challenge you will probably have is to keep an eye on all the moving pieces: Kubernets, Istio and many, many, many plugins. Luckily the environment around both tools is ever growing and with a pretty standard API, managing resources in a cluster is not the worst experience in the world.

To give an example, right out of the box Istio will give you a good array of monitoring tools like Prometheus, Kiali, Jaegger and plugin in other services is also straightforward (like Datadog or Splunk) since most of them provide easy adapters. We then built templates for alerts and dashboards that work based off of the standard metrics that Istio and Kubernetes plugins provide. This way all teams had a good baseline of observability before even starting instrumenting their applications.

Keeping track of all the dependencies and making sure everything is kept up to date is also another big challenge. Important security and performance updates are released constantly, some tools may not even be at 1.0+ yet. You'll need to invest time to make sure you have pretty good testing environment and be confident in your CI/CD process to release changes.

Service Mesh

Defining a service mesh is pretty hard because there is a lot to unpack, especially with Istio and the many capabilities it brings. Istio extends the Kubernetes API by creating its own resources and allowing you to define configurations ranging from terminating TLS and encrypting requests between services to traffic shaping and tracing across multiple services.

This also means that when something doesn't quite work it can be really hard to find out why, which is definitely the biggest challenges with a Service Mesh. Things go wrong without much explanation and require you to look into the logs for the many components running to find out clues on what is missing, misconfigured or simply broken.

To give a more concrete example let me tell about when we tried to setup automatic TLS resolution. With Istio you can set up TLS termination outside of the application, meaning the app running in the docker container uses simple HTTP but the requests are served with HTTPS. This not only simplifies the application code but would also allow us to automate the whole process by having an automated TLS certificate generation process.

All was great, teams could deploy their own TLS key and certificates and Istio would take care of TLS termination. Then, one day, all ingress traffic to the cluster started to fail, no request would make it to any app in the cluster. After long sessions of debugging the problem (from the external load balancers all the way to the VPC configuration) we finally saw some logs indicating that Istio's ingress application was not able to parse a certificate. Although we thought it had nothing to do with the issue, once we deleted the bad cert all traffic came back to normal.

We then saw on a similar github issue that we could setup a default certificate to be used when the ingress pod failed to load a certificate, saving it from looping indefinitely/crashing. Once we deployed this configuration everything went back to normal and even when someone deployed a bad cert the service mesh would handle it gracefully.

There are many other cases that required a digging really deep into logs, github issues and source code. If your team doesn't have a safe environment to learn, or simply doesn't want to deal with that kind of experience, I definitely recommend not using Istio or else you'll get burned and start posting memes on twitter about how complex it is.

Remember to smell the flowers

Seeing everything coming together is a really good feeling and while there are many reasons to worry and lots of things to learn, it feels great to intentionally destroy 90% of your resources and watch as everything comes back automatically and no downtime is noticed :)

It may be overwhelming to get started with Kubernetes and Istio, but they also have great people doing a really good job to solve really complex problems. I am thankful for the many people dedicating their time and efforts to develop those tools.

This post has been almost a brain dump from when I stopped and looked at what we've put together over an year. If you want to know more about some specific area please let me know in the comments or @MarcosBrizeno. I plan on writing a few other posts to talk about the architecture and components we use - and why we use them.

A bit of context

I am part of a team focusing on the cloud infrastructure where application teams can deploy and run their services. It is not a DevOps team or even an infrastructure team, we try to provide the best developer experience we can so that the other teams feel empowered and are enabled to do the best job they can.

Most of the services running on our platform are greenfield applications - mostly APIs, a few UIs and some background jobs. The tech stack is pretty standard, about 80% of all apps use the same language/web framework.

The teams deploying applications vary in experience with microservices, continuous integration and cloud native development.

See you space cowboy

Optimizing docker to build and run your application

Marcos Brizeno — Fri, 05 Oct 2018 12:25:58 +0000

Docker is a great tool to for running applications without worrying about dependencies, but it can also be used to build your application. In this post I'm gonna share a nice technique I've used to build and run a web app with Docker and stopped worrying about dependencies.

Building vs Running

Building an application usually requires a lot more dependencies than just running it and most languages and frameworks offer a way to build your application bundling everything you need to run it.

As an example a Java application can be compiled to a single .jar file, Go apps can be compiled to a binary and even applications in Ruby can be bundled in a SO package (like rpm). This makes it a lot easier to run the application, and even more so if you use Docker.

Once your application is bundled, you can just copy it to a Docker image and have docker run it for you - and only worry about installing Docker itself. What about actually building the whole app in a Docker image so you don't even need to worry about the dependencies even on your machine (or on a CI server)?

The main problem with building your app on the same image that will be used to run it is that this image will have to have everything required for both building and running.

Every command you add in a Dockerfile will add a layer to your image. Think of a Docker image as a git repository, the very first FROM command is the initial commit for that repo, each command after that is like a new commit that will potentially increase the size of your repository. The more commands you have in a Dockerfile the bigger your image might be.

So, how about using two different images? One for building and another one for running?

Building and Running

I was working on a small web app (using Java and Spring Boot) and while searching for best practices around docker I ran into this very useful video. It talks about different ways to reduce the size of your image (which is specially useful in the context of Kubernetes).

So I decided to apply these techniques to build and deploy this Java/Spring Boot web app, using one image to build the .jar file and another one to run it. We used gradle on this project so my first step was to use the gradle base image to build the app.

FROM gradle:jdk8 AS build-env
# Setup ARGs and ENVs
# Some RUN commands
# Copy the code
COPY --chown=gradle:gradle . .
# Some more RUN commands
# Finally, build the app
RUN ["./gradlew", "build"]

This image is actually pretty big. First of all it uses a base image that comes with a lot more stuff (comparing to the -alpine alternative), because I actually need tools like curl and others that I'd have to install myself if I had used a thiner version.

And that's actually the point of using a build image, you don't need to worry about what's there because it will be discarded once the final image is built.

Now I can define the image that will run the application:

FROM openjdk:8-alpine
COPY entrypoint.sh entrypoint.sh
COPY --from=build-env /home/gradle/build/libs/my-app-0.0.0.jar my-app.jar
ENTRYPOINT [ "sh", "entrypoint.sh" ]

This is a pretty thin image, it uses the 8-alpine version of openjdk, which is much smaller and does the job well, since Java is pretty much the only runtime dependency.

Both of these declaration live on the same Dockerfile, note how the build image has FROM ... AS build-env at the beginning. This will create a temporary image with a build-env alias and since Docker only considers the last FROM statement to be the actual image it will be discard afterwards. Then on the runtime image it copies from the build image with COPY --from=build-env.

And there you go, building and running an app using a single Dockerfile while optimizing for each context. Using this pattern can even help simplify your CI by only having to install Docker on the agents.

Thanks for taking the time to read this and hope it helps you to improve your own images :)

Tail recursion in Elixir

Marcos Brizeno — Sat, 04 Aug 2018 20:06:14 +0000

You've probably heard (and written) some recursive functions/methods before. Chances are you've also heard that recursive functions might be more expensive when compared to a non-recursive alternative.

Let's see what is probably the simples example of a recursive function, the factorial implementation. Here we define two clauses of Factorial.recursive_factorial/1, on for 0 (which returns 1) and another for any other value:

defmodule Factorial do
  def recursive_factorial(0), do: 1
  def recursive_factorial(n), do: n * recursive_factorial(n-1)
end

On each call of recursive_factorial(n) we multiply n by the result of recursive_factorial(n-1) (which is a recursive call until n reaches 0). On every step we need to keep the current value of n so we can multiply it once recursive_factorial(n-1) is returned.

This is why recursive calls are usually considered to be more expensive (specially in terms of space), each call requires the stack to be saved until the result comes back. Saving a stack, even if it is just that n might start causing trouble when the number gets too high (say 500_000 or more, depending on where you're running it).

But there is something we can do to make it better and as performant as the non recursive alternatives

Tail Recursion

Take a look at this other implementation and compare it to the previous one:

defmodule Factorial do
  def tail_recursive_factorial(n), do: tail_recursive_factorial(n, 1)
  defp tail_recursive_factorial(0, acc), do: acc
  defp tail_recursive_factorial(n, acc), do: tail_recursive_factorial(n-1, acc*n)
end

The first call to Factorial.tail_recursive_factorial/1 starts the process by calling the private Factorial.tail_recursive_factorial/2 which does the recursive process.

Notice that the multiplication is now passed to the recursive call using the second argument. This means that there is no need to store the function stack for each call!

Each call now has everything it needs and the stack can be discarded as soon as the recursive call starts. The end clause here is when n is 0 and it just returns the value on acc.

When a recursive function doesn't need to store any temporary value to be used after the recursive call, it is considered a Tail Recursive function. Let's check a more interesting problem.

The 'Strain' exercise on Exercism

On exercism.io there is a nice little exercise where you're asked to implement two functions, keep and discard. Given a list of items and a function fun, keep will return the list of items where fun returns true and discard will return the list of items where fun returns false.

Me and some friends got together to solve this and our initial implementation of keep was to execute fun on the head of the list and if it is true return a list with the the head and then recurse on tail, if not then just call again ignoring the head of the list:

def keep([], _fun) do
  []
end

def keep([head | tail] = list, fun) do
  keep_if(list, fun, fun.(head))
end

defp keep_if([head | tail] = list, fun, true) do
  [head | keep(tail, fun)]
end

defp keep_if([head | tail] = list, fun, false) do
  keep(tail, fun)
end

The main problem here is that for each call where fun is true we need to save the value stored on head and after the recursive call finishes, the return has to be appended to a list containing head.

For the test scenarios we had, this wasn't a big issue, but we wanted to go a bit further and think about how to make this a tail recursive function.

Our optimization was similar to what we did for Factorial.tail_recursive_factorial/1, we create an accumulator argument and send it on each call instead of waiting to get the result back. Here is how it ended up looking like:

defmodule Strain do
 def keep(list, fun) do
   keep(list, [], fun)
 end

 def keep([], filtered, _fun) do
   filtered
 end

 def keep([head | _tail] = list, filtered, fun) do
   keep(list, filtered, fun, fun.(head))
 end

 defp keep([head | tail], filtered, fun, true) do
   keep(tail, filtered ++ [head], fun)
 end

 defp keep([_head | tail], filtered, fun, false) do
   keep(tail, filtered, fun)
 end
end

Another nice exercise to try and apply tail recursive optimization is the 'List Operations' one where you're asked to implement basic functions that we usually take for granted, like each, map, filter etc.

Recursive functions are quite common in functional languages, most of them don't even have loops, so learning about tail recursion and practicing how to implement it is a good investment :)

Don't let duplication mislead you

Marcos Brizeno — Thu, 09 Nov 2017 14:00:11 +0000

In this article I'm going to tell you about how I started to understand what software design principles mean and hopefully it will give you some insights on how to put them in practice.

Duplication is better than the wrong abstraction

Sandi Metz is one of my favorite authors, every time I read something from her it changes how I think about improving my code. Something that I kept thinking about lately was this phrase: "Duplication is better than the wrong abstraction".

The first language I used professionally was Ruby and the Ruby community is well known for making programming fun and creating readable code, so I got into reading a lot of books/articles on how to write good code.

One of the first things I learned was to name things, but since naming is a quite subjective topic, the second thing I learned really stuck with me: Don't repeat yourself!

The famous DRY principle taught me that duplication is bad for maintenance, so try and reduce it as much as you can. When I heard "Duplication is better than the wrong abstraction" my brain crashed. And I had to reboot it.

What do I not know about this code?

It was actually "easy" to understand why duplication is better than the wrong abstraction, my friend Luciano Ramalho said "it's better to walk than to take the wrong bus" and I think that is a pretty good metaphor. But it was only when I faced my self with a duplication and resisted the urge to extract a new method that I realized why duplication is better than the wrong abstraction.

I was taking a look at a code I had just written following the Shameless Green approach (from 99 Bottle of OOP - amazing book by Sandi Metz and Katrina Owen by the way) and found a easy to strike duplication:

def self.create_successful_event(event)
  create(leave_id: event.leave.id,
          name: event.name,
          data: event.to_json,
          status: SUCCESS)
end

def self.create_failed_event(event)
  create(leave_id: event.leave.id,
          name: event.name,
          data: event.to_json,
          status: FAILED)
end

Instead of just extracting to a method and passing stuff as parameter I thought to myself: "what do I not know about this code?". And then it hit me: duplication is not about two similar pieces of code, it's about an abstraction that is yet to exist.

The real problem is not just the duplication, the event object is completely exposed! It gets passed as a parameter and then completely shattered to make up this hash object that is then passed to the create method.

Other code smells might help

What I didn't know about this code, and in hindsight it looks kinda obvious, was that the event object should be responsible for building that data hash.

When an object cares more about receiving parameters than itself, it usually means that a feature is misplaced - a code smell known as Feature Envy. To remove the original duplication what I ended up doing was creating a new method on event to return the data hash and then just calling it instead of exposing the object.

def self.create_successful_event(event)
  create(event.data.merge(status: SUCCESS))
end

def self.create_failed_event(event)
  create(event.data.merge(status: FAILED))
end

I know data is a bad method name :/

Refactoring with confidence

I got into the habit of reading the code and identifying smells first, instead of just diving into the code to make improvements, and I think it really helped me refactor with confidence.

There are many automated tools that help identify code smells and I try to use them, but nothing beats intuition! Invest some time reading books on refactoring, code smells, design patterns and specially reading code.

Take some time in the morning to just read through your code, resist the urge to make changes though, focus on how the pieces fit together instead. This way you'll build the foundations of your intuition and understanding what all those design principles really mean.

This post is a longer version of this twitter thread

The Godclass

Marcos Brizeno — Thu, 11 May 2017 12:44:54 +0000

Alright, you got assigned to a team that is working on this - quite old - project and you're really looking forward into getting your hands dirty with some legacy code.

After a brief technical onboarding on the whiteboard by the techlead you go with her to a pairing station and start setting up the local environment for your first day of work. After exploring the code a bit you get to the Listing class and, as you scroll the never ending file, you already start hating this Godclass.

And, as you wonder how that class got this big, you start imagining how the first weeks of work on this project were. Maybe after a discussion with some "business people" the team figured it would be necessary to have the Listing class. From there it was easy to just add more and more stuff to it. This is a auction system after all, so it makes sense that the central element is a listing.

As you read the methods and test cases, it all start to make sense. Since a listing can be in many states - like draft, preview, published, ... - you need to control all of that. And each state enables different behavior, so sure there are quite some ifs spread around the code. A listing also contains a lot of information, so there is definitely a lot of fields in the object. But hey, at least it has a lot of testing!

As you see yourself starting to agree that all this messy code actually makes sense, you stop and slap yourself in the face (but that's all in your head, so don't worry, no one saw it). No class, no matter how important, should be this big and violate the Single Responsibility Principle like that!

A class becomes a Godclass

Of course, right? No one creates a Godclass from day one, as the old saying goes: it takes a village to raise a Godclass.

A class turns into a Godclass because it just makes sense! Think about a domain and what would be a good candidate for a Godclass: an e-commerce will probably have a Product, an auction system will have a Listing, and is very likely that an aviation application would have a Flight and a Passenger. It just makes sense.

Usually a Godclass will capture the essence of your domain, like a Product in an e-commerce, and it will probably become a central point of the design. Everything in an e-commerce platform revolves around people buying products, so the way you create a product, the way you show them and how you handle inventory is key.

Are we all destined to create a Godclass? Is it just a matter of time?

What if there's no Godclass?

If you work with an old enough system, you probably have a Godclass hanging around on your codebase (hopefully you'll have only one).

Now think for a minute, what if there was on Listing class on an auction application? How would you go about modeling the domain without using this name? How would you group the behavior for publishing a listing and handling bids without using the word "listing"?

Think for a minute. Ok, just a few seconds will do, I know we're all busy these days.

...

It's hard right? It just makes sense to have a Listing.

Not using a name that is central to your domain is probably a bad idea, since this name will capture the essence of your domain. But the exercise of completely eradicate the Godclass and banish the name from the vocabulary might be interesting.

If you don't have a Godclass, all that behavior will have to be split among other classes right? I mean, you'll have to support the same features and use cases, so the code will have to be there, somewhere. Will a new Godclass appear? Could some behavior be absorbed by an existing class? Can you identify what can be split in other classes? How those classes will look like?

Doing this exercise will provide good hints of how you could reduce the power of the Godclass and, potentially, improve the design of your application. It will also show how well you know your domain and how sharp your modeling skills are.

Don't forget that there's much more to software development than just writing code :)

Hi, I'm Marcos Brizeno

Marcos Brizeno — Thu, 11 May 2017 11:36:45 +0000

I have been coding for 5 years.

You can find me on GitHub as MarcosX and on Twitter as @marcosbrizeno

I live in Belo Horizonte working for ThoughtWorks.

I mostly program in these languages: ["ruby", "javascript", "python", "java"] and am currently learning more about elixir and crystal.

Nice to meet you.