Forem: Ambassador Labs

The Developer Experience and the Role of the SRE Are Changing, Here's How

Kate Packard — Wed, 28 Jul 2021 20:38:26 +0000

Mario Loria of CartaX talked with Ambassador about the changing developer experience, the changing role of the SRE, and delivering visibility and a self-service platform for developers

The application development landscape has fundamentally changed in recent years. In a recent interview with Ambassador Labs, Mario Loria from CartaX said he believes this is still uncharted territory, particularly for developers in the cloud-native space. As he sees it, site reliability engineers (SREs) play a key role in guiding developers through the learning curve toward comprehensive self-service of the supporting platforms and ecosystem, and ultimately to service ownership. This requires a major shift in company and management culture, and developer (and SRE) mindset and tooling as well as insight to make the journey to full lifecycle ownership not just smoother and more transparent but also technically feasible.

Two worlds colliding: The monolith and service-oriented architecture

The traditional monolith continues to exist in parallel with cloud-native application development. The operations side of the equation, according to Mario, understands that this has caused a big shift in deploying, releasing, and operating applications, and now the role of SREs is to help developers understand and own this shift. Developers know how to code, but building in the necessary understanding (and ownership) of the “ship” and “run” aspects of the lifecycle introduces a steep learning curve. For developers, this means taking on new responsibilities with the support of SREs.

Enabling developers to own the full application lifecycle

In a cloud-native, service-oriented architecture, how much responsibility for an application's lifecycle should a developer own?

According to Mario, developers should own the full life cycle of services but in most cases don't: "It should not be up to me as an SRE to define how your application gets deployed or at what point it needs to be rolled back, or at what point it needs to be changed, or when its health check should be modified." Developers should be capable of — and empowered — to make these determinations.

Many SREs' experience has forced them to "ride to the rescue" when issues arise. Developers have generally been conditioned to hand over deployment and operations to SRE teams rather than investigate how they might handle issues on their own. This isn't illogical; this is how pre-cloud (and pre-DevOps) monolithic application development worked. Getting to a place in which developers can own the full application life cycle isn't always a linear path, nor is it always organizationally sanctioned or supported.

How does an organization and its developers round the bend on the learning curve?

Understand the changing developer experience to support developer ownership

Mario shared a set of prescriptions on how organizations, SRE teams, and developers might better embrace and support the new development paradigm.

Refocus culture and top-down support

An organization and its leadership needs to get behind the end-to-end "developer-as-service-owner" mindset as part of their higher-level strategy. Leadership needs to make this clear from the top down that there is a focus (and accountability) on ownership. Developers learning this new culture come to it from their own history and experience, which can vary considerably. Changing the developer mindset starts with empathy — understanding their goals, practices and skills — and closely follows with good communication.

Rethink communication patterns

Mario believes that supporting developer ownership, and getting developers to embrace this model as well, comes down to communication. Communication has to change from an "it's their [SRE] problem" to a "this is our [developer] problem" frame of reference. Even down to the individual developer and their experience, each problem they encounter is likely something that affects the business. Everyone is pushing toward the same goals. If that fundamental idea of shared responsibility is broken, it's not really possible to change the culture to adopt and sustain the new developer experience.

Consider SRE-supported developer education and tooling

Mario's views on the role of SRE in supporting the new developer experience, and more fundamentally, developer education can be summed up with the old adage: feed someone a fish, they eat for a day; teach someone to fish, they eat for a lifetime. Mario's take is that developers should start triaging and debugging themselves when they run into issues. As he frames it, it's not that developers should never ask for SRE support, but instead that the role of SREs in cloud-native organizations should support developers learning to gather intelligence, troubleshoot basic issues, and understand what the components of their applications are doing without SRE intervention.

The SRE role, in this framework, supports the idea of SRE-supported education to help developers gain a better understanding of how to handle their services themselves rather than asking SREs to fix broken issues. Mario believes this is the way SREs and developers should work together in the long term: helping put the developer in the driver's seat while training them to drive high-performance machines (applications).

Reshape the developer and the SRE experience

To persuade developers to take on the responsibility, the SRE function needs to provide the platform surface area -- or control plane -- and visibility to make this possible. That is, giving developers control and providing visibility into what the repercussions of their actions can be. A lot of questions surface that developers will need to know how to answer effectively to, as Mario put it, “shift left”, make more decisions, and do more autonomously with interactive self-service.

SREs can create the platforms, or pure infrastructure, to empower developers. Developers need to ship their code safely, and the more they understand about how to do this, the more self-sufficient they become. Meanwhile the more SREs can focus on creating the "platform as a service", or “paved path”, that helps reduce cognitive load for a developer taking on full code-ship-run ownership, the more clarity the developer gets into the processes they need to ship and run their code without breaking anything.

In many ways, this reshapes not just the developer experience but also the SRE experience.

Ease the climb: Centralizing information and creating the single pane of glass

"In terms of developing, deploying, and operating [in the cloud]... especially if you think of someone you just onboarded into the team, there's a mountain they have to climb to actually understand how we manage our services end-to-end. Going back to the frontend we should aim to create something like a single pane of glass, where instead of having all these different places to look or tune, or to try and understand what's going on, we can see everything transparently…".

Giving the developer the tools needed to bring everything together is likely to hinge on something like a developer control plane (DCP), where information can be centralized and made visible.

From an operational standpoint, considerable thought needs to go into how much time and effort the business wants to invest in building or running all the constituent components themselves. While dependent on use case, cloud-native development is all about achieving speed, and this is facilitated by the centralization of distributed information and the creation of a single pane of glass to make a frictionless means for pulling disparate sources of information together to, in a sense "ease the climb".

Conclusion: Developers should work with SREs as collaborators, not first responders

For developers who want to thrive in the cloud-native development space, learning to rely on SREs in a new way will be a key success factor. SREs should become trusted partners for deploying, releasing, and running services, and not just treated as first responders who are responsible for dealing with incidents. Developers should take the opportunity to share their pain points and also learn about tooling and best practices from SRE teams, with the goal of “paving the path” to developer autonomy, self-service, and full service ownership.

Debugging Go Microservices in Kubernetes with VScode

Peter ONeill — Fri, 09 Apr 2021 14:04:58 +0000

Tutorial: Learn to debug Go microservices locally while testing against dependencies in a remote Kubernetes cluster

Many organizations adopt cloud native development practices with the dream of shipping features faster. Although the technologies and architectures may change when moving to the cloud, the fact that we all still add the occasional bug to our code remains constant. The snag is that many of your existing local debugging tools and practices can’t be used when everything is running in a container or on the cloud.

Easy and efficient debugging is essential to being a productive engineer, but when you have a large number of microservices running in Kubernetes the approach you take to debugging has to change. For one, you typically can’t run all of your dependent services on your local machine. This then opens up the challenges of remote debugging (and the associated fiddling with debug modes and exposing ports correctly). However, there is another way. And the CNCF Telepresence tool enables this path.

This article walks you through using Telepresence to seamlessly connect your local development machine to a remote Kubernetes cluster, allowing you to use your favorite debugging tools with all of your microservices. Giving you the ability to comfortably debug your remote apps with your existing skills.

The Difficulty with Debugging Applications Running in Kubernetes

Splitting your application into microservices introduces a number of challenges. Particularly with debugging. Splitting an application into several, if not dozens, of microservices creates a complex dependency tree that becomes nearly impossible to replicate in staging.

Sure, you can use unit tests with tools like GoMock and GoStub to simulate external dependencies or introduce synthetic data into the mix, but it still leaves you unsure if it will work with data from your actual services.

And after you have deployed your service into your cluster, running a remote debugging session can be tricky to get right, due to the complicated configuration of ports and protocols that need to be set via your Kubernetes service YAML. You often lose the ability to use your favorite debugging tools and strategies.

Step 1: Deploy a Sample Microservice Application

In this article, we’re going to be working with the sample Edgey Corp application written in Go. There are detailed instructions for deploying this application in my previous blog post. Or just apply the YAML from the URL to your Kubernetes cluster, clone the repository, and jump right into the action.

Apply the manifest for the Edgey Corp app to your K8s cluster

kubectl apply -f [https://raw.githubusercontent.com/datawire/edgey-corp-go/main/k8s-config/edgey-corp-web-app-no-mapping.yaml](https://raw.githubusercontent.com/datawire/edgey-corp-go/main/k8s-config/edgey-corp-web-app-no-mapping.yaml)

Git clone the code for the Edgey Corp app to your machine

git clone [https://github.com/datawire/edgey-corp-go.git](https://github.com/datawire/edgey-corp-go.git)

Now that we have the Edgey Corp app running in our Kubernetes cluster, and we have the code on our local machine let’s pop it open with VSCode.

Let’s check the application is running successfully. Run a kubectl get pods to see the pods are up and running for the 3 microservices. verylargedatastore, verylargejavaservice and dataprocessingservice.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
verylargedatastore-cd998dfc6-k5bkw 1/1 Running 0 19s
verylargejavaservice-77748f79d6-dx4kj 1/1 Running 0 20s
dataprocessingservice-f5b644d95-s98hp 1/1 Running 0 20s

With the sample application up and running we can configure VScode to start debugging.

Step 2: Configure VScode and Delve

VScode is a great all-purpose IDE with countless extensions to get you up and running quickly. We are going to use the Go extension made by the Google team with the Go debugger tool called Delve. Here are links to each tool:

You’ll want to make sure that Delve is both installed to your local system and connected to VScode through the Go extension.

Install Delve to your local system

go install github.com/go-delve/delve/cmd/dlv@latest

Connect Delve to VScode

Open VScode now
Access the command palette with (command+shift+p)
Select Go: install/update tools
Check the box for DLV and any other options you might find useful
Click Ok

Run your Go code in debugging mode

Open the project directory for the Edgy Corp Go App in VScode
Open up the file dataprocessingservice/main.go
Click the debug icon on the right side.
Lastly, click “Run and Debug”

You should now see the debug console starting to log output.

API server listening at: 127.0.0.1:48808
Welcome to the DataProcessingGoService!

Let’s check that it’s working by navigating to localhost:3000 in your browser. You can also navigate to localhost:3000/color to see how the Go application response to the color endpoint.

Step 3: Intercept Your Service with Telepresence

So debugging one microservice in isolation is fine, but it still leaves us assuming what is going to happen when it is connected to the rest of the microservice in our application. Let’s introduce Telepresence here to debug our local Go microservice as part of the larger application.

Install telepresence CLI

macOS

sudo curl -fL [https://app.getambassador.io/download/tel2/darwin/amd64/latest/telepresence](https://app.getambassador.io/download/tel2/darwin/amd64/latest/telepresence) -o /usr/local/bin/telepresence

Linux

sudo curl -fL [https://app.getambassador.io/download/tel2/linux/amd64/latest/telepresence](https://app.getambassador.io/download/tel2/linux/amd64/latest/telepresence) -o /usr/local/bin/telepresence

Make the binary executable

sudo chmod a+x /usr/local/bin/telepresence

Connect your local machine to your Kubernetes cluster.
```
$ telepresence connect
```
Test the connection by cURL-ling our front-end service via its Kubernetes service name.
```
$ curl [http://verylargejavaservice.default:8080/color](http://verylargejavaservice.default:8080/color)
“green”
```
Notice two things here:

A) You are able to refer to the remote Service directly via its internal cluster name as if your development machine is inside the cluster

B) The color returned by the remote DataProcessingService is “green”, versus the local result you saw above of “blue”

Let’s take our connecting to the cluster a step further now by initiating an intercept from our remote dataprocessingservice to our local debug session running on http://localhost:3000. This will send any traffic destined for the remote service down to our debugger

$ telepresence intercept dataprocessingservice --port 3000
Launching Telepresence Daemon v2.1.2 (api v3)
Connecting to traffic manager…
Connected to context peteroneilljr-office-hours ([https://34.67.161.22](https://34.67.161.22))
Using deployment dataprocessingservice
intercepted
Intercept name: dataprocessingservice
State : ACTIVE
Destination : 127.0.0.1:3000
Intercepting : all TCP connections

Let’s try to cURL our front-end service once again. This time we should see our local debugger logging the interaction between the front-end service and our local dataprocessingservice.
```
$ curl [http://verylargejavaservice.default:8080/](http://verylargejavaservice.default:8080/)
```
Awesome! Now that our local debugger is hooked up to the remote cluster we should be all set to start debugging!

Step 4: Step Through Your Breakpoints

Let’s set a breakpoint on the getColor function. Hover over just to the left of the line number and click on the red dot. Once it’s set, visit localhost:3000 in your browser, VScode should pop to the foreground. Click the play button to close the request and the webpage should finish loading.

Now your IDE is processing traffic directly from the cluster, let’s open up our sample application in a web browser:

http://verylargejavaservice.default:8080

VScode will pop open on the breakpoint again. This time let’s click the step button (the one with the clockwise rotating arrow) until the breakpoint reaches the fmt.Println and you see the c: “green” in the variables window. If you try to adjust the variable directly in the IDE variables pane you will receive the following error.

Failed to set variable — literal string can not be allocated because function calls are not allowed without using ‘call’

More details about the bug here: https://github.com/golang/vscode-go/issues/1173

To work around this issue we can adjust the variable in the debug terminal with the command call c = “orange”

Now let’s end the breakpoint and send the info back to the web browser. Click the play button in the top command bar.

Awesome! You should see the verylargejavaservice loading with the orange background colors. We’ve successfully made a request to our front-end service running in our remote cluster, where Telepresence intercepted the traffic sending it through our local debugging session, where we inspected and updated the data to send back the orange color variable.

Introducing Telepresence into the development flow gives us a bridge from our local development environment to our remote Kubernetes cluster. Allowing us to test and debug traffic from the remote cluster as if it was on our local machine, in other words, giving us feedback from our local service to see how it will perform when running in the remote cluster.

Step 5: (Bonus) Clone your Pod’s environment variables to your debug session

In some scenarios, a service can inherit environment variables from the cluster or configuration. In cases like this, cloning the remote deployments environment variables to our local service will be necessary to ensure closer parity between the two environments. To do this we will leverage Telepresence to copy the environment variables from the remote service and save them to a .env file. Then we will tell VScode to add these variables to the debug environment.

To let VScode know that we want to include an environment file, we need to create a launch configuration specifying where the file is going to be.

Stop your last debugging session if it is still running.
Click the Gear Icon in the top right corner to open the launch.json file.
From here click the Add Configuration button.
Select Go: Launch File from the dropdown menu.
Add the line "envFile": "${workspaceFolder}/go-debug.env"

    {
     "name": "Launch with env file",
     "type": "go",
     "request": "launch",
     "mode": "debug",
     "program": "${file}",
     "envFile": "${workspaceFolder}/go-debug.env"
    }

Now that VScode knows where the file is going to be located let’s generate the file with Telepresence. This time run the intercept from the terminal window within VScode to ensure the file is created in the project directory.

$ telepresence intercept dataprocessingservice --port=3000 --env-file=go-debug.env

Finally, start the debug session. Click the debug icon on the right side and click the launch session button.

Now your local process is running with all the same variables as if it was running in the remote cluster.

You can also locally access volumes mounted into your remote Services. This is useful if you are storing configuration, tokens, or other state required for the proper execution of the service. We’ll cover this in more detail in a future tutorial.

Learn More About Telepresence

Today, we’ve learned how to use Telepresence to easily debug a Go microservice running in Kubernetes. Now, instead of trying to mock out dependencies or fiddle around with remote debugging, we can iterate quickly with an instant feedback loop when locally debugging using our favorite IDE and tools.

If you want to learn more about Telepresence, check out the following resources:

Read the docs
Watch the demo video
Read more about Intercepts
Learn about Preview URLs for easy collaboration with teammates
Join our Slack channel to connect with the Telepresence community

Go & Kubernetes: Rapidly Developing Golang Microservices

Peter ONeill — Wed, 03 Mar 2021 19:06:55 +0000

Build a cloud development environment with Telepresence & Golang

Kubernetes is a container orchestration platform that enables its users to deploy and scale their microservice applications at any scale: from one service to thousands of services. Unleashing the power of Kubernetes is often more complicated than it may initially seem — the learning curve for application developers is particularly steep. Knowing what to do is just half the battle, then you have to choose the best tools to do the job. So how do Go developers create a development workflow on Kubernetes that is fast and effective?

Application developers face two unique challenges when trying to create productive development workflows on Kubernetes.

Most development workflows are optimized for local development, and Kubernetes applications are designed to be native to the cloud.
As Kubernetes applications evolve into complex microservice architectures, the development environments also become more complex as every microservice adds additional dependencies. These services quickly start to need more resources than are available in your typical local development environment.

In this tutorial, we’ll set up a development environment for Kubernetes and make a change to a Golang microservice. Normally to develop locally, we’d have to wait for a container build, push to registry and deploy to see the effect of our code change. Instead, we’ll use Telepresence to see the results of our change instantly.

Step 1: Deploy a Sample Microservices Application

For our example, we’ll make code changes to a Go service running between a resource-intensive Java service and a large datastore. We’ll start by deploying a sample microservice application consisting of 3 services:

VeryLargeJavaService: A memory-intensive service written in Java that generates the front-end graphics and web pages for our application
DataProcessingService: A Golang service that manages requests for information between the two services.
VeryLargeDataStore: A large datastore service that contains the sample data for our Edgey Corp store.

Note: We’ve called these VeryLarge services to emphasize the fact that your local environment may not have enough CPU and RAM, or you may just not want to pay for all that extra overhead for every developer.

In this architecture diagram, you’ll notice that requests from users are routed through an ingress controller to our services. For simplicity’s sake, we’ll skip the step of deploying an ingress controller in this tutorial. If you’re ready to use Telepresence in your own setup and need a simple way to set up an ingress controller, we recommend checking out the Ambassador Edge Stack which can be easily configured with the K8s Initializer.

Let’s deploy the sample application to your Kubernetes cluster:

    kubectl apply -f [https://raw.githubusercontent.com/datawire/edgey-corp-go/main/k8s-config/edgey-corp-web-app-no-mapping.yaml](https://raw.githubusercontent.com/datawire/edgey-corp-go/main/k8s-config/edgey-corp-web-app-no-mapping.yaml)

Step 2: Set up your local Go development environment

We’ll need a local development environment so that we can edit the DataProcessingService service. As you can see in the architecture diagram above, the DataProcessingService is dependent on both the VeryLargeJavaService and the VeryLargeDataStore, so in order to make a change to this service, we’ll have to interact with these other services as well. Let’s get started!

Clone the repository for this application from GitHub.

    git clone [https://github.com/datawire/edgey-corp-go.git](https://github.com/datawire/edgey-corp-go.git)

Change directories into the DataProcessingService

    cd edgey-corp-go/DataProcessingService

Start the Go server:

    go build main.go && ./main

See your service running!

    10:23:41 app | Welcome to the DataProcessingGoService!

In another terminal window, curl localhost:3000/color to see that the service is returning blue.

    $ curl localhost:3000/color

    “blue”

Step 3: Rapid Development with Telepresence

Instead of waiting for a container image to build, pushed to a repository, and deployed to our Kubernetes cluster we are going to use Telepresence, an open source Cloud Native Computing Foundation project. Telepresence creates a bidirectional network connection between your local development and the Kubernetes cluster to enable fast, efficient Kubernetes development.

Download Telepresence (~60MB):

    # Mac OS X
    sudo curl -fL [https://app.getambassador.io/download/tel2/darwin/amd64/latest/telepresence](https://app.getambassador.io/download/tel2/darwin/amd64/latest/telepresence) -o /usr/local/bin/telepresence

    #Linux
    sudo curl -fL https://app.getambassador.io/download/tel2/linux/amd64/latest/telepresence -o /usr/local/bin/telepresence

Make the binary executable

    $ sudo chmod a+x /usr/local/bin/telepresence

Test Telepresence by connecting to the remote cluster

    $ telepresence connect

Send a request to the Kubernetes API server:

    $ curl -ik [https://kubernetes.default.svc.cluster.local](https://kubernetes.default.svc.cluster.local)

    HTTP/1.1 401 Unauthorized
    Cache-Control: no-cache, private
    Content-Type: application/json
    Www-Authenticate: Basic realm="kubernetes-master"
    Date: Tue, 09 Feb 2021 23:21:51 GMT

Great! You’ve successfully configured Telepresence. Right now, Telepresence is intercepting the request you’re making to the Kubernetes API server, and routing over its direct connection to the cluster instead of over the Internet.

Step 4: Intercept Your Golang Service

An intercept is a routing rule for Telepresence. We can create an intercept to route traffic intended for the DataProcessingService in the cluster and instead route all of the traffic to the local version of the DataProcessingService running on port 3000.

Create the intercept

    telepresence intercept dataprocessingservice — port 3000

Access the application directly with Telepresence. Visit http://verylargejavaservice:8080. Again, Telepresence is intercepting requests from your browser and routing them directly to the Kubernetes cluster.
1. Now, we’ll make a code change. Open edgey-corp-go/DataProcessingService/main.go and change the value of the color variable from blue to orange. Save the file, stop the previous server instance and start it again with go build main.go && ./main.
2. Reload the page in your browser and see how the color has changed from blue to orange!

That’s it! With Telepresence we saw how quickly we can go from editing a local service to seeing how these changes will look when deployed with the larger application. When you compare it to our original process of building and deploying a container after every change, it’s very easy to see how much time you can save especially as we make more complex changes or run even larger services.

Learn More about Telepresence

Today, we’ve learned how to use Telepresence to rapidly iterate on a Golang microservice running in Kubernetes. Now, instead of waiting for slow local development processes, we can iterate quickly with an instant feedback loop and a productive cloud native development environment.

If you want to learn more about Telepresence, check out the following resources:

Read the docs
Watch the demo video
Read more about Intercepts
Learn about Preview URLs for easy collaboration with teammates
Join our Slack channel to connect with the Telepresence community

In our next tutorial, we’ll use Telepresence to set up a local Kubernetes development environment and then use Delve to set breakpoints and debug a broken service. To be notified when more tutorials are available, make sure to check out our website or follow us on Twitter.

Developing Python Applications on Kubernetes

Richard Li — Tue, 02 Mar 2021 21:29:59 +0000

Kubernetes has become the de-facto standard for running cloud applications. With Kubernetes, users can deploy and scale containerized applications at any scale: from one service to thousands of services. The power of Kubernetes is not free — the learning curve is particularly steep, especially for application developers. Knowing what to do is just half the battle, then you have to choose the best tools to do the job. So how do Python developers create a development workflow on Kubernetes that is fast and effective?

There are two unique challenges with creating productive development workflows on Kubernetes.

Most development workflows are optimized for local development, and Kubernetes applications are designed to be native to the cloud.
Most Kubernetes applications either start off or evolve into a microservices architecture. Thus, your development environment becomes more complex as every microservice adds additional dependencies to test code. And in turn, these services quickly become too resource-intensive and exceed the limits of your local machine.

In this tutorial, we’ll walk through how to set up a realistic development environment for Kubernetes. Typically, we’d have to wait for a container build, push to registry and deploy to see the impact of our change. Instead, we’ll use Telepresence and see the results of our change instantly.

Telepresence is an open source project that lets you run your microservice locally, while creating a bi-directional network connection to your Kubernetes cluster. This approach enables the microservice running locally to communicate to other microservices running in the cluster, and vice versa. Since you’re running the microservice locally, you’re able to benefit from any workflow or tool that you run locally.

Step 1: Deploy a Sample Microservices Application

VeryLargeJavaService A memory-intensive service written in Java that generates the front-end graphics and web pages for our application
DataProcessingService A Python service that manages requests for information between the two services.
VeryLargeDataStore A large datastore service that contains the sample data for our Edgey Corp store.

Let’s deploy the sample application to your Kubernetes cluster:

kubectl apply -f https://raw.githubusercontent.com/datawire/edgey-corp-python/master/k8s-config/edgey-corp-web-app-no-mapping.yaml

Note: This tutorial assumes you have access to a Kubernetes cluster with kubectl access. If you don’t, some options include MicroK8S and Docker Kubernetes.

Step 2: Set up your local Python development environment

Clone the repository for this application from GitHub:

git clone https://github.com/datawire/edgey-corp-python.git

Install the application dependencies with pip (you may need to type pip3 if you have Python 3 installed):

cd edgey-corp-python/DataProcessingService/
pip install flask requests

Run the application (you may need to type python3 if you have Python 3 installed):

python app.py

Test the application. In another terminal window, we’ll send a request to the service, which should return blue.

curl localhost:3000/color
blue

Step 3: Make Code Changes with Telepresence

To test a code change with Kubernetes, you typically need to build a container image, push the image to a repository, and deploy the Kubernetes cluster. This takes minutes.

Telepresence is an open source, Cloud-Native Computing Foundation project that solves exactly this problem. By creating a bidirectional network connection between your local development environment and the Kubernetes cluster, Telepresence enables fast, local development.

Download Telepresence (~60MB):

# Mac OS X
sudo curl -fL https://app.getambassador.io/download/tel2/darwin/amd64/latest/telepresence -o /usr/local/bin/telepresence
# Linux
sudo curl -fL https://app.getambassador.io/download/tel2/linux/amd64/latest/telepresence -o /usr/local/bin/telepresence

Make the binary executable:

sudo chmod a+x /usr/local/bin/telepresence

Test Telepresence by connecting to the remote Kubernetes cluster:

telepresence connect
```


4. Send a request to the Kubernetes API server:



```
curl -ik https://kubernetes.default.svc.cluster.local
HTTP/1.1 401 Unauthorized
Cache-Control: no-cache, private
Content-Type: application/json
Www-Authenticate: Basic realm="kubernetes-master"
Date: Tue, 09 Feb 2021 23:21:51 GMT
```



Congratulations! You’ve successfully configured Telepresence. Telepresence is intercepting the request you’re making to the Kubernetes API server, and routing over its direct connection to the cluster instead of over the Internet.

## Step 4: Set up an Intercept
An intercept is a routing rule for Telepresence. We can create an intercept that will route traffic intended for the `DataProcessingService` in the cluster and route all the traffic to the local version of the DataProcessingService running on port 3000.
1. Create the intercept:


```
telepresence intercept dataprocessingservice --port 3000
```


2. Access the application directly with Telepresence. In your browser, go to http://verylargejavaservice:8080. Again, Telepresence is intercepting requests from your browser and routing them directly to the Kubernetes cluster.
3. Now, let’s make a code change. Open `edgey-corp-python/DataProcessingService/app.py` and change `DEFAULT_COLOR` from `blue` to `orange`. Save the file.
4. Reload the page in your browser, and see how the color has changed from blue to orange.

That’s it! With Telepresence we saw how quickly we can go from editing a local service to seeing how these changes will look when deployed with the larger application. When you compare it to our original process of building and deploying a container after every change, it’s very easy to see how much time you can save especially as we make more complex changes or run even larger services.


## Learn More about Telepresence
Typically, developers at organizations adopting Kubernetes face challenges slow feedback loops from inefficient local development environments. Today, we’ve learned how to use Telepresence to set up fast, efficient development environments for Kubernetes and get back to the instant feedback loops you had with your legacy applications.

If you want to learn more about Telepresence, check out the following resources:
* Watch a [demo video](https://www.youtube.com/watch?v=W_a3aErN3NU), which shows more details on different features in Telepresence
* Check out the [Python Quickstart for Telepresence](http://docs/latest/telepresence/quick-start/qs-python/)
* Learn about [Preview URLs](https://www.getambassador.io/docs/pre-release/telepresence/howtos/preview-urls/#collaboration-with-preview-urls)  for easy collaboration with teammates
* [Join our Slack channel](https://d6e.co/slack) to connect with the Telepresence community

In our next tutorial, we’ll use Telepresence to set up a local Kubernetes development environment and then use Pycharm to set breakpoints and debug a broken service. To be notified when more tutorials are available, make sure to check out our [website](https://www.getambassador.io) or follow us on [Twitter](http://www.twitter.com/ambassadorlabs).

*This post was originally published on [Python Pandemonium](https://medium.com/python-pandemonium/developing-python-applications-on-kubernetes-75be68a3f0f9).*

SRE vs Platform Engineering

Richard Li — Tue, 16 Feb 2021 21:08:00 +0000

Over the past decade, engineering and technology organizations have converged on a common set of best practices for building and deploying cloud-native applications. These best practices include continuous delivery, containerization, and building observable systems.

At the same time, cloud-native organizations have radically changed how they’re organized, moving from large departments (development, QA, operations, release) to smaller, independent development teams. These application development teams are supported by two new functions: site reliability engineering and platform engineering. SRE and platform engineering are spiritual successor of traditional operations teams, and bring the discipline of software engineering to different aspects of operations.

Site Reliability Engineering and Platform Engineering

Platform engineering teams apply software engineering principles to accelerate software delivery. Platform engineers ensure application development teams are productive in all aspects of the software delivery lifecycle.

Site reliability engineering teams apply software engineering principles to improve reliability. Site reliability engineers minimize the frequency and impact of failures that can impact the overall reliability of a cloud application.

These two teams are frequently confused and the terms are sometimes used interchangeably. Indeed, some organizations consolidate SRE and platform engineering into the same function. This occurs because both roles apply a common set of principles:

Platform as product. These teams spend time understanding their internal customers, building roadmaps, having a planned release cadence, writing documentation, and doing all the things that go into a software product.
Self-service platforms. These teams build their platforms for internal use. In these platforms, best practices are encoded, so that the users of these platforms don’t need to worry about it -- they just push the button. In the Puppet Labs 2020 State of DevOps report, Puppet Labs found that High functioning DevOps organizations had more self-service infrastructure than low DevOps evolution organizations.
A constant focus on eliminating toil. As defined in the Google SRE book, toil is manual, repetitive, automatable, tactical work. The best SRE and platform teams identify toil, and work to eliminate it.

Platform Engineering

Platform engineers constantly examine the entire software development lifecycle from source to production. From this introspective process, they build a workflow that enables application developers to rapidly code and ship software. A basic workflow typically includes a source control system connected with a continuous integration system, along with a way to deploy artifacts into production.

As the number of application developers using the workflow grows, the needs of the platform evolves. Different teams of application developers need similar but different workflows, so self-service infrastructure becomes important. Common platform engineering targets for self-service include CI/CD, alerting, and deployment workflows.

In addition to self-service, education and collaboration become challenges. Platform engineers find they increasingly spend time educating application developers on best practices and how to best use the platform. Application developers also find that they depend on other teams of application developers, and look to the platform engineering team to give them the tools to collaborate productively with different teams.

Site Reliability Engineering

Site reliability engineers create and evolve systems to automatically run applications, reliably. The concept of site reliability engineering originated at Google, and is documented in detail in the Google SRE Book. Ben Treynor Sloss, the SVP at Google responsible for technical operations, described SRE as “what happens when you ask a software engineer to design an operations team.”

SREs define service level objectives and build systems to help services achieve these objectives. These systems evolve into a platform and workflow that encompass monitoring, incident management, eliminating single points of failure, failure mitigation, and more.

A key part of SRE culture is to treat every failure as a failure in the reliability system. Rigorous post-mortems are critical to identifying the root cause of the failure, and corrective actions are introduced into the automatic system to continue to improve reliability.

SRE and Platform Engineering at New Relic

One of us (Bjorn Freeman-Benson) managed the engineering organization at New Relic until 2015 as it grew from a handful of customers to tens of thousands of customers, all sending millions of requests per second into the cloud. New Relic had independent SRE and platform engineering teams that followed the general principles outlined above.

One of the reasons these teams were built separately was that the people who thrived in these roles differed. While both SREs and platform engineers need strong systems engineering skills in addition to classic programming skills, the roles dictate very different personality types. SREs tend to enjoy crisis management and get an adrenaline rush out of troubleshooting an outage. SRE managers thrive under intense pressure and are good at recruiting and managing similarly minded folks. On the other hand, platform engineers are more typical software engineers, preferring to work without interruption on big, complex problems. Platform engineering managers prefer to operate on a consistent cadence.

DevOps and GitOps

Over the past decade, DevOps has become a popular term to describe many of these practices. More recently, GitOps has also emerged as a popular term. How do DevOps and GitOps relate to platform and SRE teams?

Both DevOps and GitOps are a loosely codified set of principles of how to manage different aspects of infrastructure. The core principles of both of these philosophies -- automation, infrastructure as code, application of software engineering -- are very similar.

DevOps is a broad movement that began with a focus on eliminating traditional silos between development and operation. Over time, strategies such as infrastructure automation and engineering applications with operations in mind have gained widespread acceptance as ways better build highly reliable applications.

GitOps is an approach for application delivery. In GitOps, declarative configuration is used to codify the desired state of the application at any moment in time. This configuration is managed in a versioned source control system as the single source of truth. This ensures auditability, reproducibility, and consistency of configuration.

DevOps is a set of guiding principles for SRE, while GitOps is a set of guiding principles for platform engineering.

Unlocking application development productivity

Site reliability engineering and platform engineering are two functions that are critical to optimizing engineering organizations for building cloud-native applications. The SRE team works to deliver infrastructure for highly reliable applications, while the platform engineering team works to deliver infrastructure for rapid application development. Together, these two teams unlock the productivity of application development teams.

This story was originally published on the Ambassador blog.

Optimize the Kubernetes Developer Experience with Version 0

Richard Li — Fri, 10 Jul 2020 13:39:46 +0000

One of the core promises of microservices is development team autonomy, which should, in theory, translate into faster and better decision making. But sometimes, this theory doesn’t translate into reality.

Why is this the case?

There are a multitude of reasons for microservices not working well. Microservices, cloud-native, and Kubernetes are a new approach and culture shift, and there’s a lot of good ways and bad ways to approach the challenge.

One of the keys to success is enabling a consistent developer experience for each microservice from day 0, which is critical for unlocking team autonomy and development velocity.

Bootstrapping a Microservice

Creating microservices should be cheap and easy. This enables app dev teams to quickly build and ship new microservices to address specific business needs without being encumbered by preexisting code. At the same time, this agility and flexibility do come at a cost — applications become distributed, dynamic organisms that can be harder to develop, test, and debug.

Better Developer Experience == Better Customer Experience

In a recent Ambassador podcast, Gene Kim spoke about how a great developer experience is critical to delivering value to customers. By creating a great developer experience, developers can ship more code, which results in happier customers.

We’ve seen a similar trend in organizations that successfully adopt microservices: an emphasis on the developer experience. While it may not be a “strategic” initiative in the organization, usually there’s someone at the company who is passionate about creating a great developer workflow and is able to spend time working on continuously improving that developer workflow.

The Microservices Developer Experience

With a monolith, there’s a common application that is the target for the development workflow. With microservices, there is no longer a single common application. Every new microservice requires a developer workflow. Without due care, it’s easy to have a smorgasbord of microservices, all with poor developer workflows. In this situation, velocity actually decreases since microservices can’t be easily and rapidly shipped. This defeats the entire rationale for adopting microservices in the first place, and development slows.

At the same time, microservices presents an opportunity for improving the developer experience. By optimizing the developer experience of each microservice, teams can build the best possible developer experience for the team (and not the organization), and continue to optimize that experience as the application and team evolve.

Developer Experience, Defined

A developer experience is the workflow a developer uses to develop, test, deploy, and release software. The developer experience typically consists of both an inner dev loop and an outer dev loop. The inner dev loop is a single developer workflow. A single developer should be able to set up and use an efficient inner dev loop to code and test changes quickly. The inner dev loop is typically used for pre-commit changes. The outer dev loop is a shared developer workflow that is orchestrated by a continuous integration system. The outer dev loop is used for post-commit changes and includes automated builds, tests, and deploys.

Engineering a good inner and outer dev loop is key to a great developer experience and unlocking the potential of microservices. So how can an engineer help in building a great developer experience?

The Version 0 Strategy

The version 0 strategy involves shipping an end-to-end development and deployment workflow as the first milestone — before any features are coded. A good test of a version 0 milestone is if a developer on a different team is able to independently code, test, and release a change to the microservice without consulting the original team. This implies a version 0 has a development environment, a deployment workflow, and documentation that explains how to get started and ship. With a version 0 in place, the microservices team then begins with feature development, knowing that their ability to rapidly iterate and ship is already in place.

The Version 0 approach works well for a number of reasons:

The codebase is very simple, so there is no reverse engineering of obscure dependencies, monkey patches, or any other gremlins to get a working environment
With no features, there is less pressure from external parties who want to implement changes and adjustments to the roadmap
A great developer experience accrues benefits over time, so Version 0 maximizes the payback period
Most importantly, version 0 sets the tone for the microservice, which is that developer experience is important!

Version 0 for Engineers

Any engineer can adopt the version 0 practice (and should!). A development team should have full autonomy over a microservice, which includes the development timeline and workflow! So starting with a Version 0 will help the team rapidly bootstrap the microservice.

Version 0 for Managers

Managers can support Version 0 across the organization by asking engineering teams that are creating new microservices to start with a Version 0. As engineering organizations grow, the organization could choose to assign platform engineers focused on development workflows. These platform engineers should not implement Version 0, but instead provide tools, templates, and best practices to the microservice teams on how best to build a version 0. The Netflix engineering team adopted this approach to developer empowerment.

Summary

Every engineer has felt the pain of a bad developer workflow. A trivial one-line fix takes a half-day to complete. Microservices can exacerbate this problem. The Version 0 strategy is a simple but powerful strategy that will help integrate developer experience into your organization’s development workflow.

This article was originally published on the Ambassador blog.