How we connect to Kubernetes Pods from GitHub Actions

Richard Simpson — Mon, 06 Apr 2020 18:15:51 +0000

Ever wanted to connect to some Pod or Service running a Kubernetes cluster, but don't want or can't setup an Ingress or Load Balancer to connect to it?

Only want to connect to it for some local workflow like GitHub Actions instead of opening it wider internet using a tool like inlets?

If you answered yes to both of these questions, then you've got a few options available to you:

Setup a secure VPN within your network. If you're using a hosted service, there's likely a VPN option available. If not, there's a new and growing ecosystem of VPN and VPN-like options that may suit your purposes, enabled by advances like WireGuard. However this option still has a lot of friction and might be overkill.
Create something like a jump server to access your internal cluster services.
Finally, if your Kubernetes API is accessible (which is many managed provider's default), you already have access to a great proxy tool in the form of kubectl port-forward.

This article is all about setting up that third option.

Creating a Service Account

If you're connecting in a non-interactive fashion, through something like a CI/CD workflow, you'll likely need to setup a way for you workflow to authentication with your cluster. A "service account" if you will. Luckily, Kubernetes has the built-in concept of a ServiceAccount that is normally used to identify Pods and other cluster-native entities. These ServiceAccount resources can be used to authenticate external tools too, as I'll show you.

Creating a ServiceAccount is pretty simple, but there are a few things you should consider when setting one up. Firstly, are you trying to only access a specific pod? If so, you'll want to create a ServiceAccount in that pod's namespace. Additionally, you'll want to make sure that account only has the permissions necessary to create a port-forward on that pod unless you're planning to use this ServiceAccount for more than just port-forwarding.

In this example, we'll say you're trying to access the Pod my-pod on namespace default.

First, let's create the ServiceAccount we'll be using:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-pod-port-forward
  namespace: default

This one's pretty straight forward. We want to make make a ServiceAccount with the name my-pod-port-forward in the namespace default. Note you can name it whatever you want, but it should descriptive and make it clear what it's used for or with.

Next, let's create the Role with the appropriate permissions that we'll use with this ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-pod-port-forward
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  resourceNames: ["my-pod"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods/portforward"]
  resourceNames: ["my-pod"]
  verbs: ["create"]

There's quite a bit here, so I'll break it down without getting too in the nitty-gritty. If you're interested in a deeper dive, I'd check out the official docs.

First, we're creating an RBAC Role with the name my-pod-port-forward in the namespace default. The name can again be anything you like, but it should be descriptive. Then, we're giving this Role three sets of permission:

The ability to access get our specify pod.
The ability to create a port-forward for that specific pod.

This is the barest set of permissions necessary to port-forward to a pod.

Note: If you want to connect to a Service or Deployment, it gets a bit more complicated. I'll add an example of what's necessary at the end of the article.

Finally, we need to bind our new Role to the ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-pod-port-forward
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: my-pod-port-forward
subjects:
- kind: ServiceAccount
  name: my-pod-port-forward
  namespace: default

This one should be pretty straight forward and follows the same naming theme as before.

With all those applied, you now have a ServiceAccount you can use! But...how do you go from ServiceAccount to kubectl?

Using the Service Account

Whenever you create a service account, it's automatically issued a Secret that's used to authenticate with the cluster's API.

You can get the Secret associated with a ServiceAccount by running:

TOKENNAME=`kubectl -n default get serviceaccount/my-pod-port-forward -o jsonpath='{.secrets[0].name}'`

In that Secret is a lot of important information, but the thing we want is the token. To get that, you can run:

kubectl -n default get secret/$TOKENNAME -o jsonpath='{.data.token}' | base64 -d

If you've ever worked with JWTs before, you'll likely recognize this. So how do we use this token? To use it, we need to create a kube config file with the appropriate context and user.

Our template:

apiVersion: v1
clusters:
- cluster:
    server: <api-server-url>
    certificate-authority-data: <api-server-ca-data>
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: service-account
  name: target-cluster
current-context: target-cluster
kind: Config
preferences: {}
users:
- name: service-account
  user:
    token: <token>

You can use your normal means of getting the cluster's API URL and certificate authority, but chances are you're already connected to the cluster you want to target. As such, you can pull these values from your own kube config! For example, to get the various clusters in you config, you can run:

grep '\- cluster:' -A 3 ${HOME}/.kube/config

Set those values and replace <token> with the token you got from those previous files, and you've got your self a service account kube config! To use it, let's say you saved it to something like ~/.kube/my-pod-port-forward.yaml.

The magic command to create the port-forward is:

kubectl port-forward pod/my-pod 80:80 --kubeconfig ~/.kube/my-pod-port-forward.yaml

With that, you should get a message like this:

Forwarding from 127.0.0.1:80 -> 80
Forwarding from [::1]:80 -> 80

And that's it! You're now able to connect to your pod locally!

GitHub Actions (and more)

Now that we've got a service account setup and a kube config made, we can get up and running in Actions! This is pretty simple, but there are a few quirks I'll call out.

First, we want to drop our kube config into our action so that we can authenticate kubectl. What I did was base64 encoded the kube config we created before, and then dropped into GitHub Secrets as KubeConfig. We're base64'ing it to make it easier to consume in the action as you'll see.

cat ~/.kube/my-pod-port-forward.yaml | base64 -w 0 # No wrap so it's all one line

Now that we have it stored in secret, we can access it from our workflow and use it for kubectl.

name: CI
on:
  push:
    branches: [ master ]
jobs:
  build:
    env:
        # Here, we're choosing to place it in the workspace folder
        # in a folder called `.kube` to keep things organized.
        # Note: You could just do KUBECONFIG if you're planning to use
        # this for all other `kubectl` actions in this workflow
        POD_KUBECONFIG: '${{ github.workspace }}/.kube/pod-kubeconfig'
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    # Here, we're creating the parent directory and writing out our decoded
    # kubeconfig to the location we stated above.
    - run: |
        mkdir -p '${{ github.workspace }}/.kube' \
          && echo '${{ secrets.KubeConfig}}' | base64 -d > $POD_KUBECONFIG

    # Finally, let's try using it. If you used `KUBECONFIG`, can can remove
    # the `--kubeconfig $POD_KUBECONFIG` as kubectl will automatically use it.
    - run: 'kubectl version --kubeconfig $POD_KUBECONFIG'

If you drop that in you .github/workflows folder and push it, should see the action run and final step print out both the client kubectl and the target cluster's version information!

Now, we need to start the port-forward. Here, we're going to use bash's built-in support for jobs to run the port-forward in the background:

    - run: 'kubectl port-forward pod/my-pod 80:80 --kubeconfig $POD_KUBECONFIG &'

With that, it should be accessible locally! If you want to try it out, you can add a step like this to see it in action:

    - run: curl http://localhost:80

And that's it! Hopefully, this works well for you, and just leave a comment below if you have any questions or run into any issues.

Credits

Alex Ellis who helped give me through the initial idea of connecting to a cluster through port-forwarding and got me thinking about the options enabled by that. (Also the man behind OpenFaaS and Inlets which you should checkout!)
Daniel Albuschat who has an excellent deeper dive into k8s auth and service accounts, including how to re-issue tokens:

Kubernetes: Certificates, Tokens, Authentication and Service Accounts

Daniel Albuschat ・ May 19 '19 ・ 9 min read

#kubernetes
All the Kubernetes Docs contributors and poor, fellow lost souls on Stack Overflow who helped me generally understand many of the concepts in this article.

Additional Notes

If you want to access a Service or Deployment instead of a specific Pod, you'll need a slightly different Role configuration:

Permissions for a Deployment.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-pod-port-forward
  namespace: default
rules:
- apiGroups: ["apps/v1"]
  resources: ["deployments"]
  resourceNames: ["<deployment name>"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/portforward"]
  verbs: ["create"]

This will be the same Role as earlier, but with the addition of get for deployments, and the removal of resourceName from the pods and pods/portforward since you wouldn't know the pod name ahead of time.

Permissions for a Service.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-pod-port-forward
  namespace: default
rules:
- apiGroups: [""]
  resources: ["services"]
  resourceNames: ["<service name>"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/portforward"]
  verbs: ["create"]

This will be the same Role as earlier, but with the addition of get for services, and the removal of resourceName from the pods and pods/portforward since you wouldn't know the pod name ahead of time.

Forem: Richard Simpson