Authorizing endpoints of external apps in k8s

Marcin Janiak — Mon, 09 Dec 2024 15:05:00 +0000

Implementing authorization at the code level is often straightforward.
Similarly, exposing external tools with built-in support for authentication (e.g Grafana) can be handled with relative ease.

The real challenge arises when dealing with simple tools or services that lack native support for authentication or authorization, requiring external mechanisms to ensure secure access.

One approach to managing this is by restricting access through a VPN or securing it with IP address ranges, which remains one of the most reliable methods.

However, I recently explored an alternative: reusing Keycloak, one of the oauth providers which we were already using, to verify access to specific endpoints.

The tool that might work for this is OAuth2 Proxy, which supports a wide range of OAuth providers such as Facebook, Google, GitLab, and Azure.

The additional requirement was to group these tools under a single URL, such as tools.example.com/tool-name, to streamline access and management.

oauth2-proxy basic setup:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
        - args:
            - --provider=keycloak-oidc
            - --oidc-issuer-url=https://{ISSUER_HOST}/realms/{REALM_NAME}
            - --code-challenge-method=S256
            - --allowed-role={ROLE}
            - --email-domain=*
            - --upstream=file:///dev/null
            - --http-address=0.0.0.0:4180
            - --client-id={CLIENT_ID}
            - --client-secret={CLIENT_SECRET}
            - --session-cookie-minimal
          env:
            - name: OAUTH2_PROXY_COOKIE_SECRET
              value: {COOKIE_SECRET}
          image: quay.io/oauth2-proxy/oauth2-proxy:latest
          imagePullPolicy: Always
          name: oauth2-proxy
          ports:
            - containerPort: 4180
              protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: default
spec:
  ports:
    - name: http
      port: 4180
      protocol: TCP
      targetPort: 4180
  selector:
    k8s-app: oauth2-proxy

Ingress configuration (assuming you already have a TLS secret named tools-example-com-tls)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tools
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 15m
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - "tools.example.com
      secretName: tools-example-com-tls
  rules:
      - host: "tools.example.com"
        http:
          paths:
                - path: /oauth2
                  pathType: Prefix
                  backend:
                    service:
                      name: oauth2-proxy
                      port:
                        number: 4180

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tools-external-auth-oauth2
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 15m
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - "tools.example.com"
      secretName: tools-example-com-tls
  rules:
      - host: "tools.example.com"
        http:
          paths:
            - path: /certificates
              pathType: Prefix
              backend:
                service:
                  name: cert-checker
                  port:
                    number: 8081

In our case, one of the tools was a dashboard for a certificate checker.

Extra bonus:
Additionally, our solution included multiple stateful services deployed at addresses following the pattern s1-app, s2-app, and so on, with each respective service exposing its own instance of tools, in our case, through ASP.NET.

For example, we might want to expose the relevant quartz-ui instances at addresses like tools.example.com/s1/quartz and tools.example.com/s2/quartz.

Given this requirement, the Ingress resource tools-external-auth-oauth2 consists of multiple paths, such as:

            - path: /s[0-9]+/quartz
              pathType: ImplementationSpecific
              backend:
                service:
                  name: tools-nginx-reverse-proxy
                  port:
                    number: 80

The nginx reverse proxy configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tools-nginx-reverse-proxy-config
  namespace: default
data:
  nginx.conf: |
    events {
      worker_connections 1024; 
    }
    http {
      resolver 10.3.0.10 valid=10s; #KubeDNS or CoreDNS internal ip
      server {
        listen 80;
        location ~ ^/s([0-9]+)/quartz{
          set $service_name "s$1-app.default.svc.cluster.local";
          rewrite ^/s([0-9]+)/quartz$ /quartz break;
          proxy_pass http://$service_name:80;

          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_http_version 1.1;
          proxy_set_header Connection "";

          proxy_read_timeout 90;
          proxy_send_timeout 90;
        }

        location ~ ^/s([0-9]+)/quartz/(.*\.(css|js|png|jpg|jpeg|gif|svg|woff2|woff|ttf))$ {
            set $service_name "s$1-app.default.svc.cluster.local";
            rewrite ^/s([0-9]+)/quartz/(.*)$ /$2 break;
            proxy_pass http://$service_name:80;
        }

        location / {
            return 404;
        }
      }
    }

---

apiVersion: apps/v1
kind: Deployment
metadata: 
  name: tools-nginx-reverse-proxy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tools-nginx-reverse-proxy
  template:
    metadata:
      labels:
        app: tools-nginx-reverse-proxy
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
      volumes:
        - name: nginx-config
          configMap:
            name: tools-nginx-reverse-proxy-config


---

apiVersion: v1
kind: Service
metadata:
  name: tools-nginx-reverse-proxy
  namespace: default
spec:
  selector:
    app: tools-nginx-reverse-proxy
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Note:
The cover image was (obviously) created by bing ai.

Useful Resources:
https://oauth2-proxy.github.io/oauth2-proxy/
https://www.keycloak.org
https://github.com/mogensen/cert-checker
https://github.com/guryanovev/CrystalQuartz

Forem: Marcin Janiak

Authorizing endpoints of external apps in k8s