Forem: Glenn Bostoen

The Unglamorous Secret to Claude Code Productivity

Glenn Bostoen — Mon, 26 Jan 2026 22:18:59 +0000

After months of hands-on experience with Claude Code (from personal experiments to production deployments and even onboarding non-technical team members) I've distilled my learnings into a practical guide for anyone looking to move beyond basic usage into truly using this tool.

The productivity divide is real (but misunderstood)

There's an emerging divide among Claude Code users: those who've figured out how to make it transformative, and those still using it like a slightly smarter Stack Overflow. The difference isn't secret prompts or hidden features. It's a fundamental shift in how you think about the tool.

The key insight: effective use of Claude Code is about delegation, not dictation. You need to break down large tasks, define projects clearly, distribute work, and adapt your approach as you learn the model's strengths and limits.

What doesn't work: treating it as a magic oracle that produces perfect code from vague descriptions.

Tool access changes everything

The single biggest upgrade to my Claude Code workflow was giving it access to the actual tools I use as a developer. This transforms it from "smart autocomplete with chat" into an agent that operates within your development environment.

When Claude Code can:

Search your codebase and understand file structures
Run tests and interpret results
Check code against linters and formatters
Query your documentation and API references
Understand git context and branch history

...the results align dramatically better with expectations. The model isn't guessing at your architecture. It's reading it.

Crucially, tools let it validate its own work. Don't use an LLM to check syntax; use a linter. Don't ask it to verify types; run the type checker. Don't have it eyeball whether tests pass; execute them. The model is good at many things, but deterministic validation isn't one of them. Every tool you give it is one less thing it can hallucinate about.

Where this breaks down: The model operates with a mental model of "if it passes locally, it's correct." This is often true—until it isn't. Environment config, data volume, service dependencies, and infrastructure quirks don't exist in your local checkout. No amount of tooling closes this gap entirely. Knowing which changes need extra scrutiny before production is still a human skill.

This principle led directly to integrating Claude Code into our CI/CD pipelines, giving it real repository access, the ability to respond to MR comments, and run automated reviews. The key was treating it as stateless: inject context via pipeline variables, let it do its work, done. No session management complexity.

CLAUDE.md: the instructions that matter

There's an underground circulation of CLAUDE.md configurations and instruction libraries. After experimenting with many approaches, here's what I've found actually moves the needle:

What works:

Project-specific context (tech stack, patterns used, naming conventions)
Explicit tool permissions and boundaries
Clear success criteria for common tasks

What doesn't work:

Overly prescriptive step-by-step instructions (the model handles decomposition better than most instructions)
Personality instructions (they don't meaningfully improve output)
Excessive safety guardrails beyond what's already built in

The communities sharing instruction files are useful, but deep understanding comes from hands-on experimentation. What works for a React project won't work for a Python backend, and copying someone else's CLAUDE.md without understanding why it works is cargo culting.

Context is everything

This might be the most important lesson: Claude Code needs the full picture before it starts working. Without complete context, you'll see duplicated utilities, inconsistent patterns, and solutions that ignore existing infrastructure.

Before starting any task, make sure it understands:

What already exists in the codebase that's relevant
The patterns and conventions already in use
What you're trying to achieve and why

Between tasks, clear the slate. Residual context from previous work will bleed into new tasks: correlating unrelated problems, carrying forward assumptions that no longer apply, or "fixing" things that weren't broken. A fresh conversation for each distinct piece of work keeps outputs focused.

The pattern I've landed on: front-load context aggressively at the start of each session, then let it work. Drip-feeding requirements mid-task leads to patches on patches. Complete context upfront leads to coherent solutions.

On-distribution choices compound

There's a concept in AI of "on distribution" vs "off distribution": whether the model already knows how to do something well or needs to be taught. This applies directly to tech stack choices.

When building AI-augmented workflows, choosing technologies Claude is already good at (TypeScript, React, Python, standard tooling) means less fighting and more flow. An exotic or niche stack isn't impossible, but you're spending context teaching the model things it could've known for free.

This isn't about limiting creativity. It's about recognizing that LLM productivity gains come from working with the model's strengths, not proving you can make it work despite them.

Onboarding non-engineers: harder than expected

The promise of Claude Code ("anyone can code now") made me curious. I ran an experiment onboarding a product designer to Claude Code for direct iteration in our codebase. The results were instructive, but not in the way I expected.

What we discovered:

Setup is the real barrier. SSH keys, Brew, Docker/Colima architecture, Git config, branch management, package dependencies. These took ~1.5 hours with two engineers helping.
The comment that stuck with me: "Now I get why project setup takes two weeks."
Once setup was done, simple changes worked well. Complex debugging was still a blocker.

The real insight: This experiment made me realize how much implicit knowledge developers carry. We forget that "just run the dev server" assumes you know what a dev server is, that ports exist, that something else might be using port 3000. Claude Code lowers the barrier for writing code, but all the surrounding knowledge (environments, dependencies, version conflicts, debugging strategies) is still required. The tool doesn't eliminate the need for developer knowledge; it just shifts where that knowledge matters most.

The sustainability question: If someone uses this weekly, they build muscle memory and it pays off. If it's monthly, they forget workflows and need hand-holding each time.

My current thinking: There's a frequency threshold, around every 2-3 days, where this becomes valuable. Below that, a prototyping tool in between might be more practical.

Pipeline-first development: the mindset shift

When AI can generate code faster than humans can review it, your pipeline becomes your primary quality gate, not your colleagues' availability.

This is the mindset shift: a proper automated pipeline matters more than ever. Tests, linting, type checks, security scans. These can't be "nice to haves" anymore. They're the foundation that makes AI-assisted development viable. If your CI is flaky or your test coverage is patchy, "CI green" means nothing, and you're back to manual review bottlenecks that can't keep pace.

The old model: Reviews as permission gates, approval required before merge, long-lived branches waiting for sign-off.

The new model:

Pipeline does the hard work automatically, no human in the loop for mechanical checks
Reviews become sanity checks for architectural decisions and obviously wrong logic
Auto-merge for low-risk changes when CI is green
Fix forward when things slip through. The cost of a bad merge is lower than MRs sitting in queues

Invest in your pipeline first. The teams getting the most from Claude Code aren't the ones with the cleverest prompts. They're the ones whose CI actually catches problems before humans need to look.

The bottom line

Claude Code isn't a replacement for engineering skill. It's an amplifier. Strong project management and software engineering fundamentals matter more, not less, when AI is generating code at speed.

The competitive advantage isn't secret prompt libraries. It's the broader skills in task decomposition, workflow orchestration, and knowing when the AI's suggestion is brilliant vs. confidently wrong.

Start with giving it access to your actual tools. Integrate it into your existing workflows rather than building parallel ones. And remember: clear context in, quality output out.

original post: https://gbostoen.dev/blog/claude-code-learnings/

The Surprising Simplicity of Temporal Worker Pools on Cloud Run

Glenn Bostoen — Mon, 26 Jan 2026 22:17:17 +0000

If you've ever spent an afternoon debugging indentation errors in Google Workflows YAML only to discover the real problem was a cryptic ${} expression, you'll understand why we made the switch to Temporal. What we didn't expect was just how simple the deployment would be.

The problem we were solving

Our workflow orchestration setup had all the classic symptoms of YAML-based configuration debt:

Verbose definitions: Representing simple workflows as graphs required dozens of steps and connector definitions
Cold start delays: Every workflow step triggered a Cloud Run job, adding 35-70 seconds of spin-up time per execution
No local testing: Changes required deployment to validate. The feedback loop was measured in deploys, not seconds
Split infrastructure: Application code lived in one place, workflow definitions in another, and every change risked breaking both

The worst part? We couldn't even test locally. Every iteration meant committing, deploying, and hoping.

Why Temporal changes everything

Temporal flips the model on its head. Instead of declarative YAML that describes what should happen, you write actual code that describes how it happens:

@workflow.defn
class IndexingWorkflow:
    @workflow.run
    async def run(self, workspace_id: str):
        # This is just Python. Full IDE support.
        # Local debugging works out of the box.
        connections = await workflow.execute_activity(
            fetch_connections,
            workspace_id,
            start_to_close_timeout=timedelta(minutes=5),
        )

        for connection in connections:
            await workflow.execute_activity(
                index_connection,
                connection.id,
                start_to_close_timeout=timedelta(minutes=30),
            )

That's it. No separate YAML file. No mysterious DSL. Just code that your IDE understands, your debugger can step through, and your tests can cover. And it runs on your infrastructure. Temporal handles orchestration and state, but the actual work happens on compute you control.

The pull-based architecture

Understanding why Temporal workers are different from Cloud Run jobs unlocks the simplicity.

Cloud Run Jobs are push-based and ephemeral:

Something triggers them → they spin up → execute → shut down
Each invocation pays the cold start tax
No shared state between executions

Temporal Workers are pull-based and persistent:

Workers run on your infrastructure, not Temporal's
They maintain a long-polling connection to Temporal for orchestration
They pull tasks when they have capacity
Workers stay warm, eliminating cold starts

This pull-based model is exactly what Google designed Cloud Run Worker Pools for. It's a resource type announced at Google Cloud Next '25 specifically for continuous, non-HTTP, pull-based background processing.

Enter Cloud Run worker pools

Worker pools solve a real problem for Temporal deployments. Unlike Cloud Run Services (designed for HTTP workloads) or Jobs (designed for batch tasks), Worker Pools are purpose-built for exactly what Temporal workers do: continuously pull tasks from a queue.

Why Worker Pools are perfect for Temporal:

No HTTP endpoint required: Workers just poll Temporal. No need to expose ports or manage health check endpoints
Lower total cost: No load balancer, no HTTP endpoint overhead, just compute
Reduced attack surface: No public URL means fewer security concerns
Instance splitting: Deploy canary releases by allocating percentages of instances to different revisions

The deployment is even simpler than Services:

gcloud beta run worker-pools deploy worker \
  --image gcr.io/my-project/worker:latest \
  --region europe-west1

Or with Terraform:

resource "google_cloud_run_v2_worker_pool" "worker" {
  name         = "temporal-worker"
  location     = "europe-west1"
  provider     = google-beta
  launch_stage = "BETA"

  scaling {
    scaling_mode       = "AUTOMATIC"
    min_instance_count = 1
    max_instance_count = 5
  }

  template {
    containers {
      image = "gcr.io/my-project/worker:latest"
      resources {
        limits = {
          cpu    = "1"
          memory = "1Gi"
        }
      }
    }
  }
}

No minScale hacks. No unused HTTP endpoints. Just a container that runs your worker code.

Note: Worker Pools are currently in public preview. For production workloads, you can still use Cloud Run Services with --min-instances 1. The architecture is identical, just with a bit more overhead.

The deployment is just another container

Here's the mental shift: you're not deploying workflows anymore. You're deploying an application that happens to execute workflows.

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "-m", "worker"]

Your worker code:

# worker.py
import asyncio
import os

from temporalio.client import Client
from temporalio.worker import Worker

from workflows import (
    IndexingWorkflow,
    GoogleDriveWorkflow,
)
from activities import (
    fetch_connections,
    index_connection,
    sync_drive,
)

async def main():
    client = await Client.connect(
        "namespace.tmprl.cloud:7233",
        api_key=os.environ["TEMPORAL_API_KEY"],
    )

    worker = Worker(
        client,
        task_queue="indexing-queue",
        workflows=[
            IndexingWorkflow,
            GoogleDriveWorkflow,
        ],
        activities=[
            fetch_connections,
            index_connection,
            sync_drive,
        ],
    )

    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())

Deploy (with your API key stored in Secret Manager):

gcloud beta run worker-pools deploy worker \
  --image gcr.io/my-project/worker:latest \
  --min-instances 1 \
  --max-instances 5 \
  --memory 1Gi \
  --cpu 1 \
  --region europe-west1 \
  --set-secrets TEMPORAL_API_KEY=temporal-api-key:latest

That's the entire deployment. No Terraform for workflow definitions. No separate infrastructure repo. Just your application container with workflow logic baked in.

Cost reality check

Let's be honest about the trade-offs:

Before (Google Workflows + Cloud Run Jobs)

Cloud Workflows: ~$2-3/month
Cloud Run job invocations: ~$35/month
Cold start compute waste: ~$40-50/month
Total: ~$80-90/month

After (Temporal Cloud + Cloud Run Worker Pools)

Temporal Cloud starter: ~€100/month (orchestration and state only)
Worker Pool (2 instances, always-on): ~$18-24/month (your compute, no load balancer or endpoint overhead)
Total: ~$120-140/month

Yes, it costs more. But here's what you get:

Developer time saved: 2-3 hours/month not fighting YAML
Execution speed: 73-78% faster workflows (no cold starts)
Local testing: Full workflow debugging before deployment
Real observability: See workflow graphs, execution history, parent-child relationships in real-time

At a loaded developer cost of €80/hour, the ROI turns positive immediately.

What simplicity actually looks like

Kill a running workflow mid-execution. Restart the worker. The workflow resumes exactly where it left off.

That's durability you'd have to build yourself with Google Workflows: tracking state in Cloud Storage, implementing retries, handling partial failures. With Temporal, it's the default behavior.

Debug a failing activity with your IDE's debugger. Set breakpoints. Inspect state. Validate fixes locally before deploying.

This is what simplicity means: removing the gap between "I think this will work" and "I know this works."

Making the switch

The migration path isn't all-or-nothing:

Spike it: Implement one workflow in Temporal, run both systems in parallel for a week
Measure: Compare execution times, reliability, developer experience
Dark launch: Run Temporal workflows in production, keep Google Workflows as fallback
Gradual rollout: 10% → 50% → 100% with rollback ready

We kept Google Workflows YAML in git history (never delete, just remove from deployment) and maintained the rollback capability for 30 days. We never needed it.

The simplicity of Temporal isn't in having fewer moving parts. It's in having the right moving parts. A persistent worker pool on Cloud Run, code-native workflow definitions, and a managed orchestration layer that handles the hard stuff.

No more YAML debugging. No more cold start delays. No more "deploy to test" cycles.

Just workflows that work.

original post: https://gbostoen.dev/blog/temporal-cloud-run/

Build containers without using Docker

Glenn Bostoen — Fri, 01 Apr 2022 14:44:38 +0000

In this post, I will quickly show you multiple ways of building your containers without having to worry about the details of Dockerfiles.

Why

Most developers that have set up a pipeline that uses Docker already are familiar with the details that come with building your own Docker image, but for someone that is new, this can be a bit overwhelming. Things that you have to keep into account like:

Advanced caching
Apply best practices regarding security
Multi-stage setup to have a lightweight final image
Keeping your base images up to date
Reproducibility of an image

If this is all completely new to you, your best bet to start with containers is by learning Buildpacks as the first step.

What

Buildpacks was started by Heroku back in 2011 and is now officially part of the Cloud Native Computing Foundation and recently went from sandbox to incubation (see post). It allows you to automatically detect based on source code which build tooling you need and it transforms your code into a ready to use image.

Gitlab

On Gitlab you have 2 approaches currently used:

docker-in-docker
Kaniko build executor

Both use a Dockerfile as a starting point, whereas Kaninko doesn't even need a docker daemon to run. This is why this is the most preferred (and advanced) approach in building containers. Below I will show different approaches in building your application starting with Auto DevOps. You can also find these approaches on an example Git repository fastify-example-buildpacks

Buildpacks with Auto DevOps

On Gitlab you can use Buildpacks out of the box with "Auto DevOps". "Auto DevOps" automatically uses a Herokuish builds as a default to build your project and publish it to your Gitlab registry. You can override this by making use of an AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED environment variable on your GitLab agent. This allows you to make use of Buildpacks which is part of the Cloud Native Foundation. For now, this is still in Beta on Gitlab, but they expect this to become a replacement for the current Herokuish builds.

include:
  - template: Auto-DevOps.gitlab-ci.yml

variables:
  AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED: 'true'

Buildpacks with Pack CLI

If you want to have more control on which Buildpacks that are used, you can use Pack CLI which needs to be installed on your agent. This is easily done with the following example:

docker:
  stage: build
  services:
    - docker:19.03.12-dind
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ''
  image: docker:19.03.12
  before_script:
    - wget https://github.com/buildpacks/pack/releases/download/v0.13.1/pack-v0.13.1-linux.tgz
    - tar xvf pack-v0.13.1-linux.tgz
    - rm pack-v0.13.1-linux.tgz
  script:
    - ./pack build $IMAGE_NAME --builder gcr.io/buildpacks/builder

This explicitly uses the Buildpacks provided by Google. You can find more information about these on Github

Buildpacks with Google Cloud Platform

You can now even deploy your app directly to cloud Run. The only command you need is the following:

gcloud beta run deploy --source=[DIRECTORY] [CLOUD_RUN_SERVICE]

Downsides

Custom build scripts are not supported so you still need to compile your typescript for example to allow your application to be packages as an image. Additionally, certain native dependencies could be an argument to start from your base images in the first place which makes the use of Buildpacks a bit obsolete. Last the images that are created are bigger because they serve a wider variety of use cases that could not be needed by your application.

Summary

Start with buidpacks and if you need to customize your container you can opt-out of the default image created for you and fully customize it to your liking with a Dockerfile.

Tracing with OpenTelemetry

Glenn Bostoen — Fri, 01 Apr 2022 14:42:11 +0000

Tracing gives you great insights into certain bottlenecks within your application. I'll go over the steps needed to enable OpenTelemetry within a Fastify application on Google Cloud Run. I got inspired by the Nearform webinar from Daily.Dev.

https://www.youtube.com/watch?v=UKaJDmwIIpE

By default Cloud Run already has tracing enabled on requests that enter your cloud run instances. So if you go to traces in the console, you should see some traces for every HTTP endpoint of your application. You can see an example below:

Now we can start adding additional instrumentation for our application. There are multiple libraries proposed by Google:

OpenTelemetry: recommended by Google
OpenCensus: only alpha support and NodeJS support not even mentioned by Google
Google Client library

OpenTelemetry

As described on the website of OpenTelemetry

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

It's still in beta, but general availability should arrive soon.

Installing

Make sure to initialize all OpenTelemetry packages at the start of your application so that it can patch the necessary calls to underlying libraries. In this example, we add HTTP, TypeORM and Postgress instrumentation with the following instrumentation packages:

import opentelemetry, { DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { TraceExporter } from '@google-cloud/opentelemetry-cloud-trace-exporter';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { TypeormInstrumentation } from 'opentelemetry-instrumentation-typeorm';

export function tracing() {
  // Enable OpenTelemetry exporters to export traces to Google Cloud Trace.
  // Exporters use Application Default Credentials (ADCs) to authenticate.
  // See https://developers.google.com/identity/protocols/application-default-credentials
  // for more details.
  const provider = new NodeTracerProvider();

  // Initialize the exporter. When your application is running on Google Cloud,
  // you don't need to provide auth credentials or a project id.
  const exporter = new TraceExporter();

  // Configure the span processor to send spans to the exporter
  provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
  provider.register();
  opentelemetry.trace.setGlobalTracerProvider(provider);
  registerInstrumentations({
    instrumentations: [
      new HttpInstrumentation(),
      new PgInstrumentation(),
      new TypeormInstrumentation({
        // see under for available configuration
      }),
    ],
  });
  return { provider };
}

The Google Cloud trace exporter automatically uses the Cloud Run service account, so make sure this service account has access to utilize the Tracing API. You can configure or verify this in the IAM Console.

Once you have this added to your application and deployed on Cloud Run, you should be able to see more in-depth traces. The trace ID can always be found in the response header of your request: 'X-Cloud-Trace-Context'.

Google Coud - Daily analysis reports

If you feed Google Cloud with this tracing data, you will also get more detailed insights. An example where we had a significant impact on specific API calls is also noticed automatically by Google Cloud.

Summary

Easy to implement and can give you more insights over time. The Google Cloud Trace dashboard is quite simple and easy to use, so the entry barrier is relatively low.

Integrate Gitlab with Google Cloud workload identity federation

Glenn Bostoen — Fri, 11 Mar 2022 11:21:50 +0000

In Gitlab 14.7, connecting to AWS, GCP and vault, and other cloud services is now possible by introducing the CI_JOB_JWT_V2 environment variable. I'll use this environment variable to impersonate a service account via workload identity federation.

Workload identity federation

Workload identity federation allows you to impersonate an existing service account on Google Cloud. Everyday use cases for workload identity federation include:

Enabling a background application or continuous integration/continuous delivery (CI/CD) pipeline that runs outside of Google Cloud to access Google Cloud resources and APIs..
Enabling users of a web application that runs outside of Google Cloud to access data stored in a Google Cloud service, such as Cloud Storage or BigQuery.

To use workload identity federation, you configure Google Cloud to trust an external identity provider such as Amazon Web Services (AWS), Azure Active Directory (AD), an OIDC-compatible identity provider, or a SAML 2.0-compatible identity provider Preview. Applications can then use credentials issued by the external identity provider to impersonate a service account by following these steps:

Setup the workload identity provider.
Obtain a credential from the trusted identity provider.
Exchange the credential for a token from the Security Token Service.
Use the token from the Security Token Service to impersonate a service account and obtain a short-lived Google access token.

Setup the workload identity provider

Setting up the proper Identity Pool

gcloud iam workload-identity-pools create POOL_ID \
    --location="global" \
    --description="DESCRIPTION" \
    --display-name="DISPLAY_NAME"

Adding an OpenID Connect Provider to the pool (more specifically our Gitlab instance).

gcloud iam workload-identity-pools providers create-oidc PROVIDER_ID \
    --location="global" \
    --workload-identity-pool="POOL_ID" \
    --issuer-uri="ISSUER" \
    --allowed-audiences="AUDIENCE" \
    --attribute-mapping="MAPPINGS" \
    --attribute-condition="CONDITIONS""

Assigning service accounts that can be impersonated by these identities and the conditions:

gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_EMAIL \
    --role=roles/iam.workloadIdentityUser \
    --member="MEMBER_EXPRESSION"

Remark: the member expression is not clear within the workload identity federation console, but you can find more details if you navigate to the connected service account and click on permissions.

Conditions

Conditions make it especially useful, because this allows you to incorperate fine grained permissions in your pipeline. You could scope to certain branches, branches and even limit the users who can trigger the build.

For mapping we could do something like this:

export MAPPINGS="attribute.project_path=assertion.project_path"

For conditions we could then add the following:

export CONDITIONS="attribute.custom_path==\"glenn.bostoen/workload-federation-poc\""

Obtaining a credential from the trusted identity provider

It's quite easy on Gitlab to retrieve your credentials, it's injected in the environment variable CI_JOB_JWT_V2. The specification of the token can be found here, it looks as follows:

{
  "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558",
  "iss": "https://gitlab.example.com",
  "aud": "https://gitlab.example.com",
  "iat": 1585710286,
  "nbf": 1585798372,
  "exp": 1585713886,
  "sub": "project_path:mygroup/myproject:ref_type:branch:ref:main",
  "namespace_id": "1",
  "namespace_path": "mygroup",
  "project_id": "22",
  "project_path": "mygroup/myproject",
  "user_id": "42",
  "user_login": "myuser",
  "user_email": "myuser@example.com",
  "pipeline_id": "1212",
  "pipeline_source": "web",
  "job_id": "1212",
  "ref": "auto-deploy-2020-04-01",
  "ref_type": "branch",
  "ref_protected": "true",
  "environment": "production",
  "environment_protected": "true"
}

You could use this environment variable directly or output it to a temporary file:

echo ${CI_JOB_JWT_V2} > .ci_job_jwt_file

Exchange the credential for a token from STS

The following shell scripts allow you to exchange your JWT token for e federated token.

#!/bin/sh -x

PAYLOAD=$(cat <<EOF
{
"audience": "//iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_ID}/providers/${PROVIDER_ID}",
"grantType": "urn:ietf:params:oauth:grant-type:token-exchange",
"requestedTokenType": "urn:ietf:params:oauth:token-type:access_token",
"scope": "https://www.googleapis.com/auth/cloud-platform",
"subjectTokenType": "urn:ietf:params:oauth:token-type:jwt",
"subjectToken": "${CI_JOB_JWT_V2}"
}
EOF
)

FEDERATED_TOKEN=$(
    curl -X POST "https://sts.googleapis.com/v1/token" \
        --header "Accept: application/json" \
        --header "Content-Type: application/json" \
        --data "${PAYLOAD}" |
        jq -r '.access_token'
)

Impersonate service account

You can now use this federated token to impersonate a service account by getting an access token for this service account.

ACCESS_TOKEN=$(
    curl -X POST "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${SERVICE_ACCOUNT_EMAIL}:generateAccessToken" \
        --header "Accept: application/json" \
        --header "Content-Type: application/json" \
        --header "Authorization: Bearer ${FEDERATED_TOKEN}" \
        --data '{"scope": ["https://www.googleapis.com/auth/cloud-platform"]}' |
        jq -r '.accessToken'
)
echo "${ACCESS_TOKEN}"

Integration on Gitlab

You could use the scripts defined above and integrate these directly or you could use the gcloud CLI to easily impersonate a service account. So before we had the setup with permanent keys on Gitlab:

gcp-auth:
    image: google/cloud-sdk:slim
    before_script:
        - gcloud auth activate-service-account --key-file ${GOOGLE_APPLICATION_CREDENTIALS}
        - gcloud config set project ${GOOGLE_PROJECT}

This can now be replaced by the following script:

gcp-auth:
  image: google/cloud-sdk:slim
  script:
    - echo ${CI_JOB_JWT_V2} > .ci_job_jwt_file
    - gcloud iam workload-identity-pools create-cred-config "${GCP_WORKLOAD_IDENTITY_PROVIDER}"
      --service-account="${GCP_SERVICE_ACCOUNT}"
      --output-file=.gcp_temp_cred.json
      --credential-source-file=.ci_job_jwt_file
    - gcloud auth login --cred-file=`pwd`/.gcp_temp_cred.json
    - gcloud config set project ${GOOGLE_PROJECT}

We just need to specify the workload identity provider we want to use and the service account we want to impersonate.

Summary

If you have already isolated the authentication step to Google Cloud in your Gitlab templates, it should be quite straightforward to switch to this new mechanism. It gives you more granularity on who or what has access to your Cloud provider and is more secure in general by not having a permanent key on your instance. It does require some extra work on your Google Cloud project, but this can also be tackled by having some Terraform project templating.