Forem: Dewan Ahmed

The Principal Developer Advocate Paradox: Scaling Impact Without Burnout

Dewan Ahmed — Thu, 13 Mar 2025 14:21:21 +0000

As a Principal Developer Advocate, you don’t own the roadmap, control the pipeline, set budgets, or dictate campaigns—yet your work influences all of them. Your impact isn’t measured by code written but by friction removed, momentum created, and voices amplified.

The challenge? The more effective you are, the more demand you generate. You risk becoming a bottleneck, pulled into every meeting, every decision, every request. But true success in this role isn’t about being everywhere—it’s about creating force multipliers.

Let’s break down the hidden challenges of this role—and how to navigate them without burning out.

Paradox of Belonging

You are connected to every team—engineering, product, marketing, community—but you don’t belong to any single one. The role can be surprisingly isolating; you're expected to bridge gaps, yet you often lack a dedicated home base. Finding circles of trust—peer advocates, engineers who value your input, and mentors—becomes essential to navigating this unique position.

Freedom-Responsibility Paradox

You have significant autonomy in what you choose to work on, but there’s an implicit expectation that your efforts drive measurable impact. Are you truly solving the right problems, or just doing what seems urgent? The solve is to create an impact framework—a structured way to assess whether your advocacy, content, and engagement are making a difference. Freedom in this role isn’t about working on what excites you; it’s about owning the mission of amplifying developer success in the highest-leverage way.

Bandwidth Challenges: From Social Resource to Force Multiplier

It’s easy to become the "go-to person"—the one in every meeting, the voice in every strategy discussion, the person responding to every community question. This leads to burnout, endless context switching, and a diluted ability to create scalable impact. The trick is to shift from being a reactive resource to a strategic force multiplier:

Writing docs that answer recurring questions so teams can self-serve instead of always coming to you.
Creating frameworks that enable others to advocate, instead of being the single point of engagement.
Focusing on content, programs, and initiatives that scale beyond your personal involvement.

Being Truly Present (Even When Writing Docs)

You’re in a meeting, constantly context-switching, your mind racing to the next three tasks, while also drafting a response to a community Slack thread. Developer Advocates juggle multiple domains—public speaking, content creation, product feedback, and internal alignment—which makes it tempting to be everywhere at once. The reality? Presence matters.

✔ When writing content, resist the urge to multitask. Well-written content scales your knowledge far beyond what any single conversation can.

✔ When engaging with teams, be fully present. Not every meeting needs you, but when you’re in the right ones, your focus is your biggest asset.

✔ Protect your deep work time—whether that’s for writing, content creation, or strategic thinking.

The Perfection Trap

In engineering, we optimize for correctness. In advocacy, speed and iteration often win. It’s tempting to overanalyze, perfect every talk, refine every article—only to realize that the audience has already moved on. Great Developer Advocacy means accepting that good content now is better than perfect content later. It’s about learning in public, iterating fast, and engaging before the moment passes.

Authority Paradox

Contrary to perception, Principal Developer Advocates have influence, not authority. You’re expected to drive cross-functional initiatives—impacting engineering roadmaps, developer experience, and community growth—but you don’t control teams, priorities, or budgets. You can’t mandate change. Instead, your success depends on persuasion, trust, and credibility. The best Developer Advocates don’t push people to do things; they make others want to take action through clarity, vision, and relentless execution.

The Power of Saying "No"

With endless requests for your time, saying “yes” to everything means saying “no” to focused, strategic work. The ability to say “no” isn’t about shutting down collaboration—it’s about protecting your ability to make an impact.

If a request isn’t aligned with your mission, redirect it to scalable solutions (docs, videos, workshops).
Just because you can do something doesn’t mean you should—guard your time for high-leverage work.
The best Developer Advocates create systems that help others succeed without needing them in the loop every time.

Time to Ship

Being a Principal Developer Advocate is a delicate balance between influence and responsibility. You don't control the tools or the teams, but your impact reverberates through every part of the organization. To thrive in this role, you must understand how to amplify your efforts without being stretched too thin.

This isn’t just about showing up—it's about showing up where it matters. By focusing on scalable strategies, managing your bandwidth, and creating systems that empower others, you can truly maximize your impact. And when you embrace the power of saying "no," you protect the precious time that allows you to do your best work, at the highest level.

The role might be paradoxical, but in its heart lies the opportunity to shape the future of developer experience, content creation, and community engagement—without burning out. The key is mastering the art of leverage and focus to become the force multiplier that everyone needs.

This blog was inspired by a LinkedIn post from Bhavik Kothari, Principal Engineer at Amazon.

Speed Up Your CI Pipelines with Docker Layer Caching

Dewan Ahmed — Fri, 24 Jan 2025 22:21:13 +0000

In this video, Harness Principal Developer Advocate @dewanahmed demonstrates how Docker Layer Caching (DLC) in Harness CI can speed up your pipelines by 8X. You'll learn how DLC optimizes builds by reusing unchanged image layers, reducing redundant processing, and cutting down build times.

Watch as Dewan compares build and push performance between Harness CI and GitHub Actions, showcasing significant time savings and improved efficiency.

To learn more, check out Harness CI docs.

Read the blog.

Speed Up Your CI Pipelines with Docker Layer Caching

Dewan Ahmed — Thu, 23 Jan 2025 16:03:21 +0000

In modern software development, speed and efficiency are paramount. Long build times can slow down releases and hinder productivity. Docker layer caching is a powerful technique that helps optimize builds by reusing previously created image layers, reducing redundant processing. In this blog, we'll explore how Harness CI features Docker Layer Caching (DLC) to enhance build performance and streamline your CI/CD pipelines.

DLC and Multi-stage Builds

Every instruction in a Dockerfile creates a layer in the final image. Docker caches these layers to avoid rebuilding them unnecessarily, which can save significant time and reduce infrastructure costs. However, when a layer changes (e.g., modifying a file copied with COPY), Docker invalidates the cache for that layer and all subsequent layers, requiring them to be rebuilt. Understanding and optimizing layer usage helps in writing more efficient Dockerfiles, achieving faster build times, and lowering compute costs.

A multi-stage Dockerfile allows you to use multiple FROM statements to break the build process into stages. This helps keep the final image lightweight by copying only the necessary files from one stage to another, discarding anything unnecessary. It speeds up builds by leveraging layer caching, reducing the need to re-run expensive steps. Plus, it enhances security by minimizing the final image's attack surface and keeps the Dockerfile organized by separating concerns.

Harness CI Intelligence: Docker Layer Caching

Harness CI Intelligence optimizes Docker builds by leveraging Docker Layer Caching (DLC) to reuse unchanged image layers, significantly reducing build times and resource costs. When enabled, DLC restores previously built layers, avoiding redundant processing and speeding up the build and push process. Harness CI supports DLC across both Harness Cloud and self-managed infrastructure, providing flexibility in managing cache storage. This intelligent caching mechanism enhances CI/CD efficiency by minimizing infrastructure usage and improving developer productivity.

Benchmarking Build and Push to Docker

Check out the following video for a demo on running a build and push to Docker step for a Go repository using GitHub Actions and Harness CI, with Harness CI achieving an 8X improvement in build times.
Speed Up Your CI Pipelines with Docker Layer Caching

The following chart summarizes the performance comparison of this benchmark (Harness CI with DLC vs. GitHub Actions):

Conclusion

Implementing Docker Layer Caching in your CI/CD pipelines can lead to significant improvements in build performance, cost savings, and overall development efficiency. By reusing unchanged layers and minimizing redundant processing, Harness CI helps teams accelerate their workflows while optimizing infrastructure usage. Whether you're running builds in Harness Cloud or a self-managed environment, enabling DLC ensures faster feedback loops and a smoother development experience. Start leveraging Docker Layer Caching today to speed up your CI pipelines and focus on delivering value faster.

End-to-end MLOps CI/CD pipeline with Harness and AWS

Dewan Ahmed — Wed, 01 May 2024 13:29:10 +0000

MLOps tackles the complexities of building, testing, deploying, and monitoring machine learning models in real-world environments.

Integrating machine learning into the traditional software development lifecycle poses unique challenges due to the intricacies of data, model versioning, scalability, and ongoing monitoring.

In this tutorial, you'll create an end-to-end MLOps CI/CD pipeline that will:

Build and push an ML model to AWS ECR.
Run security scans and tests.
Deploy the model to AWS Lambda.
Add policy enforcement and monitoring for the model.

The story

This tutorial uses a fictional bank called Harness Bank. Assume that this fictional bank recently launched a website where clients can apply for a credit card. Based on the information provided in the form, the customer's application is approved or denied in seconds. This online credit card application is powered by a machine learning (ML) model trained on data that makes the decision accurate and unbiased.

Assume that the current process to update this hypothetical ML model is manual. A data scientist builds a new image locally, runs tests, and manually ensures that the model passes the required threshold for accuracy and fairness.

In this tutorial, you'll automate the model maintenance process and increase the build and delivery velocity.

Design and architecture

Before diving into the implementation, review the MLOps architecture.

For this tutorial, assume you are given a Python data science project, and you are requested to do the following:

Build and push an image for this project.
Run security scans on the container image.
Upload model visualization data to S3.
Publish model visualization data within the pipeline.
Run test on the model to find out accuracy and fairness scores.
Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model.
Deploy the model.
Monitor the model and ensure the model is not outdated.
Trigger the pipeline based on certain git events.
(Optional) Add approval gates for production deployment.

For this tutorial, assume that the data is already processed.

Prerequisites

This tutorial requires:

A Harness account with access to the Continuous Integration, Continuous Delivery, and Security Testing Orchestration modules. If you are new to Harness, you can sign up for free.
An AWS account, credentials, and a Harness AWS connector.
A GitHub account, credentials, and a Harness GitHub connector.

Prepare AWS

You need an AWS account with sufficient permissions to create/modify/view resources used in this tutorial.

Prepare AWS credentials.

This tutorial requires two sets of AWS credentials. One set is for a Harness AWS connector, and the other is for the AWS ECR scanner for STO.

You can use an AWS Vault plugin to generate AWS credentials for the AWS connector, and you can use the AWS console to generate the AWS Access Key ID, AWS Secret Access Key, and AWS Session Token, which are valid for a shorter time.

Save these credentials securely and make a note of your AWS account ID and AWS region.

⚠️ NOTE

If you are using a personal, non-production AWS account for this tutorial, you can initially grant admin access for these credentials. Once the demo works, reduce access to adhere to the principle of least privilege.

Create ECR repos. From your AWS console, navigate to Elastic Container Registry (ECR) and create two private repositories named ccapproval and ccapproval-deploy. Under Image scan settings, enable Scan on Push for both repositories.
Create an S3 bucket. Navigate to S3 and create a bucket named something like mlopswebapp. You'll use this bucket to host a static website for the credit card approval application demo, along with a few other artifacts.

Make sure all options under Block public access (bucket settings) are unchecked, and then apply the following bucket policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": "*",
               "Action": "s3:GetObject",
               "Resource": "arn:aws:s3:::YOUR_S3_BUCKET_NAME/*"
           }
       ]
   }

After making the bucket public, your bucket page should show a Publicly accessible flag.

From your AWS console, go to AWS Lambda, select Functions, and create a function from a container image using the following configuration:

Name: creditcardapplicationlambda
Container image URI: Select Browse images and find the ccapproval-deploy image. You can choose any image tag.
Architecture: x86_64
From Advanced: Select Enable function URL to make the function URL public. Anyone with the URL can access your function. For more information, go to the AWS documentation on Lambda function URLs.

Select Create function to create the function. You'll notice an info banner confirming that the function URL is public.

Prepare GitHub

This tutorial uses a GitHub account for source control management.

Fork the MLops sample app repository into your GitHub account.
Create a GitHub personal access token with following permissions on your forked repository:

repo/content: read+write
repo/pull requests: read
repo/webhooks: read+write

Create Harness secrets

Store your GitHub and AWS credentials as secrets in Harness.

In your Harness account, create or select a project to use for this tutorial.
In your project settings, select Secrets, select New Secret, and then select Text.
Create the following Harness text secrets:

git_pat - GitHub personal access token
aws_access_key_id - Generated from AWS console
aws_secret_access_key - Generated from AWS console
aws_session_token - Generated from AWS console
aws_vault_secret - Secret access key generated by Vault plugin

Make sure the Name and ID match for each secret, because you reference secrets by their IDs in Harness pipelines.

Create AWS and GitHub connectors

Create Harness connectors to connect to your AWS and GitHub accounts.

In your Harness project settings, go to Connectors.
Select New Connector, select the AWS connector, and then create an AWS connector with the following configuration:

Name: mlopsawsconnector
Access Key: AWS Vault plugin generated
Secret Key: Use your aws_vault_secret secret
Connectivity Mode: Connect through Harness Platform

Leave all other settings as is, and make sure the connection test passes.

Create another connector. This time, select the GitHub connector and and use the following configuration:

Name: mlopsgithubconnector
URL Type: Repository
Connection Type: HTTP
GitHub Repository URL: Enter the URL to your fork of the demo repo, such as https://github.com/:gitHubUsername/mlops-creditcard-approval-model
Username: Enter your GitHub username
Personal Access Token: Use your git_pat secret
Connectivity Mode: Connect through Harness Platform

Create the Harness pipeline

In Harness, you create pipeline to represent workflows. Pipeline can have multiple stages, and each stage can have multiple steps.

In your Harness project, create a pipeline named Credit Card Approval MLops.
Add a Build stage named Train Model.
Make sure Clone Codebase is enabled.
Select Third-party Git provider, and then select your mlopsgithubconnector GitHub connector. The repository name should populate automatically.
Select Set Up Stage.

In the following sections of this tutorial, you'll configure this stage to build and push the data science image, and you'll add more stages to the pipeline to meet the tutorial's objectives.

⚠️ NOTE

You can find a sample pipeline for this tutorial in the demo repo. If you use this pipeline, you must replace the placeholder and sample values accordingly.

Build, push, and scan the image

Configure your Train Model stage to build and push the data science image and then retrieve the ECR scan results.

Select the Infrastructure tab and configure the build infrastructure for the Train Model stage:

Select Cloud to use Harness Cloud build infrastructure.
For Platform, select Linux.
For Architecture, select AMD64.

Select the Execution tab to add steps to the stage.
Select Add Step, select the Build and Push to ECR step, and configure the step as follows:

For Name, enter Harness Training.
For AWS Connector, select your mlopsawsconnector AWS connector.
For Region, enter your AWS region.
For Account ID, enter your AWS account ID.
For Image Name, enter ccapproval.
For Tags, enter <+pipeline.executionId>. You can select the Input type icon to change the input type to expression (f(x)).
For Dockerfile (under Optional Configuration), enter Dockerfile_Training_Testing.

Select Apply Changes to save the step, and then select Save to save the pipeline.

Next, you'll add steps to your build stage to retrieve the results of the ECR repo security scan.

Because scanning is enabled on your ECR repositories, each image pushed to the repo by the Build and Push to ECR step is scanned for vulnerabilities. In order to successfully retrieve the scan results, your pipeline needs to wait for the scan to finish and then request the results.

Before adding the step to retrieve the scan result, use a Run step to add a 15-second wait to ensure that the scan is complete before the pipeline requests the scan results.

Select Add Step after the Harness Training step, and select the Run step.
For Name, enter Wait for ECR Image Scan.
For Command, enter the following, and then select Apply Changes to save the step.

   echo "ECR Image Scan In Progress..."
   sleep 15

Add an AWS ECR Scan step to get the scan results.

Select Add Step after the Wait step, and select the AWS ECR Scan step.
For Name, enter Security Scans for ML Model.
For Target/Name, enter ccapproval-ecr-scan.
For Variant, enter <+pipeline.executionId>. You can select the Input type icon to change the input type to expression (f(x)).
For Container Image/Name, enter ccapproval.
For Container Image/Tag, enter <+pipeline.executionId>.
For Region, enter your AWS region.
Under Authentication, use Harness expressions referencing your AWS credential secrets:
- For Access ID, enter <+secrets.getValue("aws_access_key_id")>.
- For Access Token, enter <+secrets.getValue("aws_secret_access_key")>.
- For Access Region, enter your AWS region.
- For Log Level, enter Info.
  - Under Settings, add the following key-value pair: AWS_SESSION_TOKEN: <+secrets.getValue("aws_session_token")>

Select Apply Changes to save the step, and then select Save to save the pipeline.

At this point, you can run the pipeline to test the Build stage.

Select Run Pipeline to test the Build stage. For Git Branch, enter main.
Wait while the pipeline runs, and then check your ccapproval ECR repository to find an image with a SHA matching the pipeline execution ID. Select Copy URI to copy the image URI; you'll need it in the next section.
Make sure the image scan also ran. In the Harness Build details, you can find the scan results in the AWS ECR Scan step logs. For example:

   Scan Results: {
       "jobId": "xlf06YX6a8AupG_5igGA6I",
       "status": "Succeeded",
       "issuesCount": 10,
       "newIssuesCount": 10,
      "issuesBySeverityCount": {
           "ExternalPolicyFailures": 0,
           "NewCritical": 0,
           "NewHigh": 1,
           "NewMedium": 5,
           "NewLow": 4,
           "NewInfo": 0,
           "Unassigned": 0,
           "NewUnassigned": 0,
           "Critical": 0,
           "High": 1,
           "Medium": 5,
           "Low": 4,
           "Info": 0,
           "Ignored": 0
       }
   }

You've successfully completed the first part of this tutorial: Configuring a Build stage that builds, pushes, and scans a trained data science image.

Continue the tutorial in the next sections and continue building your MLOps pipeline.

Test and upload artifacts

Add another Build stage to your pipeline that will run tests, build a Lambda image, and upload artifacts to S3.

Edit your MLOps pipeline and add another Build stage after the Train Model stage. Name the stage Run test and upload artifacts and make sure Clone Codebase is enabled.
On the stage's Overview tab, locate Shared Paths, and add /harness/output.
On the Infrastructure tab, select Propagate from existing stage, and select your Train Model stage.
On the Execution tab, add a Run step to run pytest on the demo copebase. Select Add Step, select the Run step, and configure it as follows:

For Name, enter pytest.
For Shell, select Sh.
For Command, enter:

   pytest --nbval-lax credit_card_approval.ipynb --junitxml=report.xml

Under Optional Configuration, locate Container Registry, and select your mlopsawsconnector AWS connector.
Under Optional Configuration, locate Image, and enter the image URI from your Train Model stage execution with the image tag replaced with <+pipeline.executionId>. For example:

   AWS_ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/AWS_ECR_REPO_NAME:<+pipeline.executionId>

The data science project includes two Dockerfiles: One for building the source and one for AWS Lambda deployment. Next, you'll add a step to build and push the image using the Dockerfile designed for AWS Lambda deployment.

Select Add Step and add a Build and Push to ECR step configured as follows:

For Name, enter Build and Push Lambda Deployment Image.
For AWS Connector, select your mlopsawsconnector AWS connector.
For Region, enter your AWS region.
For Account ID, enter your AWS account ID.
For Image Name, enter ccapproval-deploy.
For Tags, enter <+pipeline.executionId>.
Under Optional Configuration, locate Dockerfile, and enter Dockerfile_Inference_Lambda.

The pytest command from the Run step generates an HTML file with some visualizations for the demo ML model. Next, add steps to upload the visualizations artifact to your AWS S3 bucket and post the artifact URL on the Artifacts tab of the Build details page.

Select Add Step, and add an Upload Artifacts to S3 step configured as follows:

For Name, enter Upload artifacts to S3.
For AWS Connector, select your mlopsawsconnector AWS connector.
For Region, enter your AWS region.
For Bucket, enter your S3 bucket name.
For Source Path, enter /harness/output/model_metrics.html. This is where the model visualization file from the pytest step is stored.

Use the Artifact Metadata Publisher plugin to post the visualization artifact URL on the build's Artifacts tab.

Add a Plugin step after the Upload Artifacts to S3 step.
For Name, enter Publish ML model visualization.
For Container Registry, select the built-in Harness Docker Connector.
For Image, enter plugins/artifact-metadata-publisher.
Under Optional Configuration, locate Settings, and add the following key-value pairs:

   file_urls: https://S3_BUCKET_NAME.s3.AWS_REGION.amazonaws.com/harness/output/model_metrics.html
   artifact_file: artifact.txt

In addition to the model visualization, the pytest command also generates a shared_env_variables.txt file to export the model's accuracy and fairness metrics. However, this data is lost when the build ends because Harness stages run in isolated containers. Therefore, you must add a step to export the ACCURACY and EQUAL_OPPORTUNITY_FAIRNESS_PERCENT values as output variables.

After the Plugin step, add a Run step configured as follows:

For Name, enter Export accuracy and fairness variables.
For Shell, select Sh.
For Commmand, enter:

   # File path
   FILE_PATH="/harness/output/shared_env_variables.txt"

   # Read the file and export variables
   while IFS='=' read -r key value; do
       case $key in
           ACCURACY)
               export ACCURACY="$value"
               ;;
           EQUAL_OPPORTUNITY_FAIRNESS_PERCENT)
               export EQUAL_OPPORTUNITY_FAIRNESS_PERCENT="$value"
               ;;
           *)
               echo "Ignoring unknown variable: $key"
               ;;
       esac
   done < "$FILE_PATH"

   echo $ACCURACY
   echo $EQUAL_OPPORTUNITY_FAIRNESS_PERCENT

Under Optional Configuration, locate Output Variables, and add the following two output variables:

   ACCURACY
   EQUAL_OPPORTUNITY_FAIRNESS_PERCENT

Save the pipeline, and then run it. Again, use main for the Git Branch.
Wait while the pipeline runs, and then make sure:

Your ccapproval and ccapproval-deploy ECR repositories have images with SHAs matches the pipeline execution ID.
Your S3 bucket has /harness/output/model_metrics.html.
The URL to the model_metrics artifact appears on the Artifacts tab in Harness.

The output variable values are in the log for the Export accuracy and fairness variables step, such as:

   0.92662
   20.799999999999997

Congratulations! So far, you've completed half the requirements for this MLOps project:

[x] Build and push an image for this project.
[x] Run security scans on the container image.
[x] Upload model visualization data to S3.
[x] Publish model visualization data within the pipeline.
[x] Run test on the model to find out accuracy and fairness scores.
[ ] Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model.
[ ] Deploy the model.
[ ] Monitor the model and ensure the model is not outdated.
[ ] Trigger the pipeline based on certain git events.
Add approval gates for production deployment.

Continue on with policy enforcement in the next section.

Add ML model policy checks

In this section, you'll author OPA policies in Harness and use a Custom stage to add policy enforcement to your pipeline.

Harness Policy As Code uses Open Policy Agent (OPA) as the central service to store and enforce policies for the different entities and processes across the Harness platform. You create individual policies, add them to policy sets, and select the entities (such as pipelines) to evaluate those policies against.

For this tutorial, the policy requirements are that the model accuracy is over 90% and the fairness margin for equal opportunity is under 21%.

In your Harness project settings, go to Policies, select the Policies tab, and then select New Policy.
For Name, enter Check fairness and accuracy scores.
For How do you want to setup your Policy, select Inline.
Enter the following policy definition, and then select Save.

   package main

   default allow = false

   allow {
       input.accuracy >= 0.9
       input.fairnessScoreEqualOpportunity <= 21
   }

   deny[msg] {
       not allow
       msg = "Deny: Accuracy less than 90% or fairness score difference greater than 21%"
   }

Select the Policy Sets tab, and then select New Policy Set. Use the following configuration:

For Name, enter Credit Card Approval Policy Set.
For Entity Type that this policy set applies to, select Custom.
For On what event should the policy set be evaluated, select On Step.
Select Add Policy, and select your Check fairness and accuracy scores policy.
For What should happen if a policy fails?, select Warn and Continue.
Select Finish, and make sure the Enforced switch is enabled.

Edit your MLOps pipeline, and add a Custom stage after the second Build stage. Name the stage Model Policy Checks.
Select a Harness Delegate to use for the Custom stage.

The Build stages run on Harness Cloud build infrastructure, which doesn't require a Harness Delegate. However, Custom stages can't use this build infrastructure, so you need a Harness Delegate.

If you don't already have one, install a delegate. Then, on the Custom stage's Advanced tab, select your delegate in Define Delegate Selector.

Add a Shell Script step to relay the accuracy and fairness output variables from the previous stage to the current stage. Configure the Shell Script step as follows:

For Name, enter Accuracy and Fairness.
For Timeout, enter 10m.
For Script Type, select Bash.
For Select script location, select Inline.

For Script, enter the following:

  accuracy=<+pipeline.stages.Harness_Training.spec.execution.steps.Export_accuracy_and_fairness_variables.output.outputVariables.ACCURACY>
  fairness_equalopportunity=<+pipeline.stages.Harness_Training.spec.execution.steps.Export_accuracy_and_fairness_variables.output.outputVariables.EQUAL_OPPORTUNITY_FAIRNESS_PERCENT>

Under Optional Configuration, locate Script Output Variables, and add the following two variables:
- accuracy - String - accuracy
- fairness_equalopportunity - String - fairness_equalopportunity

⚠️ NOTE

While you can feed the output variables directly into Policy steps, this Shell Script step is a useful debugging measure that ensures the accuracy and fairness variables are populated correctly.

Add a Policy step after the Shell Script step.

For Name, enter Enforce Fairness and Accuracy Policy.
For Timeout, enter 10m.
For Entity Type, select Custom.
For Policy Set, select your Credit Card Approval Policy Set.

For Payload, enter the following:

  {
      "accuracy": <+execution.steps.Accuracy_and_Fairness.output.outputVariables.accuracy>,
      "fairnessScoreEqualOpportunity": <+execution.steps.Accuracy_and_Fairness.output.outputVariables.fairness_equalopportunity>
  }

Save the pipeline.

The next time the pipeline runs, the policy is enforced to check if the model accuracy and fairness margins are within the acceptable limits. If not, the pipeline produces a warning and then continues (according to the policy set configuration). You could also configure the policy set so that the pipeline fails if there is a policy violation.

If you want to test the response to a policy violation, you can modify the policy definition's allow section to be more strict, such as:

allow {
    input.accuracy >= 0.95
    input.fairnessScoreEqualOpportunity <= 19
}

Since the model accuracy is around 92% and the fairness margin is around 20%, this policy definition should produce a warning. Make sure to revert the change to the policy definition once you're done experimenting.

Deploy AWS Lambda function

In Harness, you can specify the location of a function definition, artifact, and AWS account, and then Harness deploys the Lambda function and automatically routes traffic from the old version of the Lambda function to the new version on each deployment. In this part of the tutorial, you'll update an existing Lambda function by adding a Deploy stage with service, environment, and infrastructure definitions.

Edit your MLOPs pipeline, and add a Deploy stage named lambdadeployment.
Deployment Type, select AWS Lambda, and then select Continue.
Create a service definition for the Lambda deployment.

Select Add Service.
For Name, enter creditcardapproval-lambda-service.
For Set up service, select Inline.
For Deployment Type, select AWS Lambda.
Under AWS Lambda Function Definition, for Manifest Identifier, enter lambdadefinition, and for File/Folder Path, enter /lambdamanifest.

After creating the manifest under Harness File Store, add the following to the service manifest, and select Save:

   functionName: `creditcardapplicationlambda`
   role: LAMBDA_FUNCTION_ARN

Replace LAMBDA_FUNCTION_ARN with your Lambda function's ARN. You can find the Function ARN when viewing the function in the AWS console.

Under the Artifacts section for the service definition, provide the artifact details to use for the lambda deployment:

Artifact Source Identifier: ccapprovaldeploy
Region: YOUR_AWS_REGION
Image Path: ccapproval-deploy
Value - Tag: <+input> (runtime input)

Create environment and infrastructure definitions for the Lambda deployment. On the Deploy stage's Environment tab, select New Environment, and use the following environment configuration:

   name: lambda-env
   type: PreProduction

From the lambda-env, go to the Infrastructure Definitions tab, and add an infrastructure definition with the following configuration:

   name: `aws-lambda-infra`
   deploymentType: `AwsLambda`
   type: AwsLambda
     spec:
       connectorRef: `mlopsawsconnector`
       region: YOUR_AWS_REGION

Select Save to save the infrastructure definition.
On the Deploy stage's Execution tab, add an AWS Lambda Deploy step named Deploy Aws Lambda for the name. No other configuration is necessary.
Save and run the pipeline. For Git Branch, enter main, and for Tag, enter <+pipeline.executionId>, and then select Run Pipeline.

You need to provide the image tag value because the service definition's Tag setting uses runtime input (<+input>).

While the pipeline runs, you can observe the build logs showing the lambda function being deployed with the latest artifact that was built and pushed from the same pipeline.

Test the response from the lambda function.

In your AWS console, go to AWS Lambda, select Functions, and select your creditcardapplicationlambda function.
On the Test tab, select Create new event, and create an event named testmodel with the following JSON:

   {
     "Num_Children": 2,
     "Income": 500000,
     "Own_Car": 1,
     "Own_Housing": 1
   }

Select Test to execute the function with your testmodel test event. Once the function finishes execution, you'll get the result with a Function URL.

Note the Function URL resulting from the lambda function test. This is the endpoint that your ML web application would call. Depending on the prediction of 0 or 1, the web application either approves or denies the demo credit card application.

Monitor the model

There are many ways to monitor ML models. In this tutorial, you'll monitor if the model was recently updated. If it hasn't been updated recently, Harness sends an email alerting you that the model might be stale.

Edit your MLOPs pipeline, and add a Build stage after the Deploy stage.

For Name, enter Monitor Model stage.
Disable Clone Codebase.
On the Infrastructure tab, select Propagate from existing stage and select the first Build stage.

Add a Run step to find out when the model was last updated.

For Name, enter Monitor Model step.
For Shell, select Sh.
For Command, enter:

   # GitHub repository owner
   OWNER="YOUR_GITHUB_USERNAME"

   # GitHub repository name
   REPO="mlops-creditcard-approval-model"

   # Path to the file you want to check (relative to the repository root)
   FILE_PATH="credit_card_approval.ipynb"

   # GitHub Personal Access Token (PAT)
   TOKEN=<+secrets.getValue("git_pat")>

   # GitHub API URL
   API_URL="https://api.github.com/repos/$OWNER/$REPO/commits?path=$FILE_PATH&per_page=1"

   # Get the current date
   CURRENT_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

   # Calculate the date 7 days ago
   SEVEN_DAYS_AGO=$(date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ")

   # Get the latest commit date for the file
   LATEST_COMMIT_DATE=$(curl -s -H "Authorization: token $TOKEN" $API_URL | jq -r '.[0].commit.committer.date')

   # Check if the file has been updated in the last 7 days
   if [ "$(date -d "$LATEST_COMMIT_DATE" +%s)" -lt "$(date -d "$SEVEN_DAYS_AGO" +%s)" ]; then
       export model_stale=true
   else
       export model_stale=false
   fi

Under Optional Configuration, add model_stale to Output Variables.

After the Monitor Model stage, add a Custom stage named Email notification. This stage will send the email notification if the model is stale.
Add an Email step to the last Custom stage.

For Name, enter Email.
For Timeout, enter 10m.
For To, enter the email address to receive the notification, such as the email address for your Harness account.
For Subject, enter Credit card approval ML model has not been updated in a week.
For Body, enter It has been 7 days since the credit card approval ML model was updated. Please update the model.
On the step's Advanced tab, add a conditional execution so the Email step only runs if the model_stale variable (from the Monitor Model step) is true:
- For Execute this step, select If the stage executes successfully up to this point.
- Select And execute this step only if the following JEXL Condition evaluates to true.
- Enter the following JEXL condition:
```
 <+pipeline.stages.Monitor_Model_Stage.spec.execution.steps.Monitor_Model_Step.output.outputVariables.model_stale> == true
```

Save and run the pipeline. For Git Branch, enter main, and for Tag, enter <+pipeline.executionId>.

Trigger pipeline based on Git events

So far, this tutorial used manually triggered builds. However, as the number of builds and pipeline executions grow, it's not scalable to manually trigger builds. In this part of the tutorial, you'll add a Git event trigger.

Assume your team has a specific requirement where they want the MLOps pipeline to run only if there's an update to the Jupyter notebook in the codebase.

In your MLOps pipeline, select Triggers at the top of the Pipeline Studio, and then select New Trigger.
Select the GitHub webhook trigger.
On the trigger's Configuration tab:

For Name, enter trigger_on_notebook_update.
For Connector, select your mlopsgithubconnector GitHub connector.
The Repository URL should automatically populate.
For Event, select Push.

Select Continue to go to the Conditions tab.

For Branch Name, select the Equals operator, and enter main for the Matches Value.
For Changed Files, select the Equals operator, and enter credit_card_approval.ipynb for the Matches Value.

Select Continue to go to the Pipeline Input tab.

The Git Branch should automatically populate.
For Primary Artifact, enter ccapprovaldeploy.
For Tag, enter <+pipeline.executionId>.

Select Create Trigger.

The trigger webhook should automatically register in your GitHub repository. If it doesn't, you'll need to manually register the webhook:

On the list of triggers in Harness, select the Link icon to copy the webhook URL for the trigger.

In your GitHub repository, navigate to Settings, select Webhook, and then select Add Webhook.
Paste the webhook URL in Payload URL.
Set the Content type to application/json.
Select Add webhook.

A green checkmark in the GitHub webhooks list indicates that the webhook connected successfully.

With the trigger in place, whenever you push a change to the credit_card_approval.ipynb file on the main branch, the MLOps pipeline runs. In the trigger settings, you could adjust or remove the Conditions (branch name, changed files, and so on) according to your requirements, if you wanted to use a Git event trigger in a live development or production scenario.

Add an approval gate before prod deployment

Your organization might require an approval gate for your CI/CD pipeline before an artifact is deployed to production. Harness offers built-in approval steps for Jira, ServiceNow, or Harness approvals.

Assume that you have a different image for production, and a different AWS Lambda function is deployed based on that container image. In your MLOps pipeline, you can create another AWS Lambda deployment stage with another AWS Lambda deploy step for the production environment and use the approval gate prior to running that production deployment stage.

To add the approval gate, add an Approval stage immediately prior to the Deploy stage that requires approval.

For Name, enter approval-to-prod.
For Approval Type, select Harness Approval.

Add an Approval step to the Approval stage.

For Name, enter approval-to-prod.
For Timeout, enter 1d.
Use the default Message.
For User Groups, select Select User Groups, select Project, and select All Project Users.

Save the pipeline.

Next time you run the pipeline, someone from the Harness project must approve the promotion of artifact to the production environment before the final Deploy stage runs.

Use the model in a web application

In a live MLOps scenario, the ML model would likely power a web application. While this app development is outside the scope of this tutorial, check out this animation that demonstrates a simple web application developed using plain HTML/CSS/JS. The outcome of the credit card application uses the response from the public AWS Lambda function URL invocation.

Conclusion

Congratulations! Here's what you've accomplished in this tutorial:

[x] Build and push an image for this project.
[x] Run security scans on the container image.
[x] Upload model visualization data to S3.
[x] Publish model visualization data within the pipeline.
[x] Run test on the model to find out accuracy and fairness scores.
[x] Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model.
[x] Deploy the model.
[x] Monitor the model and ensure the model is not outdated.
[x] Trigger the pipeline based on certain git events.
x Add approval gates for production deployment.

Now that you've built an MLOps pipeline on Harness and used the Harness platform to train the model, check out the following guides to learn how you can integrate other popular ML tools and platforms into your Harness CI/CD pipelines:

Build and Push to GAR and Deploy to GKE - End-to-End CI/CD Pipeline

Dewan Ahmed — Tue, 02 Jan 2024 20:50:07 +0000

In this tutorial, you'll explore how to build a streamlined CI/CD pipeline using the Harness Platform, integrating the robust services of Google Artifact Registry (GAR) and Google Kubernetes Engine (GKE). GAR excels in managing and storing container images securely, while GKE offers a scalable environment for container deployment. The Harness Platform serves as a powerful orchestrator, simplifying the build and push process to GAR and managing complex deployments in GKE. You'll also cover implementing crucial approval steps for enhanced security and setting up Slack notifications for real-time updates, showcasing how these tools together facilitate a robust, streamlined CI/CD process.

Architectural and Pipeline Diagrams

Prerequisites

A Harness free plan. If you don't have one, sign up for free.
A GitHub account.
A Docker Hub account.
A GCP account with permissions for Google Artifact Registry and Kubernetes Engine.t
Access to a slack workspace and permissions to create a slack app.

There’s a bonus section in this tutorial where you’ll run security tests during the build process and create a policy for the deployment process. To follow this section, you’ll need a Harness enterprise account.

Required Setup and Configurations

Demo application

You can either fork Captain Canary Adventure (CCA) App or bring your own application (as long as it has a Dockerfile).

📝	This tutorial assumes the use of a fork of the CCA App. If you are using your own app, make the necessary changes.

GitHub and Docker authentication

Create a GitHub personal access token (PAT) that will have read access to the demo application repository. Create a Docker access token.

Image registry setup on Google Cloud Platform (GCP):

Enable the Artifact Registry API.
From Artifact Registry, click + CREATE REPOSITORY.
Give this repository a name - cca-registry, choose Docker as the format, Standard as the mode, location type Region (choose a region near you), Google-managed encryption key for encryption, have Dry Run selected, and click CREATE.

Kubernetes cluster setup with GKE:

Enable Kubernetes Engine API.
Create a GKE (autopilot) cluster by selecting a region near you.

GCP IAM and Service Account setup:

Create a GCP Service Account. Copy the email address generated for this service account. It will be in this format: SERVICE_ACCOUNT_NAME@GCP_PROJECT_NAME.iam.gserviceaccount.com.
Navigate to the artifact registry repository you created earlier, select it and click on Permissions tab. You need to create two types of access to the artifact registry repository - a public read access and a fine-grained write access. Click + ADD PRINCIPAL from the Permissions tab and paste the email address of the service account previously copied. Assign Artifact Registry Writer role to this principal. Next, click + ADD PRINCIPAL from the Permissions tab and type in allUsers for the principal and Artifact Registry Reader for the role. You might see a warning like this: > “This resource is public and can be accessed by anyone on the internet.”
Navigate to IAM & Admin, locate the service account, select it, and then click on ADD KEY → Create new key. Choose the JSON format, and a key for your service account will be downloaded to your computer. Exercise caution and refrain from sharing this key with anyone; treat it as you would a password.

Slack workspace and app setup:

To create a Slack app and incoming webhook, you'll need elevated privilege in that Slack workspace. Create a new Slack workspace for this tutorial and an incoming webhook for a specific channel.

Your newly created slack webhook will look like this: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

Treat this as sensitive information.

Harness entity setup

Create secrets:

a. GitHub Secret: Navigate to the Harness console. From Project Setup → Secrets, click + New Secret → Text, give the secret a name (for example, cca-git-pat) and paste in the previously created GitHub PAT.

b. Docker Secret: Similarly, create a docker secret (you can name it docker-secret) and use the previously created Docker PAT as the secret value.

c. Slack Webhook: Similarly, create a slack webhook secret (you can name it slack-webhook) and paste in the previously created slack webhook value.
Connectors in Harness help you pull in artifacts, sync with repos, integrate verification and analytics tools, and leverage collaboration channels. From Project Setup → Connectors → + New Connector, create the following connectors:

a. GitHub Connector: Harness platform connects to the source code repository using this connector. Give this connector a name (for example, cca-git-connector), choose URL type as Repository, connection type as HTTP, and paste in the forked Github Repository URL of the demo app. Use your GitHub username and the previously created GitHub secret for authentication. Select the connectivity mode as Connect through Harness Platform. The connection test should be successful.

b. Docker Connector: Harness platform pulls in the docker image for the Slack notification using this connector. Give this connector a name (for example, docker-connector), choose provider type as DockerHub, Docker Registry URL as https://index.docker.io/v2/, enter in your docker username and select the previously created docker secret. Select the connectivity mode as Connect through Harness Platform. The connection test should be successful.

c. Kubernetes Connector: Harness platform creates and manages resources on your GKE cluster using this connector. Give this connector a name (for example, gke-connector). Choose Use the credentials of a specific Harness Delegate… and click + Install new Delegate. The Harness Delegate is a service you run in your local network or VPC to connect all of your providers with your Harness account. Follow the instructions to install a delegate on your Kubernetes cluster and once the installation is complete, select the newly created delegate from the dropdown. The connection test should be successful.

d. GCP Connector: The GCP connector allows you to connect to your Google Cloud Platform resource and perform actions via Harness platform. Give this connector a name and select Specify credentials here under the Details section. Add a new secret name and upload the GCP service account key JSON file you previously downloaded. Select the connectivity mode as Connect through Harness Platform. The connection test should be successful.

Build and push image to GAR

First, let’s create the build and push image part of the pipeline. Click on Pipelines → + Create a Pipeline and give it a name (e.g., gar-gke-cicd-pipeline). Select the Inline option to store the pipeline definition in Harness, and then click Start.

Click on Add Stage and choose Build as the stage type. A Harness pipeline can consist of one or more stages. Give this stage a name (e.g., Push to GAR), select the Clone Codebase option (this should be enabled, by default), and choose the GitHub connector you previously created from the dropdown. The repository name should auto-populate. Click Set Up Stage.

Under Infrastructure, specify where you'll deploy your application. Select Cloud for Harness hosted builds and choose Linux/AMD64 for the Platform option.

Under Execution, click Add Step → Add Step and find Build and Push to GAR step from the Step Library. Name this step (e.g., BuildAndPushToGAR) and select the GCP connector you created earlier from the dropdown. When choosing the host, use the region selected when creating the image registry repository (e.g., northamerica-northeast1-docker.pkg.dev). Refer to Harness Developer Hub docs or GCP Artifact Registry docs for more details on selecting the region for GAR. Enter your GCP project ID under Project Id. For Image Name, use the image registry repository name followed by the application name in this format: cca-registry/cca-app. Use latest for now as the Tags. Click Apply Changes.

Now, click Run to execute the pipeline. Enter Master as the git branch name for the build (or main if you're not using the forked CCA app). A successful execution of the pipeline should resemble the following:

Deploy to GKE

If you're using a fork of the Captain Canary Adventure App, update the deployment.yaml before deploying the application to Kubernetes. The current YAML uses Harness variables, but since you're not there yet, you'll need to hardcode some values for now.

Assuming your GKE Artifact Registry repository name is cca-registry and the image name is cca-app, replace the current values in values.yaml with the following:

image:
  name: cca-registry/cca-app
  tag: latest

Click on + Add Stage in the pipeline and choose Deploy as the stage type. Give the stage a name (e.g., GKE Deploy), select Kubernetes as the deployment type, and click Set Up Stage.

Next, create a service to deploy. Choose Kubernetes as the deployment type, select the GitHub connector, and specify the paths for the manifests. For example, for the Captain Canary Adventure App, here are the manifest details:

Environments represent your deployment targets (such as QA or Prod). Each environment contains one or more Infrastructure Definitions that list your target clusters, hosts, namespaces, etc. Click on + New Environment, give this environment a name (e.g., cca-env), select Pre-Production as the environment type, and click Save.

Next, create an infrastructure definition. Click + New Infrastructure, under cluster details, select the GKE connector, provide a Kubernetes namespace where your application will be deployed (e.g., cca-ns), and click Save. For execution strategies, choose Rolling Deployment and click Use Strategy. Under optional configuration, select Enable Kubernetes Pruning. With this setting, Harness will use pruning to remove any resources present in an old manifest but no longer in the manifest used for the current deployment. You can find more information about this configuration on the Harness Developer Hub.

You’re all set! Click Save and then Run. Use master for the git branch. A successful pipeline execution will override the cca-app:latest image on your GAR repository and deploy this image to your GKE cluster.

Add Approval and Slack Notifications

In practical DevOps pipelines, gates are implemented to control artifact promotion to the production environment. Harness supports various types of approvals in Continuous Delivery (CD) pipelines. In this tutorial, you'll use the manual approval step.

Within the gke-deploy stage, click + Add Step before the Rollout Deployment step and find Harness Approval under Approval in the Step Library. Keep all default options, and you'll need to select the approver from the User Groups. Choose Project → All Project Users under user group selection. If you're the only member of this project, you'll be the sole approver. Click Apply Selected. Click Apply Changes for the manual approval step, and then click Save.

Now, let's add a notification stage so that whenever a deployment is approved in the CI/CD pipeline, a notification will be sent to a Slack channel, indicating who approved it.

From the pipeline, click on Add Stage after the gke-deploy stage, select Build as the stage type, give this stage a name (e.g., Notifications Stage), disable the Clone Codebase option, and click Set Up Stage. Under Infrastructure, choose Use a New Infrastructure → Cloud and Linux → AMD64 for the Operating System. Under Execution, click Add Step → Add Step and find Plugin in the Build section of the Step Library.

Name this step (e.g., Slack Notification), choose the Docker connector you previously created under Container Registry, for the image, use plugins/slack, and add the following key-values under Optional Configuration → Settings (assuming the id for your Slack webhook secret is slackwebhook).

Key	Value
webhook	`<+secrets.getValue("slackwebhook")>`
template	The deployment is moved to prod by `<+approval.approvalActivities[0].user.name>`

Notice the use of a Harness variable expression in the template, which retrieves the name of the approver from the previous stage.

Before running the pipeline, one more update is needed. Currently, every image built, pushed, and deployed has the same image tag, making it challenging to track based on the build number. Harness provides powerful built-in and custom variable expressions for various practical use cases.

Click on Variables for your pipeline and select + Add Variable at the pipeline level. Let’s add two variables:

Variable Name	Variable Value
imageName	cca-registry/cca-app
imageTag	`<+pipeline.sequenceId>`

For the imageTag, click the 📌 icon and select Expression. Every time you run the pipeline, the pipeline sequence ID will change, and subsequently, the image that will be built and deployed will also change.

Now that you’ve updated the pipeline to pass in the imageName and imageTag as variables, let’s update the codebase to replace the hardcoded values. Revert the changes to deployment.yaml and values.yaml you previously made. You'll observe that the values.yaml file will receive the imageName and imageTag during pipeline runtime, and then the deployment.yaml file will use those values from the values.yaml file.

Now, click Save and then Run. After a successful gar-build-and-push stage, you should see an image in the GAR repository with a numeric tag that matches the pipeline sequence ID. Right after, you should see a following prompt for approval:

You can (optionally) add a comment and click Approve. The pipeline should continue as before and you’ll see a deployment on your Kubernetes cluster. However, this time, you’ll see a slack notification resulting from your approval. To modify the text that appears on the notification, you can modify the template value in the slack plugin step settings.

Security Tests and Policy Enforcement (Bonus Section)

📝	These features are only available on Harness paid plans

Run OWASP Tests

You can scan your code repositories using OWASP Dependency-Check within a Harness pipeline. Within the gar-build-and-push stage, click on + Add Step → Add Step before the BuildAndPushGAR step. From the step library, find Owasp under the Security Tests section.

Use the following settings to configure the OWASP Dependency Check and click Apply Changes:

Setting Name	Value
Name	Owasp Tests
Scan Mode	Orchestration
Target.Name	cca-owasp-tests
Variant	master (this is the branch name for the repo)
Log Level	Info
Fail On Severity	Critical

You can have any string values for the step name and target.name. For the Variant, use the branch name for your codebase (e.g. master or main). Selecting Critical for Fail On Severity means that if there is any critical error, this test will fail and the pipeline execution will halt. You can check out the OWASP scanner reference to learn more on these configurations.

Click Save and then Run. If your codebase doesn’t have an OWASP critical bug, the pipeline should execute successfully. To enforce a fail on this OWASP scan, use a codebase with known vulnerabilities like WebGoat and you’ll see the OWASP scanner in action.

Add a policy to mandate approval step on deployment stages

Harness Policy As Code uses Open Policy Agent (OPA) as the central service to store and enforce policies for the different entities and processes across the Harness platform. In this section, you will define a policy that will deny a pipeline execution if there is no approval step defined in a deployment stage.

From Project Setup → Policies, follow the wizard to create a policy from the policy library. Use the Pipeline - Approval policy.

In the next screen, choose Project scope, trigger event On Run, and for the severity, choose Error & Exit. Next, click Yes to apply the policy.

Now, let’s remove the approval step from the gke-deploy stage. Click on Edit on the pipeline and click on the cross button on the Harness Approval step. Click Save and then Run. You should see the following error:

Add the Harness Approval back, save and run the pipeline and this time the pipeline should execute successfully. An end to end successful pipeline execution will look like this:

View the running application

While connected to your GKE cluster, execute the following command:

kubectl get svc -n cca-ns

This is assuming that you deployed the application to cca-ns namespace.

The output will be something like this:

NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
cca-app-service         LoadBalancer   34.118.227.33   34.152.47.53   80:30008/TCP   6d19h

Navigate to the IP address listed under the EXTERNAL-IP column for your case, and you should see a running Captain Canary Adventure application. As the application is running on port 80, you can omit the port number from the URL.

Homework Task

If you’d like to take this pipeline one step further, you can leverage caching to share data across stages because each stage in a Harness CI pipeline has its own build infrastructure. Check out how to save and restore cache from Google Cloud Storage (GCS).

Ephemeral CI environments using ttl.sh and Gitness

Dewan Ahmed — Mon, 18 Dec 2023 21:29:10 +0000

In the realm of software development, balancing continuous integration with maintaining quality is a significant challenge. ttl.sh and Gitness offer a solution. ttl.sh is an ephemeral Docker image registry that allows for the creation of temporary image tags with built-in expiry. Gitness, an open-source platform by Harness, simplifies the management of source code repositories and development pipelines. This blog post will explore how these tools can be used to create temporary CI environments to expedite development processes.

The Problem

Multiple developers working on different features can lead to merge conflicts and a bloated image registry. Traditional CI environments often retain Docker images from feature branches too long, which complicates image management and increases the likelihood of testing against outdated or incorrect images.

The Solution

The workflow in the diagram leverages ttl.sh and Gitness to manage these challenges:

Feature Branch Workflow: The creation of a pull request (PR) for a feature branch triggers a build in Gitness.
Image Creation: Gitness constructs a Docker image from the feature branch, using ttl.sh for tagging with a UUID and a time-to-live (TTL) limit. This process does not require credentials and maintains privacy.
Automated Testing: The image is put through automated tests. Should the tests fail, developers are notified; if they succeed, the process proceeds.
Image Promotion: After a PR passes all checks and is merged, the image is given a permanent tag and pushed to a central image registry, which at this point, requires proper credentials.

This approach ensures that only quality-assured images make it to the central registry, reducing the clutter and promoting a cleaner CI process.

Demo

Let's construct this setup step by step. Starting with pushing an image to ttl.sh through Gitness, I’ll guide you to resources for completing the remaining components of the system.

Begin with the Gitness documentation to set up your first project. You can start a new repository or link an existing one from GitHub or GitLab. The specific source code repository can be any; the essential requirement is the presence of a Dockerfile.

Navigate to "Pipelines'' within your repository and create a new pipeline. Gitness will suggest a sample pipeline. Replace it with the following YAML:

kind: pipeline
spec:
 stages:
   - name: build-and-push
     spec:
       platform:
         arch: amd64
         os: linux
       steps:
         - name: docker_build
           spec:
             inputs:
               repo: ttl.sh/xxxx-yyyy-nnnn-2a2222-4b44
               tags: 1h
             name: docker
           type: plugin
     type: ci
version: 1

Observe the image repo and tag. ttl.sh is the image repository name and the UUID (xxxx-yyyy-nnnn-2a2222-4b44) is the image name. You can replace the hard coded image name with a Gitness secret for a dynamic image name. Click on Secrets and add a new Gitness secret called random_image_name. Now you can update your pipeline YAML so that the image name is not fixed:

repo: ttl.sh/${{ secrets.get("random_image_name") }}

Save the pipeline and click on Run. After executing the pipeline, your output should resemble the following:

Next add the following steps yourself:

Add a run step to execute tests on the container you just built and pushed.
Add a trigger so that when a pull request is opened, Gitness can automatically trigger pipeline execution.
Add a slack plugin step so that failed tests trigger slack webhook and notification.
Add another run step so that when all tests pass, the image tag is retagged and pushed to a private image registry. You’ll need authentication at this step.

Security Considerations

While ttl.sh's no-credential, ephemeral approach offers flexibility and simplicity for CI environments, it introduces unique security considerations. The convenience of pushing images without credentials is counterbalanced by potential security risks — namely, the possibility of unauthorized image pulls. You can mitigate these risks through short-lived images and using UUIDs for image names:

Short-Lived Images: By design, images in ttl.sh are ephemeral. Setting a short expiration time for an image means it's available for a limited time window, reducing the risk exposure period.

Use of UUIDs: Incorporating UUIDs into image tags significantly lowers the risk of unauthorized access. The randomness and complexity of UUIDs make it exceedingly difficult for someone to guess the image name and pull it without authorization.

However, for production environments, organizations might consider deploying a private version of ttl.sh. This allows for more control over the security aspects, such as network isolation, access control, and auditing capabilities.

Takeaway

Integrating ttl.sh and Gitness creates an ephemeral CI environment that streamlines the build and test process. It helps to avoid the accumulation of outdated images and keeps the pipeline lean. This method is not just about speed; it's about maintaining a manageable and efficient development workflow. Adopting these tools can lead to more frequent and dependable software delivery for your engineering teams. Check out and follow Harness YouTube channel for

Secure Container Image Signing with Cosign and OPA

Dewan Ahmed — Tue, 28 Nov 2023 22:15:55 +0000

As the adoption of containers in modern development continues to grow, ensuring the integrity of container images has become a pivotal aspect of application deployment strategies. In this video, Harness Developer Advocate Dewan Ahmed demonstrates how to leverage the combined power of Cosign and OPA for the secure deployment of container images to your Kubernetes cluster.

You can also read the text version of this tutorial.

From Zero to Kubernetes Deployment: Harness Continuous Delivery in Action

Dewan Ahmed — Fri, 17 Nov 2023 14:45:48 +0000

Harness Continuous Delivery pipelines enable you to orchestrate and automate your deployment workflows, allowing you to push updated application images to your target Kubernetes cluster seamlessly.

In this video, Dewan Ahmed, a Developer Advocate at Harness, demonstrates how to install a Harness delegate and create entities such as Harness secrets, connectors, environments, services, and pipelines. Dewan guides you through a successful pipeline execution with a manual trigger and then shows how to create a pipeline variable to configure the Kubernetes namespace to be provided during pipeline execution.

Securing CI/CD Images with Cosign and OPA

Dewan Ahmed — Wed, 15 Nov 2023 14:01:27 +0000

With the growing adoption of containers in modern development, ensuring the integrity of container images has become central to application deployment strategies. Rapid deployment offers agility, but it also presents security challenges. Container image signing addresses these challenges, allowing engineering teams to verify that the deployed images are authentic and unchanged.

However, an authentic image is just one piece of the puzzle. Making sure these images meet organizational standards is important, and policy engine tools are crucial for that. By establishing and enforcing clear guidelines, these tools pave the way for a secure and streamlined deployment workflow.

In essence, container image signing involves adding a digital stamp to an image, affirming its authenticity. This digital assurance guarantees that the image is unchanged from creation to deployment. In this blog, I'll explain how to sign container images for Kubernetes using Cosign and the Open Policy Agent. I will also share a tutorial that demonstrates these concepts.

Choosing the Right Tools for Image Signing and Verification

Choosing the right tools for container image signing and verification is important in CI/CD pipeline security. Let's walk through the available options to find the one that best meets your needs for securing container images.

Image Signing and Verification Tools:

Notary v1, previously know as Docker Content Trust, uses The Update Framework (TUF) but has some issues with signature portability and storage. Though it established a solid foundation in image signing, it lacks some of the enhanced security features found in more recent tools.

Grafeas: While Grafeas offers a comprehensive solution for the software development lifecycle, it is not designed for public or open-source software (OSS) image verification. It's better suited for first-party integration, particularly with Google Kubernetes Engine (GKE).

Notary v2: The evolution to Notary v2 brought improvements in signature portability and integration with third-party key management solutions. However, it does not provide a certificate authority, leaving public key discovery for open-source image verification as an unresolved issue.

The Update Framework (TUF): TUF is a framework, not a tool, designed to enhance the security of software update systems. It focuses on resilience against key compromises and attacks, employing verifiable records to verify the authenticity of update files. TUF's flexibility and integration ease make it a foundational element in securing software updates, though it's not a direct image signing tool like the others.

Cosign: In this context, Cosign from the Sigstore project offers a compelling solution. Its simplicity, registry compatibility, and effective link between images and their signatures provide a user-friendly and versatile approach. The integration of Fulcio for certificate management and Rekor for secure logging enhances Cosign's appeal, making it particularly suitable for modern development environments that prioritize security and agility.

Cosign's strength lies in its verification process, which is vital in the CI/CD pipeline. When integrated with policy engines like the Open Policy Agent (OPA), Cosign ensures not only the authenticity of container images but also their compliance with organizational standards. Additionally, Cosign provides enhanced support for SPIFFE, GitHub Actions, or service account identities, and it includes warnings for signing OCI images by tag, highlighting the risks associated with tag mutability.

Policy Enforcement Options:

Open Policy Agent (OPA): OPA's strength lies in its ability to define fine-grained policies as code, offering granular control over policy enforcement. Integrated seamlessly with Kubernetes, it enforces policies in real-time, preventing unauthorized image deployments.

Kubernetes Admission Controllers: These controllers are a native part of the Kubernetes ecosystem, making them a straightforward choice for Kubernetes users. They excel at validating and enforcing policies for various Kubernetes resources, including pods, making them scalable for large deployments.

When choosing tools for container image signing and policy enforcement, it's important to balance functionality, scalability, and community support. Among the options discussed, Cosign and OPA (Open Policy Agent) stand out for their effectiveness in securing containerized applications.

Cosign simplifies the process of image signing and verification. It offers a user-friendly approach that caters to developers and security teams alike. With a growing user community and easy integration, it's a practical choice for securing container images.

OPA, on the other hand, excels in policy enforcement within Kubernetes environments. It enables you to define and enforce policies as code, granting you fine-grained control over image deployments.

In the upcoming section, let's delve into architectural diagrams and explore how the combination of Cosign and OPA offers a practical approach to sign and verify container images while enforcing them using a general policy engine.

Architectural Diagrams: Securing Container Images and Deployment

Traditional cryptography uses public-private key pairs for signing and verifying images. However, modern practices, like cosign, prefer keyless signing. In this approach, an OIDC (OpenID Connect) provider is utilized, making the process more streamlined and secure, as it removes the complexities of key management.

The process starts with an architect selecting a trusted public base image as the foundation for their application. Let's dive into two distinct but interconnected flows that demonstrate how container image signing with Cosign and policy enforcement using Open Policy Agent (OPA) play a crucial role in ensuring a secure deployment pipeline.

The process starts with an architect who selects a trusted Public Base Image as the foundation for their application. To ensure the image's integrity, the architect runs a vulnerability scanning process. This step identifies and addresses any security vulnerabilities within the base image.

Next, any unnecessary or unused components are removed from the image, minimizing potential attack vectors. The architect adds the necessary libraries, dependencies, and software packages required for the application to run efficiently. Then, the image undergoes rigorous security testing to ensure that it meets security standards and aligns with organizational policies.

The final step involves image signing with Cosign, a tool specifically designed for image signing. Cosign adds a digital stamp to the image, affirming its authenticity. This signature plays a crucial role in verifying the image's integrity during deployment. The signed image is then pushed to an artifact registry as a new, secure base image.

In the deployment phase, a developer selects a base image for their application, either an Unsigned Public Base Image or a Signed Public Base Image. If the developer chooses an Unsigned Public Base Image, the deployment process checks the image's signature and evaluates it against predefined policies using Open Policy Agent (OPA). Due to the lack of a valid signature, the image is Denied For Deployment.

On the other hand, if the developer opts for a Signed Public Base Image, the same verification and policy enforcement process is applied using OPA. This time, since the image has a valid signature, it is Allowed For Deployment.

Hands-On: Deploying Secure Containers with Cosign and OPA

Having explored the strengths of Cosign and OPA for container image signing, you're now ready to apply these tools in a practical scenario. To get a real-world feel for how these technologies can enhance the security of your Kubernetes deployments, follow this hands-on tutorial. It will guide you through the process of securely deploying container images using the combined capabilities of Cosign and OPA in your Kubernetes cluster. The Harness Software Supply Chain Assurance (SSCA) module addresses the challenges of securing your software supply chain, including image signing and verification. To explore how SSCA can enhance your pipeline security, check it out.

KubeCon 2023 NA Recap - Developer Experience is Monumental

Dewan Ahmed — Mon, 13 Nov 2023 15:26:35 +0000

KubeCon + CloudNativeCon North America has wrapped up, and the buzz it generated is still in the air. With the halls of Chicago's convention center now quiet, those of us who attended are left with a wealth of new insights and ideas. The conference proved to be a fertile ground for learning and networking, with cloud native professionals / technologists from leading open projects from around the world sharing their knowledge.

At Harness, we were right there in the thick of it, participating in the Co-Located events and engaging with the community. Our team dove into conversations, contributed to discussions, and absorbed a lot of feedback on everything from technical workflows to project management.

For those who couldn't make it, or for attendees who want to revisit the highlights, our recap will bring you the key points from KubeCon 2023. We'll cover the sessions that made us think, the keynotes that motivated us, and the informal chats that often lead to the best ideas.

The Pulse of Cloud Native: KubeCon Session Breakdown

As cloud native technology matures, the focus of KubeCon sessions has honed in on a few key areas: the construction of layered abstractions over Kubernetes, addressing the unique challenges within specific industry verticals, and the role of AI in cloud native spaces. While it's impossible to cover all the sessions within this recap, we've curated a subset of the extensive and enlightening discussions for you. Here are some highlights:

Argo and Flux

Since deploying Kubernetes to cloud or on-prem environments has largely been standardized, the community is now working on adding orchestration on top of orchestration. The word “Scale” has featured in the session title at no fewer than 30 talks. Argo and Flux have taken center stage as key orchestration tools for declarative GitOps management of Kubernetes workloads.

These projects have been in the Cloud Native Computing Foundation for a few years now. Organizations are beginning to report their success in using them to manage not just hundreds or thousands of nodes, but hundreds - to - thousands of clusters and deployed applications. The piling up of abstraction layers is leading to an everything-as-code approach for which a multitude of vendors and community projects have stepped up to offer their resources and solutions.

Challenging Verticals

With the maturing of deployment architectures, a marked shift has also taken place to solve problems in industry verticals where cloud native has been a challenge. One example is deploying to secure or air-gapped environments. Crossing the air gap remains a wrench in the automation process. Organizations that operate in that space developed a couple of practical solutions:

Develop as much as possible in unclassified environments while ensuring the application is as self-contained and portable.
Closely engage with the open source community so that CVE’s are identified and acted on quickly prior to crossing the air gap.

AI

As might be expected, artificial intelligence was prominent in several sessions. The industry is still determining whether AI (namely LLMs) has reached the peak of the hype cycle, and what role it plays in cloud native. Compared to the general buzz, there were relatively few sessions specifically on AI/ML. Even then, those sessions were less about consuming generative AI, and more about deploying custom AI/ML applications.

The focus then was largely on the massive compute resources required for AI training and deployment. The community is still catching up when it comes to developing open source alternatives to the dominant proprietary stacks, but some projects have emerged to propose community-driven approaches to model training (federated LLMs for example).

On the Decline: Yesterday's Hot Topics Take a Backseat

Since last year’s KubeCon, a few parts of the ecosystem have started to either mature or decrease in popularity. While by no means on the way out, we noticed a few of the following were less emphasized in this year’s talks and vendor showcases.

Pure Kubernetes Implementations

While the number of certified Kubernetes distributions have ticked back up to historic highs, vendors are now focusing on the application layer rather than leading with K8s. Deploying Kubernetes to the public and private cloud is now largely standardized. Organizations are therefore now turning their infrastructure attention to edge cases such as maintaining legacy VM workloads, or else are moving up the stack into serverless.

Unchecked Developer Autonomy

Security is no longer optional. The rise of software supply chain attacks and rise in open source and cloud resources have spurred companies to balance developer productivity with the need to secure their data and environments. While developer experience is still paramount, the ecosystem is realizing that “you build it, you run it” should not mean “no guardrails”. Organizations are moving toward a more centralized experience, implementing managed platforms like backstage. Tools like keycloak and OPA have risen to “must-haves” in cluster administration and permission management.

Dialogues at the Booth: Harnessing Customer Voices

Harness has been a long time advocate of the Cloud Native community and ecosystem, and is proud to help sponsor KubeCon. We had a multitude of great conversations across our ArgoCon, BackstageCon, Litmus Chaos, and main-floor KubeCon booths. Helping support the next generation of workloads by reducing friction in software delivery are top of mind for many individuals we got to interact with over the course of the week.

Engineering pillars and experiences that previously in a non-cloud-native world such as security, scalability, and robustness would be afterthoughts are front and center during the development cycle now thanks to the ever increasing burden of shifting left. The Harness Platform is well poised to help reduce significant toil as more expertise is disseminated across your evolving delivery pipelines. We are excited to see what is in store over the course of the year before KubeCon 2024.

Next Stop, Salt Lake City For KubeCon NA 2024

As we wrap up our journey at KubeCon 2023, we're taking a moment to reflect on the rich tapestry of ideas and innovations that were shared. This year's event has not only reinforced the pivotal role of cloud native technologies in shaping the future of software but also highlighted the evolving landscape where some trends rise and others give way to more pressing innovations.

At Harness, we've had the privilege of engaging directly with the community that's pushing these boundaries. Our conversations at the booth brought us face to face with the pulse of the industry—where the enthusiasm for Kubernetes (K8s) is as robust as ever, and the love for our four-legged friends (K9s) continues to bring smiles and a sense of camaraderie.

‍

Looking beyond the current cloud native vistas, we're excited about what's on the horizon. As we set our sights on KubeCon in Salt Lake City as the next flagship conference gathers adopters of cloud native technology, we carry forward the insights and feedback we've garnered here in Chicago. The dialogue doesn't end with the closing of the conference doors; it's just the beginning. We invite you to continue these conversations with us, explore how Harness can streamline your DevOps journey, and share in the excitement for what's to come.

Until we meet again, let's keep pushing the envelope in our respective fields, inspired by the collective wisdom of KubeCon 2023. And remember, whether it's for your infrastructure needs or a harness for your K9, Harness is here to support you. See you in Salt Lake City!

Nick, Dewan, Ravi

Harness Developer Hub - Ease of Authoring with Git Triggers

Dewan Ahmed — Fri, 27 Oct 2023 13:21:39 +0000

It’s been about a year since we launched Harness Developer Hub [HDH] in Beta. Today, HDH is GA and is serving tens of thousands of unique visitors every month and hundreds of thousands of pageviews every month all across the globe. All of this while supporting hundreds of contributors with varying levels of skills. The traffic and number of contributors in the public repository continues to grow as we expand the capabilities of HDH.

Looking at how HDH is architected, HDH is a Docursarus Implementation. Our site embraces documentation-as-code as a paradigm and is no different than any other modern TypeScript [Javascript] based application. We have an application that multiple contributors need to contribute to and needs to be built and deployed all throughout the day.

Over the previous year we have made two shifts in how we build and deploy. We now treat every commit as a potential release and build multiple times throughout the day with every git commit and also deploy multiple times throughout the day with every merge to our main branch. Let’s look at our current solution and then jog down memory lane how we evolved.

Current HDH Pipeline Strategy

We leverage several Harness capabilities to deliver HDH to the world.

Starting at the Repository

Our source code management solution is the source of truth for HDH and the genesis of changes being published. We have webhook events that fire on several SCM events to Harness to process.

Branch or tag creation/deletion.
Pull Request Events - Created/merged/synchronized/updated/closed
Git Pushes

These events are then processed by Harness.

Harness Build and Deploy - Conditionally from Git Hooks in the Cloud

Our goal is to provide preview/ephemeral builds for changes that are represented in a Pull Request. To do this, we need to remotely build the Docusarus instance which leverages Yarn and NPM to facilitate the build. We build on every net new commit to the PR.

We build via a Harness Cloud [hosted] build node so we do not have to manage build infrastructure and dependencies on the build node. We also leverage for performance Cache Intelligence on a conservative estimate sped up our builds more than 30%. From when we implemented the current setup, we have had over 9000 builds.

From a deployment standpoint, we deploy to our static host which is Netlify. The flexibility and extensibility of Harness allows us to bring a plugin that interacts with Netlify’s APIs. We have a decision that we make in JEXL if a build needs to head to a preview environment or if a build needs to be published to production.

Preview Logic [if branch is not main]:

<+trigger.event>.equals("PR") && <+trigger.branch>!~"/^main$/"

Production Logic [if branch is main]:

<+trigger.event>.equals("PUSH") && <+trigger.targetBranch>.equals("main")

Configuring this Harness Trigger, here is our YAML configuration looking out for a few events.

source:
type: Webhook
spec:
type: Github
spec:
type: PullRequest
spec:
connectorRef: hdh_gh_connector
autoAbortPreviousExecutions: false
payloadConditions: []
headerConditions: []
actions:
- Open
- Synchronize
- Reopen

Based on the condition, we fire a slightly different request to the Netlify API. Once we get the results of the Netlify API call, we comment back to the GitHub PR. This allows the contributor to preview their work in a live site if a preview flow is executed. In totality, the Pipeline looks as follows in the Harness Editor:

For example the Cache Intelligence step is easy to weave in during the Build Stage. Once execution will look as follows in the Harness UI:

Pipelines are designed to evolve. We had two other renditions of the Pipeline which we optimized over the year to produce what we are currently leveraging today.

Pipelines Should Evolve

We have embraced two principals as we evolved our pipelines. The KISS Principle to take a more simplistic approach and DRY Principle to cut out duplicate steps/tests. Our second rendition was Kubernetes heavy for the static site before we optimized on calling the Netlify APIs directly for a preview build; we used to maintain our own preview environment when Netlify provided this out of the box. Because we learned of this feature, we were able to easily modify our HDH Pipeline to leverage this new methodology.

If you like to continuously improve your software delivery capabilities, I would implore you to consider signing up and using the Harness Platform to help you with your goals.

Cheers,

-Ravi

Need for Automation - GitOps at Scale

Dewan Ahmed — Wed, 18 Oct 2023 14:08:20 +0000

Building a skyscraper is a lot like building software. Initially, you might experiment with materials on a different site or check how beams handle stress. Once confident, you lay the foundation and then start building upwards. Similarly, in software development, you kick things off with a proof of concept (POC) on a small scale. Once you're convinced of the tooling's capacity, you scale up to production. GitOps is no different.

Beginning typically with trials and using Git as the backbone for declarative infrastructure and applications, the challenge arises as you scale: how do you efficiently manage everything? And just as you'd need machinery to help construct a skyscraper, in the world of GitOps, tools like Terraform become invaluable, especially when setting up or "bootstrapping" these automation tools. In this blog, we'll navigate through the early, middle, and advanced stages (day 0, day 1, and day 2) of GitOps and explore how Terraform can simplify the scaling process for day 2.

Day 0 GitOps Challenges: Laying the Groundwork

Day 0 in GitOps can be likened to the preparatory phase of constructing a skyscraper where materials and designs are rigorously tested. At this initial stage, you're not delving into the complexities of scaling. Instead, you're laying the groundwork, familiarizing yourself with the fundamental principles of GitOps.

A practical starting point is to set up a basic GitOps workflow to deploy a Kubernetes manifest from a single git repository to a single cluster. At this juncture, the majority of tasks can be managed directly through the CLI or UI. The environment is simple enough that manually invoking commands or using native interfaces is sufficient. While tools like Terraform offer automation benefits, they're not essential at this stage. The primary objective of Day 0 is exploration: understanding the tools, assessing their fit, and choosing the best ones for your specific needs. The experience and knowledge garnered here form the bedrock for the challenges and complexities of the stages ahead.

Day 1 GitOps Challenges: Setting the Foundation

Day 1 in GitOps is like getting the ground ready for building a skyscraper. After your first dive into GitOps, the real work begins.

A big task is figuring out how to keep application code and configuration separate. When your app has services across multiple Git repositories, how do you make sure everything deploys smoothly and works together?

Next, as you start automating your CI pipeline, there's a tricky part: if you push manifest changes to a Git repository, you might accidentally start an endless cycle of build jobs and commits. You need to set up your pipelines right to avoid this mess.

Also, as you add more to your app, developers will need to deploy different services, like deployments and stateful sets. Each of these might have its own settings. While tools like Kustomize can help, if you're not careful, you can end up with "overlay folders hell". It becomes hard to know what's where and manage everything.

Day 1 is all about sorting out these early challenges, setting things up so you're ready for bigger tasks and scaling in the next stages.

Day 2 GitOps Challenges: Automation Complexities

As we progress to Day 2 of our GitOps journey, we encounter several challenges:

Bootstrapping: Tools like ArgoCD are great for automating deployments, but how do we set up ArgoCD itself? This is the classic bootstrapping problem in the GitOps context.
Tool Reliability: Automation is essential for system consistency. But this is true ONLY if the automation tool is reliable and consistent.
Understanding the End Goal: With declarative tools like Kubernetes and Terraform, we define the desired outcome. However, we need to clearly understand and define that end state first.

Terraform offers solutions to these Day 2 challenges:

Tackling the Bootstrapping Dilemma: One of Terraform's core strengths is its ability to set up and provision tools. So, when it comes to initializing GitOps tools like ArgoCD, Terraform can declaratively define the infrastructure, software installations, and configurations. This helps sidestep the "chicken or the egg" problem because Terraform is the tool you use to deploy your deployment tools.

Ensuring Reliability and Consistency: Terraform is designed with idempotency in mind, meaning you can run the same configuration multiple times and get the same result. This ensures that your infrastructure is reliable and consistent. If something drifts from the desired configuration, Terraform will recognize it and can revert it back, ensuring that the tool itself becomes a reliable part of the automation process.

Setting Clear End Goals with a Declarative Approach: With Terraform, you define your infrastructure as code in a declarative manner. You specify what you want the end state to look like, and Terraform figures out how to achieve it. This directly answers the challenge of needing to know your end state: you define it in your Terraform configuration, and the tool takes care of making it happen.

For a closer look at how Terraform can boost GitOps automation, the following webinar provides more insights:

Build software like skyscrapers

Just as constructing a skyscraper involves meticulous planning, testing, and execution in phases, the journey of GitOps follows a similar trajectory. We start with small-scale experiments, lay a robust foundation, and then scale up, tackling complex automation challenges along the way. Each stage has its hurdles, but as we've seen, with the right tools and strategies, these challenges can be addressed effectively. Terraform, in particular, stands out as a battle-tested tool in this journey, providing solutions to many of the complexities of Day 2.

If you're keen on elevating your GitOps game, especially in terms of automation, I strongly recommend giving the Harness Terraform provider a spin. Let's build software like skyscrapers - strong, scalable, and awe-inspiring!