Forem: Harness

Speed Up Your CI Pipelines with Docker Layer Caching

Dewan Ahmed — Thu, 23 Jan 2025 16:03:21 +0000

In modern software development, speed and efficiency are paramount. Long build times can slow down releases and hinder productivity. Docker layer caching is a powerful technique that helps optimize builds by reusing previously created image layers, reducing redundant processing. In this blog, we'll explore how Harness CI features Docker Layer Caching (DLC) to enhance build performance and streamline your CI/CD pipelines.

DLC and Multi-stage Builds

Every instruction in a Dockerfile creates a layer in the final image. Docker caches these layers to avoid rebuilding them unnecessarily, which can save significant time and reduce infrastructure costs. However, when a layer changes (e.g., modifying a file copied with COPY), Docker invalidates the cache for that layer and all subsequent layers, requiring them to be rebuilt. Understanding and optimizing layer usage helps in writing more efficient Dockerfiles, achieving faster build times, and lowering compute costs.

A multi-stage Dockerfile allows you to use multiple FROM statements to break the build process into stages. This helps keep the final image lightweight by copying only the necessary files from one stage to another, discarding anything unnecessary. It speeds up builds by leveraging layer caching, reducing the need to re-run expensive steps. Plus, it enhances security by minimizing the final image's attack surface and keeps the Dockerfile organized by separating concerns.

Harness CI Intelligence: Docker Layer Caching

Harness CI Intelligence optimizes Docker builds by leveraging Docker Layer Caching (DLC) to reuse unchanged image layers, significantly reducing build times and resource costs. When enabled, DLC restores previously built layers, avoiding redundant processing and speeding up the build and push process. Harness CI supports DLC across both Harness Cloud and self-managed infrastructure, providing flexibility in managing cache storage. This intelligent caching mechanism enhances CI/CD efficiency by minimizing infrastructure usage and improving developer productivity.

Benchmarking Build and Push to Docker

Check out the following video for a demo on running a build and push to Docker step for a Go repository using GitHub Actions and Harness CI, with Harness CI achieving an 8X improvement in build times.
Speed Up Your CI Pipelines with Docker Layer Caching

The following chart summarizes the performance comparison of this benchmark (Harness CI with DLC vs. GitHub Actions):

Conclusion

Implementing Docker Layer Caching in your CI/CD pipelines can lead to significant improvements in build performance, cost savings, and overall development efficiency. By reusing unchanged layers and minimizing redundant processing, Harness CI helps teams accelerate their workflows while optimizing infrastructure usage. Whether you're running builds in Harness Cloud or a self-managed environment, enabling DLC ensures faster feedback loops and a smoother development experience. Start leveraging Docker Layer Caching today to speed up your CI pipelines and focus on delivering value faster.

End-to-end MLOps CI/CD pipeline with Harness and AWS

Dewan Ahmed — Wed, 01 May 2024 13:29:10 +0000

MLOps tackles the complexities of building, testing, deploying, and monitoring machine learning models in real-world environments.

Integrating machine learning into the traditional software development lifecycle poses unique challenges due to the intricacies of data, model versioning, scalability, and ongoing monitoring.

In this tutorial, you'll create an end-to-end MLOps CI/CD pipeline that will:

Build and push an ML model to AWS ECR.
Run security scans and tests.
Deploy the model to AWS Lambda.
Add policy enforcement and monitoring for the model.

The story

This tutorial uses a fictional bank called Harness Bank. Assume that this fictional bank recently launched a website where clients can apply for a credit card. Based on the information provided in the form, the customer's application is approved or denied in seconds. This online credit card application is powered by a machine learning (ML) model trained on data that makes the decision accurate and unbiased.

Assume that the current process to update this hypothetical ML model is manual. A data scientist builds a new image locally, runs tests, and manually ensures that the model passes the required threshold for accuracy and fairness.

In this tutorial, you'll automate the model maintenance process and increase the build and delivery velocity.

Design and architecture

Before diving into the implementation, review the MLOps architecture.

For this tutorial, assume you are given a Python data science project, and you are requested to do the following:

Build and push an image for this project.
Run security scans on the container image.
Upload model visualization data to S3.
Publish model visualization data within the pipeline.
Run test on the model to find out accuracy and fairness scores.
Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model.
Deploy the model.
Monitor the model and ensure the model is not outdated.
Trigger the pipeline based on certain git events.
(Optional) Add approval gates for production deployment.

For this tutorial, assume that the data is already processed.

Prerequisites

This tutorial requires:

A Harness account with access to the Continuous Integration, Continuous Delivery, and Security Testing Orchestration modules. If you are new to Harness, you can sign up for free.
An AWS account, credentials, and a Harness AWS connector.
A GitHub account, credentials, and a Harness GitHub connector.

Prepare AWS

You need an AWS account with sufficient permissions to create/modify/view resources used in this tutorial.

Prepare AWS credentials.

This tutorial requires two sets of AWS credentials. One set is for a Harness AWS connector, and the other is for the AWS ECR scanner for STO.

You can use an AWS Vault plugin to generate AWS credentials for the AWS connector, and you can use the AWS console to generate the AWS Access Key ID, AWS Secret Access Key, and AWS Session Token, which are valid for a shorter time.

Save these credentials securely and make a note of your AWS account ID and AWS region.

⚠️ NOTE

If you are using a personal, non-production AWS account for this tutorial, you can initially grant admin access for these credentials. Once the demo works, reduce access to adhere to the principle of least privilege.

Create ECR repos. From your AWS console, navigate to Elastic Container Registry (ECR) and create two private repositories named ccapproval and ccapproval-deploy. Under Image scan settings, enable Scan on Push for both repositories.
Create an S3 bucket. Navigate to S3 and create a bucket named something like mlopswebapp. You'll use this bucket to host a static website for the credit card approval application demo, along with a few other artifacts.

Make sure all options under Block public access (bucket settings) are unchecked, and then apply the following bucket policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": "*",
               "Action": "s3:GetObject",
               "Resource": "arn:aws:s3:::YOUR_S3_BUCKET_NAME/*"
           }
       ]
   }

After making the bucket public, your bucket page should show a Publicly accessible flag.

From your AWS console, go to AWS Lambda, select Functions, and create a function from a container image using the following configuration:

Name: creditcardapplicationlambda
Container image URI: Select Browse images and find the ccapproval-deploy image. You can choose any image tag.
Architecture: x86_64
From Advanced: Select Enable function URL to make the function URL public. Anyone with the URL can access your function. For more information, go to the AWS documentation on Lambda function URLs.

Select Create function to create the function. You'll notice an info banner confirming that the function URL is public.

Prepare GitHub

This tutorial uses a GitHub account for source control management.

Fork the MLops sample app repository into your GitHub account.
Create a GitHub personal access token with following permissions on your forked repository:

repo/content: read+write
repo/pull requests: read
repo/webhooks: read+write

Create Harness secrets

Store your GitHub and AWS credentials as secrets in Harness.

In your Harness account, create or select a project to use for this tutorial.
In your project settings, select Secrets, select New Secret, and then select Text.
Create the following Harness text secrets:

git_pat - GitHub personal access token
aws_access_key_id - Generated from AWS console
aws_secret_access_key - Generated from AWS console
aws_session_token - Generated from AWS console
aws_vault_secret - Secret access key generated by Vault plugin

Make sure the Name and ID match for each secret, because you reference secrets by their IDs in Harness pipelines.

Create AWS and GitHub connectors

Create Harness connectors to connect to your AWS and GitHub accounts.

In your Harness project settings, go to Connectors.
Select New Connector, select the AWS connector, and then create an AWS connector with the following configuration:

Name: mlopsawsconnector
Access Key: AWS Vault plugin generated
Secret Key: Use your aws_vault_secret secret
Connectivity Mode: Connect through Harness Platform

Leave all other settings as is, and make sure the connection test passes.

Create another connector. This time, select the GitHub connector and and use the following configuration:

Name: mlopsgithubconnector
URL Type: Repository
Connection Type: HTTP
GitHub Repository URL: Enter the URL to your fork of the demo repo, such as https://github.com/:gitHubUsername/mlops-creditcard-approval-model
Username: Enter your GitHub username
Personal Access Token: Use your git_pat secret
Connectivity Mode: Connect through Harness Platform

Create the Harness pipeline

In Harness, you create pipeline to represent workflows. Pipeline can have multiple stages, and each stage can have multiple steps.

In your Harness project, create a pipeline named Credit Card Approval MLops.
Add a Build stage named Train Model.
Make sure Clone Codebase is enabled.
Select Third-party Git provider, and then select your mlopsgithubconnector GitHub connector. The repository name should populate automatically.
Select Set Up Stage.

In the following sections of this tutorial, you'll configure this stage to build and push the data science image, and you'll add more stages to the pipeline to meet the tutorial's objectives.

⚠️ NOTE

You can find a sample pipeline for this tutorial in the demo repo. If you use this pipeline, you must replace the placeholder and sample values accordingly.

Build, push, and scan the image

Configure your Train Model stage to build and push the data science image and then retrieve the ECR scan results.

Select the Infrastructure tab and configure the build infrastructure for the Train Model stage:

Select Cloud to use Harness Cloud build infrastructure.
For Platform, select Linux.
For Architecture, select AMD64.

Select the Execution tab to add steps to the stage.
Select Add Step, select the Build and Push to ECR step, and configure the step as follows:

For Name, enter Harness Training.
For AWS Connector, select your mlopsawsconnector AWS connector.
For Region, enter your AWS region.
For Account ID, enter your AWS account ID.
For Image Name, enter ccapproval.
For Tags, enter <+pipeline.executionId>. You can select the Input type icon to change the input type to expression (f(x)).
For Dockerfile (under Optional Configuration), enter Dockerfile_Training_Testing.

Select Apply Changes to save the step, and then select Save to save the pipeline.

Next, you'll add steps to your build stage to retrieve the results of the ECR repo security scan.

Because scanning is enabled on your ECR repositories, each image pushed to the repo by the Build and Push to ECR step is scanned for vulnerabilities. In order to successfully retrieve the scan results, your pipeline needs to wait for the scan to finish and then request the results.

Before adding the step to retrieve the scan result, use a Run step to add a 15-second wait to ensure that the scan is complete before the pipeline requests the scan results.

Select Add Step after the Harness Training step, and select the Run step.
For Name, enter Wait for ECR Image Scan.
For Command, enter the following, and then select Apply Changes to save the step.

   echo "ECR Image Scan In Progress..."
   sleep 15

Add an AWS ECR Scan step to get the scan results.

Select Add Step after the Wait step, and select the AWS ECR Scan step.
For Name, enter Security Scans for ML Model.
For Target/Name, enter ccapproval-ecr-scan.
For Variant, enter <+pipeline.executionId>. You can select the Input type icon to change the input type to expression (f(x)).
For Container Image/Name, enter ccapproval.
For Container Image/Tag, enter <+pipeline.executionId>.
For Region, enter your AWS region.
Under Authentication, use Harness expressions referencing your AWS credential secrets:
- For Access ID, enter <+secrets.getValue("aws_access_key_id")>.
- For Access Token, enter <+secrets.getValue("aws_secret_access_key")>.
- For Access Region, enter your AWS region.
- For Log Level, enter Info.
  - Under Settings, add the following key-value pair: AWS_SESSION_TOKEN: <+secrets.getValue("aws_session_token")>

Select Apply Changes to save the step, and then select Save to save the pipeline.

At this point, you can run the pipeline to test the Build stage.

Select Run Pipeline to test the Build stage. For Git Branch, enter main.
Wait while the pipeline runs, and then check your ccapproval ECR repository to find an image with a SHA matching the pipeline execution ID. Select Copy URI to copy the image URI; you'll need it in the next section.
Make sure the image scan also ran. In the Harness Build details, you can find the scan results in the AWS ECR Scan step logs. For example:

   Scan Results: {
       "jobId": "xlf06YX6a8AupG_5igGA6I",
       "status": "Succeeded",
       "issuesCount": 10,
       "newIssuesCount": 10,
      "issuesBySeverityCount": {
           "ExternalPolicyFailures": 0,
           "NewCritical": 0,
           "NewHigh": 1,
           "NewMedium": 5,
           "NewLow": 4,
           "NewInfo": 0,
           "Unassigned": 0,
           "NewUnassigned": 0,
           "Critical": 0,
           "High": 1,
           "Medium": 5,
           "Low": 4,
           "Info": 0,
           "Ignored": 0
       }
   }

You've successfully completed the first part of this tutorial: Configuring a Build stage that builds, pushes, and scans a trained data science image.

Continue the tutorial in the next sections and continue building your MLOps pipeline.

Test and upload artifacts

Add another Build stage to your pipeline that will run tests, build a Lambda image, and upload artifacts to S3.

Edit your MLOps pipeline and add another Build stage after the Train Model stage. Name the stage Run test and upload artifacts and make sure Clone Codebase is enabled.
On the stage's Overview tab, locate Shared Paths, and add /harness/output.
On the Infrastructure tab, select Propagate from existing stage, and select your Train Model stage.
On the Execution tab, add a Run step to run pytest on the demo copebase. Select Add Step, select the Run step, and configure it as follows:

For Name, enter pytest.
For Shell, select Sh.
For Command, enter:

   pytest --nbval-lax credit_card_approval.ipynb --junitxml=report.xml

Under Optional Configuration, locate Container Registry, and select your mlopsawsconnector AWS connector.
Under Optional Configuration, locate Image, and enter the image URI from your Train Model stage execution with the image tag replaced with <+pipeline.executionId>. For example:

   AWS_ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/AWS_ECR_REPO_NAME:<+pipeline.executionId>

The data science project includes two Dockerfiles: One for building the source and one for AWS Lambda deployment. Next, you'll add a step to build and push the image using the Dockerfile designed for AWS Lambda deployment.

Select Add Step and add a Build and Push to ECR step configured as follows:

For Name, enter Build and Push Lambda Deployment Image.
For AWS Connector, select your mlopsawsconnector AWS connector.
For Region, enter your AWS region.
For Account ID, enter your AWS account ID.
For Image Name, enter ccapproval-deploy.
For Tags, enter <+pipeline.executionId>.
Under Optional Configuration, locate Dockerfile, and enter Dockerfile_Inference_Lambda.

The pytest command from the Run step generates an HTML file with some visualizations for the demo ML model. Next, add steps to upload the visualizations artifact to your AWS S3 bucket and post the artifact URL on the Artifacts tab of the Build details page.

Select Add Step, and add an Upload Artifacts to S3 step configured as follows:

For Name, enter Upload artifacts to S3.
For AWS Connector, select your mlopsawsconnector AWS connector.
For Region, enter your AWS region.
For Bucket, enter your S3 bucket name.
For Source Path, enter /harness/output/model_metrics.html. This is where the model visualization file from the pytest step is stored.

Use the Artifact Metadata Publisher plugin to post the visualization artifact URL on the build's Artifacts tab.

Add a Plugin step after the Upload Artifacts to S3 step.
For Name, enter Publish ML model visualization.
For Container Registry, select the built-in Harness Docker Connector.
For Image, enter plugins/artifact-metadata-publisher.
Under Optional Configuration, locate Settings, and add the following key-value pairs:

   file_urls: https://S3_BUCKET_NAME.s3.AWS_REGION.amazonaws.com/harness/output/model_metrics.html
   artifact_file: artifact.txt

In addition to the model visualization, the pytest command also generates a shared_env_variables.txt file to export the model's accuracy and fairness metrics. However, this data is lost when the build ends because Harness stages run in isolated containers. Therefore, you must add a step to export the ACCURACY and EQUAL_OPPORTUNITY_FAIRNESS_PERCENT values as output variables.

After the Plugin step, add a Run step configured as follows:

For Name, enter Export accuracy and fairness variables.
For Shell, select Sh.
For Commmand, enter:

   # File path
   FILE_PATH="/harness/output/shared_env_variables.txt"

   # Read the file and export variables
   while IFS='=' read -r key value; do
       case $key in
           ACCURACY)
               export ACCURACY="$value"
               ;;
           EQUAL_OPPORTUNITY_FAIRNESS_PERCENT)
               export EQUAL_OPPORTUNITY_FAIRNESS_PERCENT="$value"
               ;;
           *)
               echo "Ignoring unknown variable: $key"
               ;;
       esac
   done < "$FILE_PATH"

   echo $ACCURACY
   echo $EQUAL_OPPORTUNITY_FAIRNESS_PERCENT

Under Optional Configuration, locate Output Variables, and add the following two output variables:

   ACCURACY
   EQUAL_OPPORTUNITY_FAIRNESS_PERCENT

Save the pipeline, and then run it. Again, use main for the Git Branch.
Wait while the pipeline runs, and then make sure:

Your ccapproval and ccapproval-deploy ECR repositories have images with SHAs matches the pipeline execution ID.
Your S3 bucket has /harness/output/model_metrics.html.
The URL to the model_metrics artifact appears on the Artifacts tab in Harness.

The output variable values are in the log for the Export accuracy and fairness variables step, such as:

   0.92662
   20.799999999999997

Congratulations! So far, you've completed half the requirements for this MLOps project:

[x] Build and push an image for this project.
[x] Run security scans on the container image.
[x] Upload model visualization data to S3.
[x] Publish model visualization data within the pipeline.
[x] Run test on the model to find out accuracy and fairness scores.
[ ] Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model.
[ ] Deploy the model.
[ ] Monitor the model and ensure the model is not outdated.
[ ] Trigger the pipeline based on certain git events.
Add approval gates for production deployment.

Continue on with policy enforcement in the next section.

Add ML model policy checks

In this section, you'll author OPA policies in Harness and use a Custom stage to add policy enforcement to your pipeline.

Harness Policy As Code uses Open Policy Agent (OPA) as the central service to store and enforce policies for the different entities and processes across the Harness platform. You create individual policies, add them to policy sets, and select the entities (such as pipelines) to evaluate those policies against.

For this tutorial, the policy requirements are that the model accuracy is over 90% and the fairness margin for equal opportunity is under 21%.

In your Harness project settings, go to Policies, select the Policies tab, and then select New Policy.
For Name, enter Check fairness and accuracy scores.
For How do you want to setup your Policy, select Inline.
Enter the following policy definition, and then select Save.

   package main

   default allow = false

   allow {
       input.accuracy >= 0.9
       input.fairnessScoreEqualOpportunity <= 21
   }

   deny[msg] {
       not allow
       msg = "Deny: Accuracy less than 90% or fairness score difference greater than 21%"
   }

Select the Policy Sets tab, and then select New Policy Set. Use the following configuration:

For Name, enter Credit Card Approval Policy Set.
For Entity Type that this policy set applies to, select Custom.
For On what event should the policy set be evaluated, select On Step.
Select Add Policy, and select your Check fairness and accuracy scores policy.
For What should happen if a policy fails?, select Warn and Continue.
Select Finish, and make sure the Enforced switch is enabled.

Edit your MLOps pipeline, and add a Custom stage after the second Build stage. Name the stage Model Policy Checks.
Select a Harness Delegate to use for the Custom stage.

The Build stages run on Harness Cloud build infrastructure, which doesn't require a Harness Delegate. However, Custom stages can't use this build infrastructure, so you need a Harness Delegate.

If you don't already have one, install a delegate. Then, on the Custom stage's Advanced tab, select your delegate in Define Delegate Selector.

Add a Shell Script step to relay the accuracy and fairness output variables from the previous stage to the current stage. Configure the Shell Script step as follows:

For Name, enter Accuracy and Fairness.
For Timeout, enter 10m.
For Script Type, select Bash.
For Select script location, select Inline.

For Script, enter the following:

  accuracy=<+pipeline.stages.Harness_Training.spec.execution.steps.Export_accuracy_and_fairness_variables.output.outputVariables.ACCURACY>
  fairness_equalopportunity=<+pipeline.stages.Harness_Training.spec.execution.steps.Export_accuracy_and_fairness_variables.output.outputVariables.EQUAL_OPPORTUNITY_FAIRNESS_PERCENT>

Under Optional Configuration, locate Script Output Variables, and add the following two variables:
- accuracy - String - accuracy
- fairness_equalopportunity - String - fairness_equalopportunity

⚠️ NOTE

While you can feed the output variables directly into Policy steps, this Shell Script step is a useful debugging measure that ensures the accuracy and fairness variables are populated correctly.

Add a Policy step after the Shell Script step.

For Name, enter Enforce Fairness and Accuracy Policy.
For Timeout, enter 10m.
For Entity Type, select Custom.
For Policy Set, select your Credit Card Approval Policy Set.

For Payload, enter the following:

  {
      "accuracy": <+execution.steps.Accuracy_and_Fairness.output.outputVariables.accuracy>,
      "fairnessScoreEqualOpportunity": <+execution.steps.Accuracy_and_Fairness.output.outputVariables.fairness_equalopportunity>
  }

Save the pipeline.

The next time the pipeline runs, the policy is enforced to check if the model accuracy and fairness margins are within the acceptable limits. If not, the pipeline produces a warning and then continues (according to the policy set configuration). You could also configure the policy set so that the pipeline fails if there is a policy violation.

If you want to test the response to a policy violation, you can modify the policy definition's allow section to be more strict, such as:

allow {
    input.accuracy >= 0.95
    input.fairnessScoreEqualOpportunity <= 19
}

Since the model accuracy is around 92% and the fairness margin is around 20%, this policy definition should produce a warning. Make sure to revert the change to the policy definition once you're done experimenting.

Deploy AWS Lambda function

In Harness, you can specify the location of a function definition, artifact, and AWS account, and then Harness deploys the Lambda function and automatically routes traffic from the old version of the Lambda function to the new version on each deployment. In this part of the tutorial, you'll update an existing Lambda function by adding a Deploy stage with service, environment, and infrastructure definitions.

Edit your MLOPs pipeline, and add a Deploy stage named lambdadeployment.
Deployment Type, select AWS Lambda, and then select Continue.
Create a service definition for the Lambda deployment.

Select Add Service.
For Name, enter creditcardapproval-lambda-service.
For Set up service, select Inline.
For Deployment Type, select AWS Lambda.
Under AWS Lambda Function Definition, for Manifest Identifier, enter lambdadefinition, and for File/Folder Path, enter /lambdamanifest.

After creating the manifest under Harness File Store, add the following to the service manifest, and select Save:

   functionName: `creditcardapplicationlambda`
   role: LAMBDA_FUNCTION_ARN

Replace LAMBDA_FUNCTION_ARN with your Lambda function's ARN. You can find the Function ARN when viewing the function in the AWS console.

Under the Artifacts section for the service definition, provide the artifact details to use for the lambda deployment:

Artifact Source Identifier: ccapprovaldeploy
Region: YOUR_AWS_REGION
Image Path: ccapproval-deploy
Value - Tag: <+input> (runtime input)

Create environment and infrastructure definitions for the Lambda deployment. On the Deploy stage's Environment tab, select New Environment, and use the following environment configuration:

   name: lambda-env
   type: PreProduction

From the lambda-env, go to the Infrastructure Definitions tab, and add an infrastructure definition with the following configuration:

   name: `aws-lambda-infra`
   deploymentType: `AwsLambda`
   type: AwsLambda
     spec:
       connectorRef: `mlopsawsconnector`
       region: YOUR_AWS_REGION

Select Save to save the infrastructure definition.
On the Deploy stage's Execution tab, add an AWS Lambda Deploy step named Deploy Aws Lambda for the name. No other configuration is necessary.
Save and run the pipeline. For Git Branch, enter main, and for Tag, enter <+pipeline.executionId>, and then select Run Pipeline.

You need to provide the image tag value because the service definition's Tag setting uses runtime input (<+input>).

While the pipeline runs, you can observe the build logs showing the lambda function being deployed with the latest artifact that was built and pushed from the same pipeline.

Test the response from the lambda function.

In your AWS console, go to AWS Lambda, select Functions, and select your creditcardapplicationlambda function.
On the Test tab, select Create new event, and create an event named testmodel with the following JSON:

   {
     "Num_Children": 2,
     "Income": 500000,
     "Own_Car": 1,
     "Own_Housing": 1
   }

Select Test to execute the function with your testmodel test event. Once the function finishes execution, you'll get the result with a Function URL.

Note the Function URL resulting from the lambda function test. This is the endpoint that your ML web application would call. Depending on the prediction of 0 or 1, the web application either approves or denies the demo credit card application.

Monitor the model

There are many ways to monitor ML models. In this tutorial, you'll monitor if the model was recently updated. If it hasn't been updated recently, Harness sends an email alerting you that the model might be stale.

Edit your MLOPs pipeline, and add a Build stage after the Deploy stage.

For Name, enter Monitor Model stage.
Disable Clone Codebase.
On the Infrastructure tab, select Propagate from existing stage and select the first Build stage.

Add a Run step to find out when the model was last updated.

For Name, enter Monitor Model step.
For Shell, select Sh.
For Command, enter:

   # GitHub repository owner
   OWNER="YOUR_GITHUB_USERNAME"

   # GitHub repository name
   REPO="mlops-creditcard-approval-model"

   # Path to the file you want to check (relative to the repository root)
   FILE_PATH="credit_card_approval.ipynb"

   # GitHub Personal Access Token (PAT)
   TOKEN=<+secrets.getValue("git_pat")>

   # GitHub API URL
   API_URL="https://api.github.com/repos/$OWNER/$REPO/commits?path=$FILE_PATH&per_page=1"

   # Get the current date
   CURRENT_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

   # Calculate the date 7 days ago
   SEVEN_DAYS_AGO=$(date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ")

   # Get the latest commit date for the file
   LATEST_COMMIT_DATE=$(curl -s -H "Authorization: token $TOKEN" $API_URL | jq -r '.[0].commit.committer.date')

   # Check if the file has been updated in the last 7 days
   if [ "$(date -d "$LATEST_COMMIT_DATE" +%s)" -lt "$(date -d "$SEVEN_DAYS_AGO" +%s)" ]; then
       export model_stale=true
   else
       export model_stale=false
   fi

Under Optional Configuration, add model_stale to Output Variables.

After the Monitor Model stage, add a Custom stage named Email notification. This stage will send the email notification if the model is stale.
Add an Email step to the last Custom stage.

For Name, enter Email.
For Timeout, enter 10m.
For To, enter the email address to receive the notification, such as the email address for your Harness account.
For Subject, enter Credit card approval ML model has not been updated in a week.
For Body, enter It has been 7 days since the credit card approval ML model was updated. Please update the model.
On the step's Advanced tab, add a conditional execution so the Email step only runs if the model_stale variable (from the Monitor Model step) is true:
- For Execute this step, select If the stage executes successfully up to this point.
- Select And execute this step only if the following JEXL Condition evaluates to true.
- Enter the following JEXL condition:
```
 <+pipeline.stages.Monitor_Model_Stage.spec.execution.steps.Monitor_Model_Step.output.outputVariables.model_stale> == true
```

Save and run the pipeline. For Git Branch, enter main, and for Tag, enter <+pipeline.executionId>.

Trigger pipeline based on Git events

So far, this tutorial used manually triggered builds. However, as the number of builds and pipeline executions grow, it's not scalable to manually trigger builds. In this part of the tutorial, you'll add a Git event trigger.

Assume your team has a specific requirement where they want the MLOps pipeline to run only if there's an update to the Jupyter notebook in the codebase.

In your MLOps pipeline, select Triggers at the top of the Pipeline Studio, and then select New Trigger.
Select the GitHub webhook trigger.
On the trigger's Configuration tab:

For Name, enter trigger_on_notebook_update.
For Connector, select your mlopsgithubconnector GitHub connector.
The Repository URL should automatically populate.
For Event, select Push.

Select Continue to go to the Conditions tab.

For Branch Name, select the Equals operator, and enter main for the Matches Value.
For Changed Files, select the Equals operator, and enter credit_card_approval.ipynb for the Matches Value.

Select Continue to go to the Pipeline Input tab.

The Git Branch should automatically populate.
For Primary Artifact, enter ccapprovaldeploy.
For Tag, enter <+pipeline.executionId>.

Select Create Trigger.

The trigger webhook should automatically register in your GitHub repository. If it doesn't, you'll need to manually register the webhook:

On the list of triggers in Harness, select the Link icon to copy the webhook URL for the trigger.

In your GitHub repository, navigate to Settings, select Webhook, and then select Add Webhook.
Paste the webhook URL in Payload URL.
Set the Content type to application/json.
Select Add webhook.

A green checkmark in the GitHub webhooks list indicates that the webhook connected successfully.

With the trigger in place, whenever you push a change to the credit_card_approval.ipynb file on the main branch, the MLOps pipeline runs. In the trigger settings, you could adjust or remove the Conditions (branch name, changed files, and so on) according to your requirements, if you wanted to use a Git event trigger in a live development or production scenario.

Add an approval gate before prod deployment

Your organization might require an approval gate for your CI/CD pipeline before an artifact is deployed to production. Harness offers built-in approval steps for Jira, ServiceNow, or Harness approvals.

Assume that you have a different image for production, and a different AWS Lambda function is deployed based on that container image. In your MLOps pipeline, you can create another AWS Lambda deployment stage with another AWS Lambda deploy step for the production environment and use the approval gate prior to running that production deployment stage.

To add the approval gate, add an Approval stage immediately prior to the Deploy stage that requires approval.

For Name, enter approval-to-prod.
For Approval Type, select Harness Approval.

Add an Approval step to the Approval stage.

For Name, enter approval-to-prod.
For Timeout, enter 1d.
Use the default Message.
For User Groups, select Select User Groups, select Project, and select All Project Users.

Save the pipeline.

Next time you run the pipeline, someone from the Harness project must approve the promotion of artifact to the production environment before the final Deploy stage runs.

Use the model in a web application

In a live MLOps scenario, the ML model would likely power a web application. While this app development is outside the scope of this tutorial, check out this animation that demonstrates a simple web application developed using plain HTML/CSS/JS. The outcome of the credit card application uses the response from the public AWS Lambda function URL invocation.

Conclusion

Congratulations! Here's what you've accomplished in this tutorial:

[x] Build and push an image for this project.
[x] Run security scans on the container image.
[x] Upload model visualization data to S3.
[x] Publish model visualization data within the pipeline.
[x] Run test on the model to find out accuracy and fairness scores.
[x] Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model.
[x] Deploy the model.
[x] Monitor the model and ensure the model is not outdated.
[x] Trigger the pipeline based on certain git events.
x Add approval gates for production deployment.

Now that you've built an MLOps pipeline on Harness and used the Harness platform to train the model, check out the following guides to learn how you can integrate other popular ML tools and platforms into your Harness CI/CD pipelines:

React App and Build Pipeline with Task and Gitness

Jim Sheldon — Fri, 05 Apr 2024 14:01:13 +0000

Task is a task runner/automation tool written in Go and distributed as a single binary.

A Taskfile is written in YAML, which may provide a shorter learning curve than alternative tools, like GNU Make. Task can also leverage Go's robust template engine.

Task can be useful in build and test pipelines. Logic for running builds and tests can be written once in Taskfile.yml (or other supported filenames), then used in both local development workflows and under automation.

Gitness is an open source development platform from Harness that hosts your source code repositories and runs your software development lifecycle pipelines.

In this guide, we'll craft a Taskfile.yml for a sample React app, suitable for both local development and integration with your pipelines powered by Gitness.

Prerequisites

Install Task and verify you can run task in your terminal.

This guide was tested with Task version 3.35.1.



$ task --version
Task version: v3.35.1 (h1:zjQ3tLv+LIStDDTzOQx8F97NE/8FSTanjZuwgy/hwro=)

Install Docker and verify you can run docker in your terminal.

This guide was tested with Docker version 24.0.7.



$ docker --version
Docker version 24.0.7, build afdd53b

Install Node and verify you can run node in your terminal.

This guide was tested with Node version 20.12.0.



$ node --version
v20.12.0

ℹ️ Note
Installing Node also installs `npm` and `npx` binaries.

Create React Application

Use the Create React App project to automatically generate the application.



npx create-react-app my-react-app

Your my-react-app directory structure will look like this.



my-react-app/
├── README.md
├── node_modules/
├── package-lock.json
├── package.json
├── public/
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src/
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── reportWebVitals.js
    └── setupTests.js

Change to the application directory.



cd my-react-app

Start the application.



npm start

Open http://localhost:3000 in your browser.

You now have a working React application! 🎉

In your terminal, type Ctrl-C to stop the process.

Create `Taskfile.yml`

Create a Taskfile.yml file in your my-react-app directory with this configuration.



version: '3'

tasks:

  npm-install:
    cmds:
      - npm install

  build:
    deps: [npm-install]
    cmds:
      - npm run build

  test:
    deps: [npm-install]
    cmds:
      - npm test

The build task runs npm run build, which creates a production build of your application.

The test task runs npm test, which runs unit tests. There is only one unit test in src/App.test.js.

Note that npm-install task is a dependency of the build and test tasks.

Build and Test

Run task build in your terminal.

By default, npm test will run tests interactively. To simulate running tests under Continuous Integration, set the CI environment variable.

Run CI=true task test in your terminal.

Note that task passed the CI=true variable to the npm test command.

Setup Gitness

Install Gitness
Create a project named demo
Either create a repository named my-react-app and push your sample app code, or import the repository jimsheldon/my-react-app, which I created for this guide

Create Pipeline

Open http://localhost:3000/demo/my-react-app/pipelines in your browser and select New Pipeline.

Enter build-and-test in the Name field (this will automatically populate the YAML Path field), then select Create.

Enter this Gitness pipeline in the YAML editor.



kind: pipeline
spec:
  stages:
    - name: task example
      type: ci
      spec:
        steps:
          - name: install task
            type: run
            spec:
              container: alpine
              script: |
                apk add curl
                sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d
                ./bin/task --version

          - name: build
            type: run
            spec:
              container: node:20
              script: |
                ./bin/task build

          - name: test
            type: run
            spec:
              container: node:20
              script: |
                ./bin/task test

This Gitness pipeline has a single stage with three steps. The install task step uses the alpine Docker image to install the task binary into the workspace, where it can be reused by the following build and test steps, which run in the node Docker image.

Select Save and Run.

Observe the pipeline execution.

Conclusion

This workflow improves the separation of concerns between local development and pipeline execution.

As long as the application can be built and tested with task build and task test commands, developers always keep the same workflow, and the pipeline does not need to be modified.

For example, the developers could switch to yarn by changing Taskfile.yml to use yarn commands rather than npm commands. This change would be transparent to developers, who would continue to run task build and task test, and the Gitness pipeline yaml would not need to be changed.

Next steps:

See more examples of using Task with Gitness
Create triggers to automatically run the pipeline when commits are pushed to the repository
Learn how Gitness manages your data
Learn how to manage Gitness projects and roles

Build and Push to GAR and Deploy to GKE - End-to-End CI/CD Pipeline

Dewan Ahmed — Tue, 02 Jan 2024 20:50:07 +0000

In this tutorial, you'll explore how to build a streamlined CI/CD pipeline using the Harness Platform, integrating the robust services of Google Artifact Registry (GAR) and Google Kubernetes Engine (GKE). GAR excels in managing and storing container images securely, while GKE offers a scalable environment for container deployment. The Harness Platform serves as a powerful orchestrator, simplifying the build and push process to GAR and managing complex deployments in GKE. You'll also cover implementing crucial approval steps for enhanced security and setting up Slack notifications for real-time updates, showcasing how these tools together facilitate a robust, streamlined CI/CD process.

Architectural and Pipeline Diagrams

Prerequisites

A Harness free plan. If you don't have one, sign up for free.
A GitHub account.
A Docker Hub account.
A GCP account with permissions for Google Artifact Registry and Kubernetes Engine.t
Access to a slack workspace and permissions to create a slack app.

There’s a bonus section in this tutorial where you’ll run security tests during the build process and create a policy for the deployment process. To follow this section, you’ll need a Harness enterprise account.

Required Setup and Configurations

Demo application

You can either fork Captain Canary Adventure (CCA) App or bring your own application (as long as it has a Dockerfile).

📝	This tutorial assumes the use of a fork of the CCA App. If you are using your own app, make the necessary changes.

GitHub and Docker authentication

Create a GitHub personal access token (PAT) that will have read access to the demo application repository. Create a Docker access token.

Image registry setup on Google Cloud Platform (GCP):

Enable the Artifact Registry API.
From Artifact Registry, click + CREATE REPOSITORY.
Give this repository a name - cca-registry, choose Docker as the format, Standard as the mode, location type Region (choose a region near you), Google-managed encryption key for encryption, have Dry Run selected, and click CREATE.

Kubernetes cluster setup with GKE:

Enable Kubernetes Engine API.
Create a GKE (autopilot) cluster by selecting a region near you.

GCP IAM and Service Account setup:

Create a GCP Service Account. Copy the email address generated for this service account. It will be in this format: SERVICE_ACCOUNT_NAME@GCP_PROJECT_NAME.iam.gserviceaccount.com.
Navigate to the artifact registry repository you created earlier, select it and click on Permissions tab. You need to create two types of access to the artifact registry repository - a public read access and a fine-grained write access. Click + ADD PRINCIPAL from the Permissions tab and paste the email address of the service account previously copied. Assign Artifact Registry Writer role to this principal. Next, click + ADD PRINCIPAL from the Permissions tab and type in allUsers for the principal and Artifact Registry Reader for the role. You might see a warning like this: > “This resource is public and can be accessed by anyone on the internet.”
Navigate to IAM & Admin, locate the service account, select it, and then click on ADD KEY → Create new key. Choose the JSON format, and a key for your service account will be downloaded to your computer. Exercise caution and refrain from sharing this key with anyone; treat it as you would a password.

Slack workspace and app setup:

To create a Slack app and incoming webhook, you'll need elevated privilege in that Slack workspace. Create a new Slack workspace for this tutorial and an incoming webhook for a specific channel.

Your newly created slack webhook will look like this: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

Treat this as sensitive information.

Harness entity setup

Create secrets:

a. GitHub Secret: Navigate to the Harness console. From Project Setup → Secrets, click + New Secret → Text, give the secret a name (for example, cca-git-pat) and paste in the previously created GitHub PAT.

b. Docker Secret: Similarly, create a docker secret (you can name it docker-secret) and use the previously created Docker PAT as the secret value.

c. Slack Webhook: Similarly, create a slack webhook secret (you can name it slack-webhook) and paste in the previously created slack webhook value.
Connectors in Harness help you pull in artifacts, sync with repos, integrate verification and analytics tools, and leverage collaboration channels. From Project Setup → Connectors → + New Connector, create the following connectors:

a. GitHub Connector: Harness platform connects to the source code repository using this connector. Give this connector a name (for example, cca-git-connector), choose URL type as Repository, connection type as HTTP, and paste in the forked Github Repository URL of the demo app. Use your GitHub username and the previously created GitHub secret for authentication. Select the connectivity mode as Connect through Harness Platform. The connection test should be successful.

b. Docker Connector: Harness platform pulls in the docker image for the Slack notification using this connector. Give this connector a name (for example, docker-connector), choose provider type as DockerHub, Docker Registry URL as https://index.docker.io/v2/, enter in your docker username and select the previously created docker secret. Select the connectivity mode as Connect through Harness Platform. The connection test should be successful.

c. Kubernetes Connector: Harness platform creates and manages resources on your GKE cluster using this connector. Give this connector a name (for example, gke-connector). Choose Use the credentials of a specific Harness Delegate… and click + Install new Delegate. The Harness Delegate is a service you run in your local network or VPC to connect all of your providers with your Harness account. Follow the instructions to install a delegate on your Kubernetes cluster and once the installation is complete, select the newly created delegate from the dropdown. The connection test should be successful.

d. GCP Connector: The GCP connector allows you to connect to your Google Cloud Platform resource and perform actions via Harness platform. Give this connector a name and select Specify credentials here under the Details section. Add a new secret name and upload the GCP service account key JSON file you previously downloaded. Select the connectivity mode as Connect through Harness Platform. The connection test should be successful.

Build and push image to GAR

First, let’s create the build and push image part of the pipeline. Click on Pipelines → + Create a Pipeline and give it a name (e.g., gar-gke-cicd-pipeline). Select the Inline option to store the pipeline definition in Harness, and then click Start.

Click on Add Stage and choose Build as the stage type. A Harness pipeline can consist of one or more stages. Give this stage a name (e.g., Push to GAR), select the Clone Codebase option (this should be enabled, by default), and choose the GitHub connector you previously created from the dropdown. The repository name should auto-populate. Click Set Up Stage.

Under Infrastructure, specify where you'll deploy your application. Select Cloud for Harness hosted builds and choose Linux/AMD64 for the Platform option.

Under Execution, click Add Step → Add Step and find Build and Push to GAR step from the Step Library. Name this step (e.g., BuildAndPushToGAR) and select the GCP connector you created earlier from the dropdown. When choosing the host, use the region selected when creating the image registry repository (e.g., northamerica-northeast1-docker.pkg.dev). Refer to Harness Developer Hub docs or GCP Artifact Registry docs for more details on selecting the region for GAR. Enter your GCP project ID under Project Id. For Image Name, use the image registry repository name followed by the application name in this format: cca-registry/cca-app. Use latest for now as the Tags. Click Apply Changes.

Now, click Run to execute the pipeline. Enter Master as the git branch name for the build (or main if you're not using the forked CCA app). A successful execution of the pipeline should resemble the following:

Deploy to GKE

If you're using a fork of the Captain Canary Adventure App, update the deployment.yaml before deploying the application to Kubernetes. The current YAML uses Harness variables, but since you're not there yet, you'll need to hardcode some values for now.

Assuming your GKE Artifact Registry repository name is cca-registry and the image name is cca-app, replace the current values in values.yaml with the following:

image:
  name: cca-registry/cca-app
  tag: latest

Click on + Add Stage in the pipeline and choose Deploy as the stage type. Give the stage a name (e.g., GKE Deploy), select Kubernetes as the deployment type, and click Set Up Stage.

Next, create a service to deploy. Choose Kubernetes as the deployment type, select the GitHub connector, and specify the paths for the manifests. For example, for the Captain Canary Adventure App, here are the manifest details:

Environments represent your deployment targets (such as QA or Prod). Each environment contains one or more Infrastructure Definitions that list your target clusters, hosts, namespaces, etc. Click on + New Environment, give this environment a name (e.g., cca-env), select Pre-Production as the environment type, and click Save.

Next, create an infrastructure definition. Click + New Infrastructure, under cluster details, select the GKE connector, provide a Kubernetes namespace where your application will be deployed (e.g., cca-ns), and click Save. For execution strategies, choose Rolling Deployment and click Use Strategy. Under optional configuration, select Enable Kubernetes Pruning. With this setting, Harness will use pruning to remove any resources present in an old manifest but no longer in the manifest used for the current deployment. You can find more information about this configuration on the Harness Developer Hub.

You’re all set! Click Save and then Run. Use master for the git branch. A successful pipeline execution will override the cca-app:latest image on your GAR repository and deploy this image to your GKE cluster.

Add Approval and Slack Notifications

In practical DevOps pipelines, gates are implemented to control artifact promotion to the production environment. Harness supports various types of approvals in Continuous Delivery (CD) pipelines. In this tutorial, you'll use the manual approval step.

Within the gke-deploy stage, click + Add Step before the Rollout Deployment step and find Harness Approval under Approval in the Step Library. Keep all default options, and you'll need to select the approver from the User Groups. Choose Project → All Project Users under user group selection. If you're the only member of this project, you'll be the sole approver. Click Apply Selected. Click Apply Changes for the manual approval step, and then click Save.

Now, let's add a notification stage so that whenever a deployment is approved in the CI/CD pipeline, a notification will be sent to a Slack channel, indicating who approved it.

From the pipeline, click on Add Stage after the gke-deploy stage, select Build as the stage type, give this stage a name (e.g., Notifications Stage), disable the Clone Codebase option, and click Set Up Stage. Under Infrastructure, choose Use a New Infrastructure → Cloud and Linux → AMD64 for the Operating System. Under Execution, click Add Step → Add Step and find Plugin in the Build section of the Step Library.

Name this step (e.g., Slack Notification), choose the Docker connector you previously created under Container Registry, for the image, use plugins/slack, and add the following key-values under Optional Configuration → Settings (assuming the id for your Slack webhook secret is slackwebhook).

Key	Value
webhook	`<+secrets.getValue("slackwebhook")>`
template	The deployment is moved to prod by `<+approval.approvalActivities[0].user.name>`

Notice the use of a Harness variable expression in the template, which retrieves the name of the approver from the previous stage.

Before running the pipeline, one more update is needed. Currently, every image built, pushed, and deployed has the same image tag, making it challenging to track based on the build number. Harness provides powerful built-in and custom variable expressions for various practical use cases.

Click on Variables for your pipeline and select + Add Variable at the pipeline level. Let’s add two variables:

Variable Name	Variable Value
imageName	cca-registry/cca-app
imageTag	`<+pipeline.sequenceId>`

For the imageTag, click the 📌 icon and select Expression. Every time you run the pipeline, the pipeline sequence ID will change, and subsequently, the image that will be built and deployed will also change.

Now that you’ve updated the pipeline to pass in the imageName and imageTag as variables, let’s update the codebase to replace the hardcoded values. Revert the changes to deployment.yaml and values.yaml you previously made. You'll observe that the values.yaml file will receive the imageName and imageTag during pipeline runtime, and then the deployment.yaml file will use those values from the values.yaml file.

Now, click Save and then Run. After a successful gar-build-and-push stage, you should see an image in the GAR repository with a numeric tag that matches the pipeline sequence ID. Right after, you should see a following prompt for approval:

You can (optionally) add a comment and click Approve. The pipeline should continue as before and you’ll see a deployment on your Kubernetes cluster. However, this time, you’ll see a slack notification resulting from your approval. To modify the text that appears on the notification, you can modify the template value in the slack plugin step settings.

Security Tests and Policy Enforcement (Bonus Section)

📝	These features are only available on Harness paid plans

Run OWASP Tests

You can scan your code repositories using OWASP Dependency-Check within a Harness pipeline. Within the gar-build-and-push stage, click on + Add Step → Add Step before the BuildAndPushGAR step. From the step library, find Owasp under the Security Tests section.

Use the following settings to configure the OWASP Dependency Check and click Apply Changes:

Setting Name	Value
Name	Owasp Tests
Scan Mode	Orchestration
Target.Name	cca-owasp-tests
Variant	master (this is the branch name for the repo)
Log Level	Info
Fail On Severity	Critical

You can have any string values for the step name and target.name. For the Variant, use the branch name for your codebase (e.g. master or main). Selecting Critical for Fail On Severity means that if there is any critical error, this test will fail and the pipeline execution will halt. You can check out the OWASP scanner reference to learn more on these configurations.

Click Save and then Run. If your codebase doesn’t have an OWASP critical bug, the pipeline should execute successfully. To enforce a fail on this OWASP scan, use a codebase with known vulnerabilities like WebGoat and you’ll see the OWASP scanner in action.

Add a policy to mandate approval step on deployment stages

Harness Policy As Code uses Open Policy Agent (OPA) as the central service to store and enforce policies for the different entities and processes across the Harness platform. In this section, you will define a policy that will deny a pipeline execution if there is no approval step defined in a deployment stage.

From Project Setup → Policies, follow the wizard to create a policy from the policy library. Use the Pipeline - Approval policy.

In the next screen, choose Project scope, trigger event On Run, and for the severity, choose Error & Exit. Next, click Yes to apply the policy.

Now, let’s remove the approval step from the gke-deploy stage. Click on Edit on the pipeline and click on the cross button on the Harness Approval step. Click Save and then Run. You should see the following error:

Add the Harness Approval back, save and run the pipeline and this time the pipeline should execute successfully. An end to end successful pipeline execution will look like this:

View the running application

While connected to your GKE cluster, execute the following command:

kubectl get svc -n cca-ns

This is assuming that you deployed the application to cca-ns namespace.

The output will be something like this:

NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
cca-app-service         LoadBalancer   34.118.227.33   34.152.47.53   80:30008/TCP   6d19h

Navigate to the IP address listed under the EXTERNAL-IP column for your case, and you should see a running Captain Canary Adventure application. As the application is running on port 80, you can omit the port number from the URL.

Homework Task

If you’d like to take this pipeline one step further, you can leverage caching to share data across stages because each stage in a Harness CI pipeline has its own build infrastructure. Check out how to save and restore cache from Google Cloud Storage (GCS).

Ephemeral CI environments using ttl.sh and Gitness

Dewan Ahmed — Mon, 18 Dec 2023 21:29:10 +0000

In the realm of software development, balancing continuous integration with maintaining quality is a significant challenge. ttl.sh and Gitness offer a solution. ttl.sh is an ephemeral Docker image registry that allows for the creation of temporary image tags with built-in expiry. Gitness, an open-source platform by Harness, simplifies the management of source code repositories and development pipelines. This blog post will explore how these tools can be used to create temporary CI environments to expedite development processes.

The Problem

Multiple developers working on different features can lead to merge conflicts and a bloated image registry. Traditional CI environments often retain Docker images from feature branches too long, which complicates image management and increases the likelihood of testing against outdated or incorrect images.

The Solution

The workflow in the diagram leverages ttl.sh and Gitness to manage these challenges:

Feature Branch Workflow: The creation of a pull request (PR) for a feature branch triggers a build in Gitness.
Image Creation: Gitness constructs a Docker image from the feature branch, using ttl.sh for tagging with a UUID and a time-to-live (TTL) limit. This process does not require credentials and maintains privacy.
Automated Testing: The image is put through automated tests. Should the tests fail, developers are notified; if they succeed, the process proceeds.
Image Promotion: After a PR passes all checks and is merged, the image is given a permanent tag and pushed to a central image registry, which at this point, requires proper credentials.

This approach ensures that only quality-assured images make it to the central registry, reducing the clutter and promoting a cleaner CI process.

Demo

Let's construct this setup step by step. Starting with pushing an image to ttl.sh through Gitness, I’ll guide you to resources for completing the remaining components of the system.

Begin with the Gitness documentation to set up your first project. You can start a new repository or link an existing one from GitHub or GitLab. The specific source code repository can be any; the essential requirement is the presence of a Dockerfile.

Navigate to "Pipelines'' within your repository and create a new pipeline. Gitness will suggest a sample pipeline. Replace it with the following YAML:

kind: pipeline
spec:
 stages:
   - name: build-and-push
     spec:
       platform:
         arch: amd64
         os: linux
       steps:
         - name: docker_build
           spec:
             inputs:
               repo: ttl.sh/xxxx-yyyy-nnnn-2a2222-4b44
               tags: 1h
             name: docker
           type: plugin
     type: ci
version: 1

Observe the image repo and tag. ttl.sh is the image repository name and the UUID (xxxx-yyyy-nnnn-2a2222-4b44) is the image name. You can replace the hard coded image name with a Gitness secret for a dynamic image name. Click on Secrets and add a new Gitness secret called random_image_name. Now you can update your pipeline YAML so that the image name is not fixed:

repo: ttl.sh/${{ secrets.get("random_image_name") }}

Save the pipeline and click on Run. After executing the pipeline, your output should resemble the following:

Next add the following steps yourself:

Add a run step to execute tests on the container you just built and pushed.
Add a trigger so that when a pull request is opened, Gitness can automatically trigger pipeline execution.
Add a slack plugin step so that failed tests trigger slack webhook and notification.
Add another run step so that when all tests pass, the image tag is retagged and pushed to a private image registry. You’ll need authentication at this step.

Security Considerations

While ttl.sh's no-credential, ephemeral approach offers flexibility and simplicity for CI environments, it introduces unique security considerations. The convenience of pushing images without credentials is counterbalanced by potential security risks — namely, the possibility of unauthorized image pulls. You can mitigate these risks through short-lived images and using UUIDs for image names:

Short-Lived Images: By design, images in ttl.sh are ephemeral. Setting a short expiration time for an image means it's available for a limited time window, reducing the risk exposure period.

Use of UUIDs: Incorporating UUIDs into image tags significantly lowers the risk of unauthorized access. The randomness and complexity of UUIDs make it exceedingly difficult for someone to guess the image name and pull it without authorization.

However, for production environments, organizations might consider deploying a private version of ttl.sh. This allows for more control over the security aspects, such as network isolation, access control, and auditing capabilities.

Takeaway

Integrating ttl.sh and Gitness creates an ephemeral CI environment that streamlines the build and test process. It helps to avoid the accumulation of outdated images and keeps the pipeline lean. This method is not just about speed; it's about maintaining a manageable and efficient development workflow. Adopting these tools can lead to more frequent and dependable software delivery for your engineering teams. Check out and follow Harness YouTube channel for

Unlocking Efficiency: Exploring Churn Rate With Harness Software Engineering Insights (SEI)

Francois Akiki — Fri, 15 Dec 2023 16:32:44 +0000

Improving the effectiveness and productivity of developers is crucial for the success of an engineering team. To achieve this, the team should adopt a metric-driven approach to identify the root causes that scales down the team's productivity. Out of the many metrics that engineering leaders can use to understand an engineering team's productivity is Churn Rate. In this article, you'll learn how high churn rates can impact the team and how Harness SEI can help lower churn rates in your organization.

What is Churn Rate?

In the context of software development, churn rate refers to the amount of work that is added or removed from a sprint backlog during a sprint. It measures the scope change and provides insights into the volatility of the sprint backlog.

A high churn rate indicates a lot of changes in the sprint backlog, which may result in delays in completing the sprint, and may also indicate that the requirements are not well-defined.
A low churn rate suggests that the sprint backlog is stable and the requirements are well-defined.

Why Churn Rate Matters?

Unveiling Sprint Volatility:

Churn Rate acts as a window into the dynamic nature of your sprint backlog. By understanding the scope changes during a sprint, teams can identify and address volatility, fostering a more stable and predictable development environment.

Optimizing Workflows:

SEI's Churn Rate empowers teams to optimize workflows by pinpointing areas of change mid-sprint. This insight allows for targeted strategies, ensuring that the development team can adapt swiftly and stay on course despite evolving project requirements.

Enhancing Collaboration:

With Churn Rate, collaboration between product managers and engineers reaches new heights. The metric provides a common ground for discussions, enabling both teams to make informed decisions and align efforts seamlessly.

The Formula Unveiled:

Churn Rate = (Points added mid-sprint + Points removed mid-sprint + Positive difference of changes in planned issues) / Points committed at the start of the sprint

Points added mid-sprint: The sum of story points for items added during the sprint.
Points removed mid-sprint: Identifies the reduction in story points due to the removal of items during the sprint.
Positive difference in planned issues: Reflects the positive changes in story points for planned issues during the sprint.

‍

Measure scope change in your sprint using SEI's Churn Rate metric

‍

Incorporating Churn Rate into SEI

SEI seamlessly integrates Churn Rate into its comprehensive suite of metrics, leveraging the power of the Trellis Framework. As part of the 40+ third-party integrations, Churn Rate ensures that your software factory's performance is not just measured but optimized, setting the stage for unparalleled efficiency.

Unlocking Potential with SEI's Churn Rate

In conclusion, Churn Rate emerges as a catalyst for transformative change in software development. SEI's commitment to providing actionable insights and workflow automation reaches new heights with this innovative metric. By understanding and harnessing Churn Rate, software delivery teams can not only increase productivity but also build a foundation for sustained excellence.

In a world where adaptability is key, SEI's Churn Rate becomes the compass that guides your software engineering ship through the ever-changing seas of development, ensuring you not only navigate challenges but thrive in the face of change. Experience the power of Churn Rate with Software Engineering Insights – where productivity meets precision.

Engineering teams can leverage Harness Software Engineering Insights to first baseline the people, processes and tooling bottlenecks and then drive a continuous improvement process. To learn more, schedule a demo with our experts.

‍Check out the original blog here!

Introducing the Harness SRM Backstage Plugin

Francois Akiki — Fri, 01 Dec 2023 04:53:28 +0000

We are excited to introduce the latest addition to our suite of Open Source Backstage Plugins - the Harness SRM Backstage Plugin. This plugin is designed to seamlessly integrate with your Backstage Instance as well as with Harness IDP to help with development team's workflow, ensuring that Service Level Objectives (SLOs) and error budgets are not just a metric but a part of your daily development practice.

Why Focus on SLOs?

Good Site Reliability Engineering (SRE) practices hinge on the continuous monitoring and adherence to SLOs. These objectives are pivotal in maintaining the reliability and performance of services. However, in the fast-paced world of software development, it's often challenging for developers to continually engage with separate observability tools. This is where our plugin bridges the gap.

Streamlining Observability

The Harness SRM Backstage Plugin is more than just a tool; it's a solution to a common oversight in the development process. By aggregating SLO data from Harness Service Reliability Management Module, this plugin brings critical insights directly to the developers' dashboard. This integration means that your team no longer needs to switch contexts or platforms to monitor their SLOs.

High Availability of Adequate Information

With this plugin, information about SLOs is not tucked away in a separate tool but is readily available in the Developer Portal. This accessibility ensures that your team is always aware of the current status of your services, leading to quicker responses and resolution if SLOs are at risk of being breached.

Empowering Developers

By making SLO data readily available within the familiar environment of the Developer Portal, the Harness SRM Backstage Plugin empowers developers to take proactive steps in maintaining and improving service reliability. This approach not only enhances individual productivity but also fosters a culture of accountability and ownership within the team.

Conclusion

In summary, the Harness SRM Backstage Plugin will help with how development teams interact with and respond to SLOs. By integrating critical data into the daily workflow, it ensures that adhering to good SRE practices is not an additional task but a seamless part of the development process.

‍Signup today for SRM Module and start using the Plugin to experience the difference this plugin makes in your team's efficiency and the reliability of your services.

‍Check out the original blog here!

Secure Container Image Signing with Cosign and OPA

Dewan Ahmed — Tue, 28 Nov 2023 22:15:55 +0000

As the adoption of containers in modern development continues to grow, ensuring the integrity of container images has become a pivotal aspect of application deployment strategies. In this video, Harness Developer Advocate Dewan Ahmed demonstrates how to leverage the combined power of Cosign and OPA for the secure deployment of container images to your Kubernetes cluster.

You can also read the text version of this tutorial.

Harness Chaos Engineering Faults Landscape

Francois Akiki — Tue, 28 Nov 2023 03:38:11 +0000

Harness Chaos Engineering provides a library of chaos faults using which chaos experiments are constructed and run. It is simple and intuitive to construct the chaos experiments using the given set of chaos faults. Before we delve into the details of the faults, let's look at the anatomy of a chaos experiment and the role of the chaos faults in the chaos experiments.

Quick Review of a Chaos Fault and a Chaos Experiment

A chaos fault is an actual fault or some distress injected around a system resource like CPU, Memory, Network, Time, IO system, Nodes, etc. A chaos experiment is an attempt to measure the resilience of a system when one or more chaos faults are run on it. In Harness Chaos Engineering, an experiment not only runs chaos faults, but it measures the resilience of the system in the context of the faults that were run.

When a chaos experiment is completed running, it provides a “Resilience Score,” which indicates the resilience of the target system against the faults that are injected. The Resilience Score is the % of successful steady state measurements measured during the chaos experiment execution.

Resilience Score of a Chaos Experiment

A developer who is designing and implementing the chaos experiment controls the meaning of a Resilience Score. The higher the number of steady state checks or probes passed during the experiment execution, the more it contributes to the resilience score. The steady state measurements in Harness CE are done through the Resilience Probes. Many resilience probes can be attached to a fault. The more probes you add to the faults inside the experiment, the more realistic the resilience score of the experiment will become.

The Resilience score of a chaos experiment = The percentage of successful resilience probes in the experiment.

Construction of a Chaos Experiment

‍

Chaos Fault Landscape in Harness Chaos Engineering

Faults for Kubernetes Resources

Runs via the Kubernetes Chaos Infrastructure or Agent

Supported Faults:

All Pod related faults, Node faults, http faults, IO/database chaos, network faults and load chaos. These faults are certified for the cloud Kubernetes services like EKS, AKE and GKE as well as for the on-prem versions like RedHat OpenShift, SuSE Rancher and VMware Tanzu.

Faults for VMWare Resources

Runs via the Kubernetes Chaos Infrastructure or Agent

Supported Faults:

Chaos faults are either injected through the VCenter APIs or through the VMware Tools directly on the operating system running inside the VM. Some faults such as VM power off, VM disk detach and VM host reboot are performed at the VCenter Level. Most of the common faults related to CPU/Memory/IO/Disk stress, http, DNS and Network faults are performed through the operating system through VMware tools. All the common faults are supported for the VMs running on Linux or Windows.

Faults for Linux Resources

Runs via the dedicated LinuxChaos Infrastructure or Agent

Supported Faults:

All faults related to resource stress, network and process. DNS error and spoof, Time Chaos and Disk are also supported. With ssh fault, network switches can also be targeted.

‍

Faults for Windows Resources

Runs via the Kubernetes Chaos Infrastructure or Agent

Supported Faults:

All faults related to resource stress, network and process. Time Chaos and Disk fill are also supported. These are supported for Windows instances that are running on Azure, VMware and AWS.

‍

Faults for AWS Resources

Runs via the Kubernetes Chaos Infrastructure or Agent

Supported Faults:

Deep coverage in the chaos faults for EBS, EC2, ECS and Lambda. AZ down faults for NLB, ALB and CLB. Some faults for RDS. All Kubernetes faults are supported for EKS on AWS.

Faults for GCP Resources

Runs via the Kubernetes Chaos Infrastructure or Agent

Supported Faults:

Faults for GCP VM disk and instance. All Kubernetes faults for GKE.

Faults for Azure Resources

Runs via the Kubernetes Chaos Infrastructure or Agent

Supported Faults:

Faults for Azure VM disk, instance and web app. All Kubernetes faults for AKS.

Faults for Cloud FoundryResources

Runs via the dedicated LinuxChaos Infrastructure or Agent

Supported Faults:

Support is extended for Pivotal Cloud Foundry as well as any other Cloud Foundry versions. Faults for Cloud Found App like Delete App, remove routes to app, stop app, unbind service from app etc are supported.

Faults for Spring Boot Resources

Runs via the Kubernetes Chaos Infrastructure or Agent

Supported Faults:

Chaos faults for Spring Boot Apps. App Kill, CPU Stress, Memory Stress, Latency, Exceptions and any chaos monkey fault as wrapper.

Conclusion

Harness Chaos Engineering supports a wide variety of chaos faults that spans across Operating systems, Cloud platforms and Kubernetes. These faults enable end users to verify the resilience of the code being deployed to the target systems or of the systems serving business critical applications. Check out all the Harness Chaos Faults on the Harness Developer Hub.

[

](https://app.harness.io/auth/#/signup?module=chaos&utm_source=harness_io&utm_medium=cta&utm_campaign=ce&utm_content=hero)

‍

‍Sign up FREE to experience the ease of resilience verification using chaos experiments. Harness provides a free plan that enables you to run a few chaos experiments free of charge for unlimited time.

‍

‍Check out the original blog here!

Start or Expand Your DevSecOps Education Journey - Introducing STO Developer Certification

Francois Akiki — Tue, 28 Nov 2023 01:44:17 +0000

As systems grow more distributed, complex, and critical to our daily lives the fog of development starts to set in; no one person has complete end to end visibility into the entire workings of a system. To combat the fog of development, dissemination of expertise across your pipelines is crucial. Security in an application and infrastructure context is a good model of this with the DevSecOps movement.

Software development teams are becoming more tasked with shifting left requirements to produce hygienic software. In the real world, software ages like milk and not like wine, so what was hygienic today might not be hygienic tomorrow. Your CI/CD pipelines are conduits of change and are excellent spots to disseminate expertise and ensure compliance and standards to security posture.

Harness’s Security Testing Orchestration or STO module is purpose built for your pipelines by orchestrating and prioritizing results from a multitude of scanning tools. Most organizations will have more than one scanning tool because tools can be granular or vertical in focus around a few pillars such as intent, language, and distribution. Because of the complexity of modern systems, everyone involved with the development of these systems should take a stake in helping secure these systems.

Security, Everyone’s Responsibility

At Harness, we view security as an important skill to have, we are offering our STO Developer Certification for free so everyone can up-level their DevSecOps skills. Taking a look at our study guide, will provide a great foundation around application vulnerability management.

STO Dev Cert Study Guide

Because so many components are in modern software today, keeping up with the bill of materials / how components age can be tricky. Harness STO can help you identify and prioritize issues that do need to be addressed. Having Harness STO as part of your pipeline is a prudent capability and being certified in Harness STO is a great skill to have.

Study and Sign Up Today

Getting certified at the developer level on Harness Security Testing Orchestration is a great milestone in your DevSecOps journey. Register for the exam from the Harness Developer Hub once you feel comfortable taking the exam.

-Harness Product Education Engineering Team

‍

‍Check out the original blog here!

How to use Self-Service Onboarding in Harness Internal Developer Portal

Debabrata Panigrahi — Fri, 17 Nov 2023 16:57:36 +0000

In this tutorial, Debabrata a Developer Relations Engineer at Harness, dives into the essentials of creating a basic service onboarding pipeline within the Harness Internal Developer Portal (IDP), that runs on Backstage v1.17. This feature is a game-changer for platform engineers and developers looking to streamline their application development processes.

What You'll Learn:

🛠️ Setting Up the Pipeline: We start by guiding you through the process of creating a Harness pipeline for service onboarding, including the creation of Build or Custom stages.
📝 Using Software Templates: Learn how to interact with software templates to collect user requirements efficiently.
🔄 Automating Service Onboarding: Discover how the Harness pipeline automates the onboarding of new services, from fetching skeleton code to creating new repositories.
🐍 Scripting with Cookiecutter: We'll show you how to use a Python CLI, cookiecutter, to generate a basic Next.js app and set up a repository.
🌐 Managing Variables and Authentication: Understand how to manage pipeline variables and authenticate requests within the pipeline.
🎨 Creating a Software Template Definition: We guide you through creating a template.yaml file in IDP, powered by Backstage Software Template.

From Zero to Kubernetes Deployment: Harness Continuous Delivery in Action

Dewan Ahmed — Fri, 17 Nov 2023 14:45:48 +0000

Harness Continuous Delivery pipelines enable you to orchestrate and automate your deployment workflows, allowing you to push updated application images to your target Kubernetes cluster seamlessly.

In this video, Dewan Ahmed, a Developer Advocate at Harness, demonstrates how to install a Harness delegate and create entities such as Harness secrets, connectors, environments, services, and pipelines. Dewan guides you through a successful pipeline execution with a manual trigger and then shows how to create a pipeline variable to configure the Kubernetes namespace to be provided during pipeline execution.

Forem: Harness

Speed Up Your CI Pipelines with Docker Layer Caching

DLC and Multi-stage Builds

Harness CI Intelligence: Docker Layer Caching

Benchmarking Build and Push to Docker

Conclusion

End-to-end MLOps CI/CD pipeline with Harness and AWS

The story

Design and architecture

Prerequisites

Prepare AWS

Prepare GitHub

Create Harness secrets

Create AWS and GitHub connectors

Create the Harness pipeline

Build, push, and scan the image

Test and upload artifacts

Add ML model policy checks

Deploy AWS Lambda function

Monitor the model

Trigger pipeline based on Git events

Add an approval gate before prod deployment

Use the model in a web application

Conclusion

React App and Build Pipeline with Task and Gitness

Prerequisites

Create React Application

Create Taskfile.yml

Build and Test

Setup Gitness

Create Pipeline

Conclusion

Build and Push to GAR and Deploy to GKE - End-to-End CI/CD Pipeline

Architectural and Pipeline Diagrams

Prerequisites

Required Setup and Configurations

Demo application

GitHub and Docker authentication

Image registry setup on Google Cloud Platform (GCP):

Kubernetes cluster setup with GKE:

GCP IAM and Service Account setup:

Slack workspace and app setup:

Harness entity setup

Build and push image to GAR

Deploy to GKE

Add Approval and Slack Notifications

Security Tests and Policy Enforcement (Bonus Section)

Run OWASP Tests

Add a policy to mandate approval step on deployment stages

View the running application

Homework Task

Ephemeral CI environments using ttl.sh and Gitness

The Problem

The Solution

Demo

Security Considerations

Takeaway

Unlocking Efficiency: Exploring Churn Rate With Harness Software Engineering Insights (SEI)

What is Churn Rate?

Why Churn Rate Matters?

Unveiling Sprint Volatility:

Optimizing Workflows:

Enhancing Collaboration:

The Formula Unveiled:

Incorporating Churn Rate into SEI

Unlocking Potential with SEI's Churn Rate

Introducing the Harness SRM Backstage Plugin

Why Focus on SLOs?

Streamlining Observability

High Availability of Adequate Information

Empowering Developers

Conclusion

Secure Container Image Signing with Cosign and OPA

Harness Chaos Engineering Faults Landscape

Quick Review of a Chaos Fault and a Chaos Experiment

Resilience Score of a Chaos Experiment

Construction of a Chaos Experiment

Chaos Fault Landscape in Harness Chaos Engineering

Conclusion

Start or Expand Your DevSecOps Education Journey - Introducing STO Developer Certification

Create `Taskfile.yml`