Forem: Jim Sheldon

React App and Build Pipeline with Task and Gitness

Jim Sheldon — Fri, 05 Apr 2024 14:01:13 +0000

Task is a task runner/automation tool written in Go and distributed as a single binary.

A Taskfile is written in YAML, which may provide a shorter learning curve than alternative tools, like GNU Make. Task can also leverage Go's robust template engine.

Task can be useful in build and test pipelines. Logic for running builds and tests can be written once in Taskfile.yml (or other supported filenames), then used in both local development workflows and under automation.

Gitness is an open source development platform from Harness that hosts your source code repositories and runs your software development lifecycle pipelines.

In this guide, we'll craft a Taskfile.yml for a sample React app, suitable for both local development and integration with your pipelines powered by Gitness.

Prerequisites

Install Task and verify you can run task in your terminal.

This guide was tested with Task version 3.35.1.



$ task --version
Task version: v3.35.1 (h1:zjQ3tLv+LIStDDTzOQx8F97NE/8FSTanjZuwgy/hwro=)

Install Docker and verify you can run docker in your terminal.

This guide was tested with Docker version 24.0.7.



$ docker --version
Docker version 24.0.7, build afdd53b

Install Node and verify you can run node in your terminal.

This guide was tested with Node version 20.12.0.



$ node --version
v20.12.0

ℹ️ Note
Installing Node also installs `npm` and `npx` binaries.

Create React Application

Use the Create React App project to automatically generate the application.



npx create-react-app my-react-app

Your my-react-app directory structure will look like this.



my-react-app/
├── README.md
├── node_modules/
├── package-lock.json
├── package.json
├── public/
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src/
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── reportWebVitals.js
    └── setupTests.js

Change to the application directory.



cd my-react-app

Start the application.



npm start

Open http://localhost:3000 in your browser.

You now have a working React application! 🎉

In your terminal, type Ctrl-C to stop the process.

Create `Taskfile.yml`

Create a Taskfile.yml file in your my-react-app directory with this configuration.



version: '3'

tasks:

  npm-install:
    cmds:
      - npm install

  build:
    deps: [npm-install]
    cmds:
      - npm run build

  test:
    deps: [npm-install]
    cmds:
      - npm test

The build task runs npm run build, which creates a production build of your application.

The test task runs npm test, which runs unit tests. There is only one unit test in src/App.test.js.

Note that npm-install task is a dependency of the build and test tasks.

Build and Test

Run task build in your terminal.

By default, npm test will run tests interactively. To simulate running tests under Continuous Integration, set the CI environment variable.

Run CI=true task test in your terminal.

Note that task passed the CI=true variable to the npm test command.

Setup Gitness

Install Gitness
Create a project named demo
Either create a repository named my-react-app and push your sample app code, or import the repository jimsheldon/my-react-app, which I created for this guide

Create Pipeline

Open http://localhost:3000/demo/my-react-app/pipelines in your browser and select New Pipeline.

Enter build-and-test in the Name field (this will automatically populate the YAML Path field), then select Create.

Enter this Gitness pipeline in the YAML editor.



kind: pipeline
spec:
  stages:
    - name: task example
      type: ci
      spec:
        steps:
          - name: install task
            type: run
            spec:
              container: alpine
              script: |
                apk add curl
                sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d
                ./bin/task --version

          - name: build
            type: run
            spec:
              container: node:20
              script: |
                ./bin/task build

          - name: test
            type: run
            spec:
              container: node:20
              script: |
                ./bin/task test

This Gitness pipeline has a single stage with three steps. The install task step uses the alpine Docker image to install the task binary into the workspace, where it can be reused by the following build and test steps, which run in the node Docker image.

Select Save and Run.

Observe the pipeline execution.

Conclusion

This workflow improves the separation of concerns between local development and pipeline execution.

As long as the application can be built and tested with task build and task test commands, developers always keep the same workflow, and the pipeline does not need to be modified.

For example, the developers could switch to yarn by changing Taskfile.yml to use yarn commands rather than npm commands. This change would be transparent to developers, who would continue to run task build and task test, and the Gitness pipeline yaml would not need to be changed.

Next steps:

See more examples of using Task with Gitness
Create triggers to automatically run the pipeline when commits are pushed to the repository
Learn how Gitness manages your data
Learn how to manage Gitness projects and roles

Build Arm Docker images five times faster on native hardware

Jim Sheldon — Tue, 31 Oct 2023 19:24:44 +0000

The ARMv8 (a.k.a. ARM64) CPU architecture, announced 12 years ago, was the first Arm architecture to support a 64-bit instruction set. Apple famously introduced its ARM64-based M1 chip in November of 2020. In June of this year, Apple completed its transition away from Intel-based CPUs by launching a new Mac Pro model powered by the M2 revision of the chip. In the PC market, Arm notebooks are expected to increase to 27% market share by 2027. In the cloud computing space, all major service providers offer Arm solutions.

We have entered a world where Arm can’t be ignored. If your company isn’t adapting your applications and running infrastructure on Arm, chances are your competition is.

Cost and Energy Savings with Arm

The Arm architecture offers significant potential cost and energy savings. A recent study showed that moving to Arm-based cloud infrastructure can achieve up to 40% better performance per dollar over comparable Intel-based infrastructure.

Companies like Squeaky reduced their cloud spending by 35% by switching to Arm. When analyzing CPU, memory, and latency metrics after the migration, Squeaky reports "no performance degradation" on the new hardware.

Database company Aerospike saw annual costs decrease by an estimated 27% after switching to Graviton2 processors on AWS EC2. They also cut carbon emissions by an estimated 49% while achieving aggressive targets for transaction throughput and data access latency.

Docker and Arm

The 2022 StackOverflow Developer Survey found that Docker is becoming a "fundamental tool for Professional Developers, increasing from 55% to 69%." According to the CNCF 2022 annual survey, 44% of respondents use containers for nearly all applications and business segments, and another 35% say they use containers for at least a few production applications.

This means that the first challenge many companies face when adopting Arm is how to build their application containers for this different architecture. While Docker supports building images for other CPU architectures under emulation, this can significantly impact performance.

Native Builds vs. Emulation

To visualize the difference in performance, we built base Docker images for Python, Ruby, HAProxy and Redis using both emulation and native ARM64 hardware.

This graph shows build time in minutes for native Arm hardware compared to emulation ¹.

These results show builds completed five times faster on average with native Arm hardware. This is valuable time your developers could be using for development, rather than waiting for pipelines.

While Arm clearly has incredible momentum, some companies in the continuous integration space have been slow to provide hosted resources.

Comment for #5631

mfittko commented on Apr 20, 2023

It's just way more convenient to work with arm based images and not having to deal with multi arch, if you have an ARM based mac, so big +1 here, too ;) it would just be so much more sustainable to be able to work and build for a single platform (of choice) - think of the environment! All these wasted cpu cycles!

View on GitHub

Here at Harness, Arm infrastructure has been available in Harness CI Cloud since the beginning.

To sign up for Harness Cloud and get 2,000 free monthly build credits for your pipelines, see the original blog post for this article.

Tested with minimum 4 vCPUs and 16GB memory ↩

Build and test a Golang app with Gitness

Jim Sheldon — Fri, 20 Oct 2023 19:33:22 +0000

Gitness is an open source development platform from Harness that hosts your source code repositories and runs your development life cycle pipelines.

This tutorial covers how to build and test a Golang application with Gitness.

Installation

You can install Gitness on any computer running Docker.

Run this command to start Gitness:

docker run -d \
  -p 3000:3000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp/gitness:/data \
  --name gitness \
  --restart always \
  harness/gitness

Visit http://localhost:3000 in your browser and you will be greeted with the Gitness login page.

Click the Sign Up link.

Enter your desired user ID, email address and password, then click Sign Up.

ℹ️ Note
By default, the first user to sign up will be granted administrator access. To learn more about how to configure Gitness, refer to the configuration documentation.

Create Project

Create a project which will contain the Git repository.

Click the New Project button.

Give the project a name and optional description, then click the Create Project button.

Import Repository

Expand the New Repository drop-down menu and select Import Repository.

jimsheldon/go-outyet, is a simple Golang application that reports whether a version of Golang has been released yet.

Enter https://github.com/jimsheldon/go-outyet in the Repository URL field, then click Import Repository.

Create Pipeline

Click on the imported repository, then open the Pipelines view from the menu on the left.

Click the New Pipeline button.

Give the pipeline a name, the YAML path will be generated automatically (it can be changed if desired), then click the Create button.

Gitness can automatically generate a pipeline based on the code in the repository.

Click the Generate button above the pipeline editor.

Examine the pipeline that was generated, you will see run steps that build and test the app, and a Docker build plugin step.

The build step compiles the binary, and the test step runs unit tests from the main_test.go file.

The Docker build step builds a Docker image from the Dockerfile file, but does not publish the image (the dry_run: true setting prevents publishing).

Click Save and Run button.

ℹ️ Note
Gitness will commit the pipeline file to your repository and create a default pipeline trigger for pull request events, which can be customized if desired.

Run Pipeline

Click the Run pipeline button in the window that appears.

You will see the pipeline execution view.

Congratulations, you just imported a repository and ran a pipeline! 🎉

In this tutorial, we have just scratched the surface of what is possible with Gitness 😄

Next Steps

Create triggers to automatically run the pipeline on certain events
Define conditions to only run specific steps for certain events
Set values dynamically at runtime with expressions
Review samples for other languages and write some pipelines of your own!

Resources

Cover image by Josh Olalde on Unsplash

Managing the ‘Git’ in ‘GitOps’: 4 Ways to Structure Code in Your GitOps Repos

Jim Sheldon — Wed, 15 Jun 2022 14:57:55 +0000

Implementing GitOps practices will take your software delivery pipelines to the next level. Declarative, immutable, and continuously reconciled infrastructure brings many benefits when managed through GitOps best practices. Over the years, I have helped many development teams build and improve their GitOps workflows. In this blog, I will share four approaches to managing code used in those pipelines.

The “Ops” half of “GitOps” refers to configuration code, or Infrastructure as Code (IaC). Software depends on the resources managed by this code to function. Managing this configuration in Git repositories offers many benefits. Often the structure of this code is an afterthought, which leads to significant refactoring in the future.

Application and Infrastructure Code in One Repository

The first example manages application code and infrastructure code in the same repository. A single long-lived branch exists (main).

Example

Below is a Node.js project with application code in the root, and YAML files in the kubernetes directory. Changes to development.yaml apply to the development environment, changes to production.yaml apply to the production environment.

Benefits

Infrastructure code and application code in the same repository keeps everything versioned together. There is no need to connect the dots between multiple repositories to reproduce the state of the application and configuration at a certain point in time.
One repository means less context switching for developers. Developers don’t need to change repositories when making changes to infrastructure code.

Drawbacks

No privilege separation. Developers with access to the repository will be able to change both application and infrastructure code.

Some organizations require separation between application and infrastructure code. The examples below all manage application and infrastructure code in their own repositories. This improves privilege separation as each Git repository can set its own user privileges.

Separate Infrastructure Repository, Multiple Branches

You may be familiar with Git branching workflows such as Gitflow. Gitflow has fallen out of favor recently as trunk-based development has gained popularity. There are good reasons to avoid more than one long-lived branch in your Git repository. Yet, multiple long-lived branches are still worth considering in certain cases.

Example

Below is a Helm chart repository with two long-lived branches, development, and production. Changes always originate in the development branch. Promotion to production requires merging development into production. The development environment uses the development-values.yaml values file in the development branch. The production environment uses the production-values.yaml values file in the production branch.

Benefits

Low risk of configuration drift when promoting changes between environments. Merging branches ensures that no changes will be missed.
Improved privilege separation between development and production changes. For example, GitHub supports branch protection rules. The owner of the repository can control which users can commit to a branch.

Drawbacks

This is a “one lane road” for your infrastructure code. Changes in the development branch can block production changes (without cherry-picking desired changes).

Separate Infrastructure Repository, Directory-Based

Now, let’s consider a repository where a single long-lived branch exists (main). Each environment has its own directory.

Example

Below is a Terraform repository with separate development and production directories. Changes to development use the development.tfvars tfvars file in the development directory. Changes to production use the production.tfvars tfvars file in the production directory.

Benefits

Changes made in the development directory do not affect the production directory.

Drawbacks

Increased risk of configuration drift between environments. There is a high burden on the developer to understand differences between directories.
No privilege separation. Users can make changes to both development and production environments.

Multiple Infrastructure Repositories, One per Environment

Let’s consider an approach where each environment has its own dedicated repository. Each repository has a single long-lived branch (main).

Example

Here is an example Terraform project, where development and production are separate repositories. Changes to development use the development.tfvars tfvars file in the development repository. Changes to production use the production.tfvars tfvars file in the production repository.

Benefits

Highest level of privilege separation. Any feature that your Git host provides around user/group access at the repository level will be available to you. Only users that need to make changes to production will be able to commit changes to the production repository.
Easier to bring up new environments, or migrate existing environments. When bringing up a new environment, create a new repository. There is no need to integrate with an existing repository to bring up a new environment.

Drawbacks

Higher risk for configuration drift between environments. There is a high burden on the developer to understand differences between repositories.

Conclusion

In my experience, one repository per environment is the most future-proof method for managing your GitOps code. The privilege and environment separation benefits outweigh the potential drawbacks. If you decide the separation is not required in the future, you can collapse multiple repositories into one. The good news is that whatever method you choose, Harness’ suite of products supports them all.

Whether you are building, testing, and publishing artifacts with Harness CI, deploying with Harness CD, or taking your pipelines to the next level with Harness GitOps (currently in beta), we’ve got you covered! Also, every Harness pipeline can take advantage of advanced features around governance, chaos engineering, and more.

Come see how Harness can help accelerate your GitOps journey. Sign up for a 14-day free trial and follow our Kubernetes CD Quickstart guide to deploy an application to your cluster. If you are looking for a guided tour, book a demo.

We would love to answer any questions you might have in our forum, community Slack, or at an upcoming Harness & Drone User Group virtual meetup.

The post Managing the ‘Git’ in ‘GitOps’: 4 Ways to Structure Code in Your GitOps Repos appeared first on Harness.

Run your own Drone CI

Jim Sheldon — Sat, 02 Apr 2022 15:22:05 +0000

Started in 2012, Drone CI is a container-native, event-based, multi-architecture, multi-OS platform for busy teams looking to tighten their software development lifecycles.

Since its acquisition by Harness in August of 2020, Drone CI has made significant improvements in usability and features. Let's see how, in just a few minutes, you can start experimenting with running Drone CI pipelines on any computer running Docker.

Prerequisites

Be comfortable running commands in the terminal
GitHub account
ngrok account
Computer that can run amd64 or arm64 Docker containers (this includes the new M1-based Macs)
- In your terminal, verify you have docker and docker-compose commands available

Overview

Drone CI supports multiple SCM solutions, such as GitHub, Bitbucket, Gitea, GitLab and more.

This guide will focus on GitHub. Because we have chosen GitHub, we need to allow GitHub to send webhooks to the Drone CI process on our computer. We will use ngrok to create a tunnel to the internet so our Drone CI process can receive webhook traffic from GitHub.

Create the `ngrok` tunnel

Login to https://ngrok.com
Open https://dashboard.ngrok.com/get-started/setup, follow the steps to install the ngrok binary and connect to your account
Create the tunnel by running ngrok http 8080 in your terminal, do not stop the process until you are finished with this guide
ngrok will create your tunnel and provide you with a URL such as https://34f3-146-168-58-81.ngrok.io

Your terminal should display session information similar to this screenshot:

Create the GitHub OAuth Application

Follow Create an OAuth Application
- For the "Homepage URL", enter the ngrok URL from the previous step
- For the "Authorization callback URL", enter the same ngrok URL, with /login appended
- You will be given the "Client ID" and a "Client secret", make a note of each (the secret is only displayed once, if you misplace it, you will need to generate a new client secret)

Your OAuth application should look similar to this screenshot (for extra authenticity, upload the official Drone CI logo):

Note: ngrok URLs are temporary with the free plan. If you terminate your ngrok session and create a new one, your URL will change and you will need to update the "Homepage URL" and "Authorization callback URL" in your OAuth Application.

Start the Drone and Drone Runner processes

The Drone server process requires at least one runner process where pipelines will execute. Drone CI supports multiple runners, but the Docker runner is the most popular.

Fork the repository https://github.com/jimsheldon/drone-quickstart and clone the forked repository to your computer
Edit the file run.sh
- Set DRONE_GITHUB_CLIENT_ID and DRONE_GITHUB_CLIENT_SECRET using the "Client ID" and "Client secret" from the previous section
- Set DRONE_GITHUB_ADMIN to your GitHub username
- Set DRONE_SERVER_HOST to your ngrok URL, without the https:// (if your ngrok URL is https://34f3-146-168-58-81.ngrok.io, set DRONE_SERVER_HOST to 34f3-146-168-58-81.ngrok.io)
- Save your changes
Open a new terminal session and change to the directory where you checked out the repository
- Execute run.sh by typing ./run.sh and hitting enter
- Open your ngrok URL in a browser, you should be greeted with "Welcome to Drone"! 🎉
- Click "Continue" and you will be asked to authorize your OAuth application
- Next you will see the "Complete your Drone Registration" screen
- Click "Submit" and you will see the Drone CI interface, filter by "quickstart" to see your forked repository
- Click on the repository, then click "Activate Repository"
- Your repository should now be activated

Run your first pipeline

Edit the .drone.yml file in the root off your repository. Change the echo hello world message to something of your choice. If your CPU is amd64 (Intel, AMD) you should remove the platform section, which is currently specific to arm64 (Apple M1, Raspberry Pi)
Push your change to a branch and open a pull request
Your pull request will trigger a check in Drone CI called "continuous-integration/drone/pr"
Click the "Details" link, you will see your pipeline execution in Drone CI! Click the "hello" step to see your message printed

You now have your very own Drone CI Enterprise installation! Drone CI Enterprise includes a free trial of up to 5,000 builds. To learn more about licensing options, see the Drone CI Enterprise FAQ.

The official Drone CI documentation provides example pipelines in virtually every language. Try it out on one of your own projects!

Teardown

In the directory where you ran run.sh run docker-compose down.
In the terminal where your ngrok session is running, type CTRL-C.

Note: ngrok URLs are temporary with the free plan. If you terminate your ngrok session, the next time you run ngrok http 8080 your URL will change, and you will need to update the "Homepage URL" and "Authorization callback URL" in your GitHub OAuth Application, as well as the DRONE_SERVER_HOST value in run.sh.

Conclusion

Hopefully this gives you a quick way to get started running Drone CI pipelines from your GitHub repositories.

If you would like to learn more and participate in the Drone CI and Harness communities, here are some helpful links:

Official Drone CI documentation: https://docs.drone.io
Harness Slack community: https://harnesscommunity.slack.com (#drone channel)
Harness community forum: https://community.harness.io
Harness & Drone User Group: https://www.meetup.com/harness/
Drone CI official Twitter account: https://twitter.com/droneio

Forem: Jim Sheldon

React App and Build Pipeline with Task and Gitness

Prerequisites

Create React Application

Create Taskfile.yml

Build and Test

Setup Gitness

Create Pipeline

Conclusion

Build Arm Docker images five times faster on native hardware

Cost and Energy Savings with Arm

Docker and Arm

Native Builds vs. Emulation

Comment for #5631

Build and test a Golang app with Gitness

Installation

Create Project

Import Repository

Create Pipeline

Run Pipeline

Next Steps

Resources

Managing the ‘Git’ in ‘GitOps’: 4 Ways to Structure Code in Your GitOps Repos

Application and Infrastructure Code in One Repository

Example

Benefits

Drawbacks

Separate Infrastructure Repository, Multiple Branches

Example

Benefits

Drawbacks

Separate Infrastructure Repository, Directory-Based

Example

Benefits

Drawbacks

Multiple Infrastructure Repositories, One per Environment

Example

Benefits

Drawbacks

Conclusion

Run your own Drone CI

Prerequisites

Overview

Create the ngrok tunnel

Create the GitHub OAuth Application

Start the Drone and Drone Runner processes

Run your first pipeline

Teardown

Conclusion

Create `Taskfile.yml`

Create the `ngrok` tunnel