Forem: Oscar Medina

Enable Self-service Deployments for Devs in EKS

Oscar Medina — Mon, 09 May 2022 20:51:16 +0000

If you are interested in establishing a smooth collaboration with your Platform Team and n Dev Teams, and allowing Dev Teams to deploy freely and at will without Platform Team being the bottleneck, please checkout my article on the HashiCorp Learn Series.

Oscar Medina, senior cloud architect at Amazon Web Services, joined us on a live stream recently to demo a collaborative workflow to enable self-service Amazon EKS deployments for developers, using CDK for Terraform. Check out the recording of the live demo, and read on for a tutorial of the workflow that platform and developer teams can use to collaborate on Kubernetes deployments with HashiCorp Terraform, Sentinel policies, CDK for Terraform, and EKS.

<iframe width="560" height="315" src="https://www.youtube.com/embed/ijgKc6vGVyM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Securing Apps Deployed via Jenkins X Using Ambassador Edge Stack

Oscar Medina — Wed, 24 Jun 2020 19:55:08 +0000

Overview

Microservice orchestrated by Kubernetes sometimes need not be public-facing.

There are many options to secure an endpoint, however, in this post we walk through configuring Datawire's Ambassador Edge Stack to protect an app which has been put through CI/CD and published via Jenkins X. We enable SSO to leverage existing identity management using Azure Active Directory all in AWS and EKS.

I also walk you through installing Jenkins X on AWS EKS first, then installing and configuring Ambassador. Of course, if you have a Jenkins X cluster already, you can skip to the Setup Ambassador Edge Stack section.

GitHub Repo: You can clone the repo specific branch.

Create EKS Cluster

For this scenario, we will use AWS EKS

Create an EKS cluster using eksctl using the following command

eksctl create cluster  \
--name drymartini  \
--full-ecr-access

Once the cluster is Active you are ready to install Jenkins X.

Configure Route 53 DNS

Part of our scenario is to configure Jenkins X with a custom domain. To do this, we must first configure a Route 53 Managed Zone then delegate your subdomain to Route 53. In my case, my domain is managed by Google Domains, and so I’ve created an NS record pointing to the NS servers given to me by Route 53 after I created my Hosted Zone.

Create a Managed Zone for subdomain
Delegate your subdomain to AWS Route 53 from the vendor hosting your domain.
Test it using dig jx.eks.sharepointoscar.com (in my case).

NOTE: Testing the Hosted Zone can be done from the AWS Console, and ultimately you want a response with NOERROR

Install Jenkins X using Boot

With the cluster created and DNS configured, we are ready to install Jenkins X using Jenkins X Boot

We will be using a stable distribution of the open-source Jenkins X project, otherwise known as CJXD CloudBees Jenkins X Distribution. The advantage of using this edition vs. the straight OSS is that it has been tested, and certified for EKS, it also allows for a controlled platform upgrade within your enterprise environment.

Install Jenkins X CLI

# download binary for CJXD (CloudBees Jenkins X Distro) based on your platform.
# https://docs.cloudbees.com/docs/cloudbees-jenkins-x-distribution 
curl -L https://storage.googleapis.com/artifacts.jenkinsxio.appspot.com/binaries/cjxd/latest/jx-darwin-amd64.tar.gz | tar xzv

# move to bin
sudo mv jx /usr/local/bin

# test
➜ jx version
NAME               VERSION
jx                 2.0.1245+cjxd.8
Kubernetes cluster v1.14.10-gke.27
kubectl            v1.15.0
helm client        2.14.3
git                2.21.0
Operating System   Mac OS X 10.14.6 build 18G3020

Execute JX Boot

Our next step is to install Jenkins X on top of the EKS cluster. We do this by using a declarative yaml approach.

During the curl of the jx binary, a couple of yaml files were downloaded to the same directory where we executed that command.

We will use the jx-requirements-eks.yml

Step 1 - Modify the jx-requirements-eks.yml file with basic settings

The basic settings we will modify in order to execute an initial jx boot installation will be as follows (please modify your file accordingly)

cluster:
 clusterName: drymartini
 environmentGitOwner: jenkins-oscar #github organization
 environmentGitPublic: true #my org is on free tier, so public repos
 provider: eks
 region: us-west-2

Next we modify the main ingress field as follows:

ingress:
 domain: jx.eks.sharepointoscar.com #Route 53 Managed Zone
 externalDNS: true
 ignoreLoadBalancer: true
 namespaceSubDomain: . # personally a dot makes for cleaner FQDN entries
 tls:
   email: me@sharepointoscar.com
   enabled: true
   production: true

HashiCorp Vault Initial Configuration

In order for the jx boot command to run successfully the first time, we need to specify an IAM username in the specific jx-requirements-eks.yml file section as follows:

vault:
 aws:
   autoCreate: true
   iamUserName: aws_admin@sharepointoscar.com #IAM Username
 disableURLDiscovery: true

For details on IAM permissions, please refer to the documentation.

Step 2 - Execute JX Boot CLI Command

Having modified the appropriate fields in the jx-requirements-eks.yml, we are ready to execute the jx boot CLI command as follows:

$ jx boot -r jx-requirements.eks.yml

The CLI will ask a few questions.

The cluster configuration repository is cloned immediately. This repository will be added to your GitHub Organization, and any future changes will require a Pull Request, the GitOps way!

IMPORTANT: Any future executions of jx boot, will require you to execute it from the root of the cluster configuration repository that was cloned during the first run. This effectively means the original jx-requirements-eks.yml file is no longer used in subsequent executions.

Setup Ambassador Edge Stack

To setup Ambassador, you can follow the simple instructions on their wonderful Docs site.

Install from MacOS

Download it with a curl command:

sudo curl -fL https://metriton.datawire.io/downloads/darwin/edgectl -o /usr/local/bin/edgectl && sudo chmod a+x /usr/local/bin/edgectl

The installer will provision a load balancer, configure TLS, and provide you with an edgestack.me subdomain. The edgestack.me subdomain allows the Ambassador Edge Stack to automatically provision TLS and HTTPS for a domain name, so you can get started right away.

Once you have it configured, you are ready to move to next step.

Create AD App Registration

Setup Azure App Registration as per the instructions on Docs site.

Configure Filter

Create a file oauth_filter.yaml and paste this content. Then deploy it via kubectl apply -f oauth_filter.yaml

apiVersion: getambassador.io/v2
kind: Filter
metadata:
  name: azure-skiapp-ad
  namespace: jx-staging
spec:
  OAuth2:
    authorizationURL: https://login.microsoftonline.com/<AD_TENANT_ID>/v2.0  ## URL where Ambassador Edge Stack can find OAuth2 descriptor
    clientID: <APP_CLIENT_ID> ## OAuth2 client from your IdP
    secret: <APP_SECRET> ## Secret used to access OAuth2 client
    protectedOrigins:
    - origin: https://skiapp.dev.sharepointoscar.com

Configure Filter Policy

Create a file oauth_filter_policy.yaml and paste this content. Then deploy it via kubectl apply -f oauth_filter_policy.yaml

apiVersion: getambassador.io/v2
kind: FilterPolicy
metadata:
  name: azure-skiapp-policy
  namespace: jx-staging
spec:
  rules:
      # Requires authentication on requests from hostname
    - host: "skiapp.dev.sharepointoscar.com"
      path: "*"
      filters:
        - name: azure-skiapp-ad

Additional Config via Ambassador Portal

To access the portal, execute the CLI as follows:

edgectl login --namespace=ambassador <yourinstancename>.edgestack.me

Create new Host entry. In my case, it is for skiapp.dev.sharepointoscar.com. I let Ambassador handle TLS as you can see.

Create a new Mapping, it should look like the following.

Import App Into Jenkins X

Import app from GitHub as follows. NOTE: Please make sure to fork the app before importing it. You are effectively importing it from your own GitHub Organization.

jx import --url <GITHUB_REPO_URL>

Modify App Ingress Config

When you import an application into Jenkins X, it automatically creates a Helm Chart! In this step, we need to modify the Helm Chart to tell Jenkins X, it should not use exposecontroller component to expose the app, which it did automatically.

Modify the Helm Chart

Under the helm/myapp chart, you will find a file called values.yaml which we need to modify as follows.

Original Service configuration.

service:
  name: skiapp
  type: ClusterIP
  externalPort: 80
  internalPort: 8080
  annotations:
    fabric8.io/expose: "true"
    fabric8.io/ingress.annotations: "kubernetes.io/ingress.class: nginx"

Modified Service

service:
  name: skiapp
  type: ClusterIP
  externalPort: 80
  internalPort: 8080
  annotations:
    getambassador.io/config: |
        ---
        apiVersion: ambassador/v1
        kind: Mapping
        name: skiapp-service_mapping
        host: skiapp.dev.sharepointoscar.com
        prefix: /
        service: skiapp:80

With this configuration in place, I can go to https://skiapp.dev.sharepointoscar.com and will get redirected to Microsoft login screen.

Watch the Webinar

https://www.youtube.com/watch?v=PUEno5NTSKA&feature=youtu.be

Summary

We now have an app which was put through CI/CD using Jenkins X - protected using Ambassador, and using SSO backed by Azure Active Directory. You can use Okta or any other IdP you environment uses, as long as it is using OAuth2 in this case.

Cheers,

@SharePointOscar

Interpolating HashiCorp Vault Secrets into Jenkins X Helm Chart App

Oscar Medina — Fri, 24 Jan 2020 17:48:37 +0000

Overview

If you have used Jenkins X, you know that HashiCorp Vault is installed by default in the CJXD (CloudBees Jenkins X Distro). However, there hasn't really been an example in the docs about how to use secrets stored in Vault when deploying an app that has the need for them.

In this guide, I walk you through deploying the Cars REST API a full-stack app with a backend in MongoDB. The repo branch titled Vault-Secrets is the one you want to look at.

TIP: With the recently announced tighter Vault integration with Kubernetes, there is actually a better way. We can inject Vault secrets directly into pods and make them available. My friend Nick Jackson explains how to do this on his post - Dynamic Database Credentials with Vault and Kubernetes

Nick and I will be showing you how to do this on Jenkins X very soon, so stay tuned!

Prerequisites

Either the OSS or CJXD editions of Jenkins X working with Vault installed.
Clone the Cars REST API app

1. Creating Vault Secret

The first thing you want to do is create the Vault Secret. Call it mongodb and add the following keys:

mongodb-username
mongodb-password
mongodb-root-password

Accessing HashiCorp Vault UI

To access the Vault instance in Jenkins X, you will need to first retrieve the configuration by executing the following:

> $ echo eval `jx get vault-config`

# copy the output to set env variables (yours are different of course)
export VAULT_ADDR=https://vault.cjxd.sharepointoscar.com export VAULT_TOKEN=r.4bbbbbbbX4jDB4QOKoHDCid4Sxtk5v

What we are doing is retrieving the configuration. Then setting up Environment Variables (copy n paste them on your terminal). If you then execute env | grep VAULT you will see the variables are set.

From the UI, follow the instructions to create a secret. It should look something similar to the following:

2. Modify the Environment Repo (Staging)

NOTE
Though I am told that you can accomplish secret interpolation by setting this at the app level only, I have not tried it and so I want you to also include this on your target environment where the app is being deployed. In my case, it is Staging. Take a look at the actual file in my repo.

You will want to check out the file located at env/values.yaml (this file can be empty based on my experience, later gets populated again)
You will want to create a new file, name it env/values.tmpl.yaml. The file should look like the sample code below. At the bottom is where I add my app secrets and set them as environment variables for the Staging environment.


PipelineSecrets: {}
cleanup:
  Annotations:
    helm.sh/hook: pre-delete
    helm.sh/hook-delete-policy: hook-succeeded
  Args:
  - --cleanup
expose:
  Annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: hook-succeeded
  Args:
  - --v
  - 4
  config:
    domain: cjxd.sharepointoscar.com
    exposer: Ingress
    http: "true"
    tlsacme: "false"
jenkins:
  Servers:
    Global: {}
prow: {}

cars-rest-api:
  env:
    MONGODB_USERNAME: vault:mongodb:mongodb-username
    MONGODB_PASSWORD: vault:mongodb:mongodb-password
    MONGODB_DATABASE: cars-rest-api

3. Modify App Helm Charts

There are several files we need to modify for two charts, the main chart is named cars-rest-api and the Preview app helm chart. Here is the structure depicting the charts in question.


➜ tree charts -l 2
charts
├── cars-rest-api
│   ├── Chart.yaml
│   ├── Makefile
│   ├── README.md
│   ├── requirements.lock
│   ├── requirements.yaml
│   ├── templates
│   │   ├── NOTES.txt
│   │   ├── _helpers.tpl
│   │   ├── deployment.yaml
│   │   ├── ksvc.yaml
│   │   └── service.yaml
│   └── values.yaml
└── preview
    ├── Chart.yaml
    ├── Makefile
    ├── requirements.yaml
    └── values..yaml

The requirements.yaml changes

First, we must add the dependencies to the main chart. There is nothing new here, we add the MongoDB dependency to our chart using Helm standard approach, which is by modifying the requirements.yaml as shown below.

dependencies:
- name: mongodb
  alias: cars-rest-api-db
  version: 7.6.5
  repository: https://kubernetes-charts.storage.googleapis.com
  condition: cars-rest-api-db.enabled

Next, we modify values.yaml and add to the env: variables as well as the mongoDB dependency, which I've aliased as cars-rest-api-db (shown on requirements.yaml above)

NOTE: It is important to notice that the reference to the Vault keys within the MongoDB chart parameters is within double-quotes. Took me 4 full days to realize this issue and I grew more gray hair for sure.

The values.yaml changes

The values.yaml file should look like this.

 .....
env:
  MONGDB_USERNAME: vault:mongodb:mongodb-username
  MONGDB_PASSWORD: vault:mongodb:mongodb-password
  MONGDB_ROOT_PASSWORD: vault:mongodb:mongodb-root-password
....

# MongoDB Configuration
cars-rest-api-db:
  ## Whether to deploy a mongodb server to satisfy the application database requirements.
  ## To use an external database set this to false and configure the externaldb parameters
  enabled: true

  ## Enable authentication
  ## ref: https://docs.mongodb.com/manual/tutorial/enable-authentication/
  #
  usePassword: true
  # existingSecret: jx-auth

  ## MongoDB custom user and database
  ## ref: https://github.com/bitnami/bitnami-docker-mongodb/blob/master/README.md#creating-a-user-and-database-on-first-run
  ##
  mongodbUsername: "vault:mongodb:mongodb-username"
  mongodbDatabase: "cars_rest_api"
  mongodbPassword: "vault:mongodb:mongodb-password"

  ## MongoDB admin password
  ## ref: https://github.com/bitnami/bitnami-docker-mongodb/blob/master/README.md#setting-the-root-password-on-first-run
  ##
  mongodbRootPassword: "vault:mongodb:mongodb-root-password"
  ...

The Preview Chart Changes

For the Preview chart, we modify two files. The values.yaml and the requirements.yaml, very similar changes. However, we name things differently.

On the requirements.yaml we alias the MongoDB as preview-db and make sure we add it below the existing dependencies, BUT before the preview chart definition as shown below.

 dependencies:
- alias: expose
  name: exposecontroller
  repository: http://chartmuseum.jenkins-x.io
  version: 2.3.92
- alias: cleanup
  name: exposecontroller
  repository: http://chartmuseum.jenkins-x.io
  version: 2.3.92

- name: mongodb
  alias: preview-db
  version: 5.3.0
  repository: https://kubernetes-charts.storage.googleapis.com

    # !! "alias: preview" must be last entry in dependencies array !!
  # !! Place custom dependencies above !!
- alias: preview
  name: cars-rest-api
  repository: file://../cars-rest-api

The values.yaml file contains the same environment variables as the main chart.

 .....
  env:
    MONGODB_USERNAME: vault:mongodb:mongodb-username
    MONGODB_PASSWORD: vault:mongodb:mongodb-password
    MONGODB_DATABASE: cars-rest-api
    MONGODB_ROOT_PASSWORD: vault:mongodb:mongodb-root-password
## MongoDB chart configuration
## ref: https://github.com/helm/charts/blob/master/stable/mongodb/values.yaml
##
preview-db:
  ## Whether to deploy a mongodb server to satisfy the applications database requirements.
  ## To use an external database set this to false and configure the externaldb parameters
  enabled: true

  ## Enable authentication
  ## ref: https://docs.mongodb.com/manual/tutorial/enable-authentication/
  # NOTE: make sure this secret is in 
  usePassword: true
  #existingSecret: preview-auth

  ## MongoDB custom user and database
  ## ref: https://github.com/bitnami/bitnami-docker-mongodb/blob/master/README.md#creating-a-user-and-database-on-first-run
  ##
  mongodbUsername: "vault:mongodb:mongodb-username"
  mongodbDatabase: "cars_rest_api"
  mongodbPassword: "vault:mongodb:mongodb-password"

    ## MongoDB admin password
  ## ref: https://github.com/bitnami/bitnami-docker-mongodb/blob/master/README.md#setting-the-root-password-on-first-run
  ##
  mongodbRootPassword: "vault:mongodb:mongodb-root-password"
.....

Summary

On this post, we walked through the required steps to ensure you are able to retrieve and interpolate Vault secrets within your custom app being put through CI/CD in Jenkins X.

Here is the video of the webinar session where I show it in action.

https://www.youtube.com/embed/LRj_yHKxG80

Cheers,

@SharePointOscar

Using Rollout.io Feature Flags for apps deployed via Jenkins X CI/CD

Oscar Medina — Fri, 30 Aug 2019 17:27:20 +0000

Overview

I believe any company that produces software, will always have the need to test if a new feature is working for its target audience. We know that giants like Facebook do this, and they do it well.

How can you elevate your feature deployment game? Use feature flags to target a group of users with a specific feature, get feedback and improve said feature all while leveraging Jenkins X CI/CD and Environments!

What are feature flags? Here is directly from rollout.io

Feature flags are at the core of Rollout.io. Using feature flags, you can easily enable and disable features in your application, giving you full control over how users are able to interact with your application.
- rollout.io

Prerequisites

To get started, you will need to do the following:

Sign up for a rollout.io account.
Ensure you have to Rollout Environments (Production, Staging)
You have jx installed and a cluster running in GKE for example.
This post is based on a Jenkins X QuickStart I created and is available to you already. It is called rollout-app. To follow along, simply create it using jx create quickstart, then select it.

Scenario

Our scenario is simple. We have a NodeJS application, and we would like to test if a button placement on the UI makes sense. So we will use Rollout.io Feature Flags to configure our NodeJS app accordingly.

The app will be put through CI/CD in Jenkins X. The Staging environment is the place where our intended audience will view the new feature, in this scenario it is just a button. No other environment will have that button show up.

Rollout Dashboard Configuration

Create a Custom Property, call it JenkinsX Environment of type string by going to left navigation App Settings > Custom Properties, click Add New Custom Property
Create a Flag within the Rollout Staging environment, by clicking on the left nav under Staging > Experiments, then click New Experiment and select Create Flag1 Confusing right, but that should work.

Call the flag jenkinsx environment, this will map to an internal name of ski-rollout.jenkinsx_environment which we create via code.

Create a Target Group and configure it as shown below

We are creating a Target Group that is targeting the Staging Environment in Jenkins X.

NOTE: Enter the values as strings (multiple), ensure that you add the exact name of your environment namespace, which you can obtain by executing jx get env and view the NAMESPACE column.

Create an Experiment and configure it to target the group you created in the previous step. It should be configured as follows:

Ensure, that the dropdown, is set to true.

The App Code

Integrating Rollout into our NodeJS application is actually quite simple. For this example, I've added the necessary code into the server.js file, ideally you organize your files as needed to avoid having this file cluttered.


var http = require('http');
var fileSystem = require('fs');
var Rox = require('rox-node');
var express = require('express');
var app = express();
var context= {};

//setup Rollout app settings container and flag
const appSettingsContainer = {
    jenkinsx_environment: new Rox.Flag()
  };

// this property must exist in the Rollout Dashboard.
Rox.setCustomStringProperty('JenkinsX Environment', function(context){
    return context.jenkinsx_environment;
  });

// change the name accordingly
Rox.register('ski-rollout', appSettingsContainer);


// Rollout Staging Env
async function setupRox() {
    console.log('calling Rox.setup for Staging...');

    // the parameter for setup, is the ID of the Staging Environment in the Rollout Dashboard.
    // you can use other environment IDs but those must be defined in the Rollout Dashboard.
    var _result =  await Rox.setup('5d016c4223864938a85c1d33', {

      });

    await sleep (2000);
    return _result;
 }


 setupRox().then((value) => {

    if (appSettingsContainer.jenkinsx_environment.isEnabled(context)) {
        console.log('----- We are in Staging Jenkins X environment! --------');
     }
     else {
        console.log('------ What Jenkins X environment? : '+ context.jenkinsx_environment+' ---------');
     }

 });


function getJXEnvironment() {
    var _env = '';
    _env = fileSystem.readFileSync('/var/run/secrets/kubernetes.io/serviceaccount/namespace', 'utf8');

    return _env;
}

// Routes - we pass two variables to the HTML to preform approrpiate actions based on conditions.
app.get('/', function(req, res) {

    // first ensure we have our file contents, which contains the k8s namespace we are in.
    context = { jenkinsx_environment: getJXEnvironment() };
    console.log('----------- app.get() - called getJXEnvironment() and got: '+ context.jenkinsx_environment+' so rendering ---------------------');
    res.render('pages/index',{env:context.jenkinsx_environment,renderButton:appSettingsContainer.jenkinsx_environment.isEnabled(context)});
});

app.listen(8080);

console.log('------ Ok your app is listening on port 8080! -------- ');

And here is what the end result looks like. Notice the URL and the button showing for this environment.

Conclusion

With this basic configuraton in place. Our button will only now show when the app is running in the staging environment. This post merely scratches the surface of what is possible with Feature Flags. I encourage you to explore releasing features in this manner.

Some of the nice capabilities I am looking forward to try are:

Targeting Rules
Gradual Rollout & Rollback
Multivariate Testing

Learn more at rollout.io

Cheers,

@SharePointOscar

NOTE: This post originally published at sharepointoscar.com

Increasing CI/CD Pipeline Observability in Jenkins X

Oscar Medina — Mon, 29 Jul 2019 22:32:59 +0000

Overview

You might have heard of Observability given that folks have been talking about this for a while now. Sure, you might think it is just the latest tech buzzword. However, the practice has been around for a long time now.

Observability is certainly relevant today given the Microservices architectures, distributed systems, and the characteristics of modern applications being deployed at a faster pace by leveraging CI/CD pipelines to Kubernetes, in this case using Jenkins X. Indeed old practices of setting up monitoring after an app is deployed, are no longer acceptable.

Let’s face it, modern apps call for modern instrumentation, not only once they are deployed; even at build-time having proper instrumentation can help you gain insights into what is happening at various stages of the build and release process. This may include spotting any latency issues, performance and dependency download times. In other words, instrumentation and monitoring should be baked into our deployment pipeline in Jenkins X!

Given that Jenkins X is the native CI/CD platform for Kubernetes, we must start thinking of Observability in the context of the build and release of our containerized applications via this platform, and not after the deployment process itself.

What we are doing today

Today, I walk you through the process of increasing observability in your build and release pipeline by implementing tracing for a couple of events such as npm install and npm test which are part of a sample NodeJS application.

NOTE: Tracing is only a small portion of other things that need to be in place. Logging and Metrics are also required. The combination and aggregation of this data allows you to understand how observable your pipeline is.

Diagram by: Peter Bourgon

Leveraging Third-Party Tools

Jenkins X was built with extensibility and flexibility in mind. Today, you can easily create QuickStarts for a language not implemented. You can also build addOns to augment the platform functionality. There are currently addOns for istio, prometheus and anchore to name a few. Given this extensibility, we encourage our community to build these components and share with everyone.

If you look around, you’ll find that Honeycomb.io is at the forefront of Observability. We are collaborating with them to eventually have a Honeycomb addOn for Jenkins X

In this post, we use the Honeycomb.io API to trace our pipeline events.

Tracing CI/CD Pipeline Events

In this scenario we want to trace start and end times for certain events. In our example NodeJS app, we have commands such as npm install and npm test, which are part of our build-pack pipeline out of the box. To do start tracing, we modify the Tekton pipeline and inject calls to the Honeycomb.io API before and after these specific build pack named steps.

NOTE: Please be sure to sign up for honeycomb.io to obtain your API Key

Create Kubernetes Secret

Once we have our API Key, we want to create a Kubernetes Secret which is required to make API calls within our pipeline. To do this, we create it in the jx and jx-staging namespaces. For each namespace execute the following command (be sure to modify the namespace value as needed).

>$ kubectl create secret generic honeycomb-creds —from-literal=BUILDEVENT_APIKEY=<KEY>  --namespace=<NAMESPACE>

Modify Tekton Pipeline

Now that we have our Kubernetes Secret in place, we will modify the jenkins-x.yaml file, which currently has exactly one line as follows:

buildpack: javascript

Let's go over the important components of the YAML file once modified. The first items I'd like to highlight, are the environment variables needed. We need to provide honeycomb.io three key pieces of information:

CI Provider: this is the JENKINS-X environment variable. Honeycomb will add additional metadata fields to our dataset
BUILDEVENT_DATASET: this indicates which dataset we want to populate (you can have many).
BUILDEVENT_APIKEY: the Kubernetes Secret value, which is the API Key provided via the honeycomb site

buildPack: javascript
pipelineConfig:
  env:
  - name: JENKINS-X
    value: JENKINS-X
  - name: BUILDEVENT_DATASET
    value: jx
  - name: BUILDEVENT_APIKEY
    valueFrom:
      secretKeyRef:
        key: BUILDEVENT_APIKEY
        name: honeycomb-creds

The build-pack used for a NodeJS app is javascript as detected by the language. Hence the single line we had as the contents of the jenkins-x.yaml file.

Because we know which build-pack is being used, we can determine which named steps exist. Typically they include an npm install and npm test.

Therefore, we want to inject a timestamp before and after each of these named steps are called. The following shows how I inject this.

  pipelines:
    overrides:
    - name: npm-install
      pipeline: pullRequest
      stage: build
      steps:
      - command: echo ===== pullrequest:build:before  sending honeycomb step trace  ===============
      - name: honeycomb-npm-install-set-step-start-timestamp
        sh: echo $(date +%s) > step_start
      - name: honeycomb-npm-install-before-timestamp
        sh: echo =================================  $(cat step_start)  =================================
      type: before

    - name: npm-install
      pipeline: pullRequest
      stage: build
      steps:
      - command: echo ===== pullrequest:build:after   sending honeycomb step trace  ===============
      - name: honeycomb-npm-install-after-timestamp
        sh: echo =================================  $(cat step_start)  =================================
      - name: honeycomb-step-log-after-npm-install
        sh: ./buildevents step "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" $(echo npm-install | sum | cut -f 1 -d \ ) $(cat step_start) npm-install
      type: after
    - name: npm-test
      pipeline: pullRequest
      stage: build
      steps:
      - name: honeycomb-npm-test-set-step-start-timestamp
        sh: echo $(date +%s) > step_start
      - name: honeycomb-npm-test-before-timestamp
        sh: echo =================================  $(cat step_start)  =================================
      type: before
    - name: npm-test
      pipeline: pullRequest
      stage: build
      steps:
      - name: honeycomb-npm-test-after-timestamp
        sh: echo =================================  $(cat step_start)  =================================
      - name: honeycomb-step-log-after-npm-test
        sh: ./buildevents step "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" $(echo npm-test | sum | cut -f 1 -d \ ) $(cat step_start) npm-test
      type: after

Now that I have captured timestamps for these two named steps, I want to send an API call to honeycomb as follows. You will notice how I am using a binary called buildevents this is downloaded during the setup of my pipelines which I discuss shortly.

By concatenating a few metadata pieces that exist as environment variables in Jenkins X, I build a unique name which is needed by honeycomb to track things correctly.

    - pipeline: pullRequest
      stage: build
      steps:
      - name: honeycomb-build-name-concat
        sh: echo the build is "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" and HONEYCOMB_BUILD_START=$(cat build_start)
      - name: honeycomb-send-success
        sh: ./buildevents build "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" $(cat build_start) success
      type: after

We do the same for the Release pipeline...

# release pipeline releated calls

    - name: npm-install
      pipeline: release
      stage: build
      steps:
      - command: echo ===== release:build:before  sending honeycomb step trace  ===============
      - name: release-honeycomb-npm-install-step-start-timestamp
        sh: echo $(date +%s) > release_step_start
      - name: release-honeycomb-npm-install-before-timestamp
        sh: echo ================================= release  release-npm-install step start  $(cat release_step_start)  =================================
      type: before

    - name: npm-install
      pipeline: release
      stage: build
      steps:
      - name: release-honeycomb-npm-install-after-timestamp
        sh: echo ================================= release npm-install step end  $(cat release_step_start)  =================================
      - name: release-honeycomb-step-log-after-npm-install
        sh: ./buildevents step "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" $(echo release-npm-install | sum | cut -f 1 -d \ ) $(cat release_step_start) release-npm-install
      type: after

    - pipeline: release
      stage: build
      steps:
      - name: release-honeycomb-build-name-concat
        sh: echo the build is "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" and HONEYCOMB_BUILD_START=$(cat release_start)
      - name: release-honeycomb-build-send-success
        sh: ./buildevents build "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" $(cat release_start) success
      type: after

This is portion of the pipeline executes first hence the setup node. There are several things we want to accomplish in the setup of our pipelines.

Download the Build Events binary provided by Honeycomb, make it executable
Create the timestamp we will use to track the pullRequest pipeline execution. I do this by saving a temporary file with the timestamp.

This is done for both pipelines, as we are working with both.


    pullRequest:
      setup:
        steps:
          - command: echo =========================== pullrequest:setup downloading Honeycomb.io buildevents binary ===========================
          - name: pullrequest-download-honeycomb-binary
            sh: curl -L -o buildevents https://github.com/honeycombio/buildevents/releases/latest/download/buildevents-linux-amd64
          - name: honeycomb-set-binary-permissions
            sh: chmod 755 buildevents
          - name: honeycomb-display-buildevents-version
            sh: ./buildevents --version
          - name: honeycomb-setup-build-timestamp
            sh: echo $(date +%s) > build_start
          - name: honeycomb-output-debug
            sh: echo the build is "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" and HONEYCOMB_BUILD_START=$(cat build_start)   =======================================================
    release:
      setup:
        steps:
        - command: echo =========================== release:setup downloading Honeycomb.io buildevents binary ===========================
        - name: release-download-honeycomb-binary
          sh: curl -L -o buildevents https://github.com/honeycombio/buildevents/releases/latest/download/buildevents-linux-amd64
        - name: release-honeycomb-set-binary-permissions
          sh: chmod 755 buildevents
        - name: release-honeycomb-setup-build-timestamp
          sh: echo $(date +%s) > release_start
        - name: release-honeycomb-output-debug
          sh: echo the build is "${APP_NAME}-${PULL_NUMBER}-${VERSION}-${BUILD_NUMBER}" and HONEYCOMB_BUILD_START=$(cat release_start)   =======================================================

Once this pipeline executes, the dashboard on the honeycomb.io site will show us the execution tracing as follows.

As you can see, we have a unique name for our build being traced, underneath that we are tracking the two events npm install and npm test time spans. We can easily see how long our dependencies are taking to download, and how long does it take to run the tests for the app.

Conclusion

Hopefully I’ve enticed you to at least look into why you might consider incorporating observability into your build and release process. There is a lot more that can be done. In a future post, we will cover additional setup.

Jenkins World | DevOps World 2019

I'll be demoing this solution at various times while at the conference this year. You can still register and get a big discount by using code: PREVIEW. Instead of your cost being $1499, with discount is only $799

You can find my full schedule on the official website

Cheers,

@SharePointOscar