Forem: Webhook Relay

Building simple Airtable integrations 🌟🔥

war-bunny — Mon, 02 Oct 2023 10:41:57 +0000

Let's learn how to integrate Airtable. In the first example, we will receive an HTML form from a website and add a row to Airtable. This can be used for:

You can use this form to:

Allow users to contact you
A way for users to submit bug reports
Collect user feedback
Waitlist signups for your product

Example form on https://synpse.net/contact/

Prerequisites

Webhook Relay account.
Airtable account with a workspace.
Static website of your choice. We are using websites hosted both on GitHub and Cloudflare pages, however with this setup it doesn't matter where the website is hosted.

Prepare Airtable

In order to start adding rows to Airtable you will need to prepare few things:

Create a new base
Find your base ID and table name. To do this, check your URL when you are in the table view. It should look something like this: https://airtable.com/appXXXX/tblXXXX/viwtXXXX. The first part is your base ID and the second part is your table name. From here, appXXXX is the base ID and tblXXXX is the table ID.

Create forwarding configuration

You will need to create forwarding config to a public destination here https://my.webhookrelay.com/new-public-destination. The URL will be https://api.airtable.com/v0/appXXXX/tblXXXX.

🚨 Don't just copy/paste the URL from the browser! they are not the same as your browser uses https://airtable.com/appXXXX/tblXXXX/viwtXXXX but we need to send webhooks to https://api.airtable.com/v0/appXXXX/tblXXXX 🚨

Don't try to send requests to this yet as we will need to set up an Airtable webhook integration first.

Setup Airtable webhook integration

Next step is to transform the incoming HTML form request into a webhook that will add or create a new record in your Airtable table:

Go to the Functions page
Create a new function and copy paste the code into the composer:

local json = require("json")
local time = require("time")

if r.RequestMethod == "POST" then 
    -- Only POST requests allowed 
else
    r:StopForwarding()
    return
end

-- Taking fields from the HTML form and 
-- using them to prepare Airtable webhook
local encoded_payload = {
  records={
      {
          fields= {
          Email= r.RequestFormData.email[1],
          Name = r.RequestFormData.name[1],
          Added= time.format(time.unix(), "2006-01-02", "UTC") ,
          Message = r.RequestFormData.message[1],
      }
      }

  }
}

-- Encoding payload to JSON
local encoded_payload, err = json.encode(encoded_payload)
if err then error(err) end

r:SetRequestHeader("Authorization", "Bearer " .. cfg:GetValue("SECRET_API_TOKEN"))
r:SetRequestHeader("Content-Type", "application/json")
r:SetRequestBody(encoded_payload)
r:SetRequestMethod("POST")

Once added:

Get your personal access token from here https://airtable.com/create/tokens. Give it a name and ensure it has data.records:write scope and set Access to the base you want to add records to.
Open CONFIG VARIABLES tab of the function and create a new variable SECRET_API_TOKEN with the value of your personal access token:

Setting up HTML form

We will use a simple HTML script. Replace the https://XXXXXX.hooks.webhookrelay.com URL with your own Webhook Relay input endpoint. You can find it in the Bucket details:

<iframe name="form-i" style="display:none;"></iframe>
<!-- Forwarding to Webhook Relay which transforms requests and adds it to the Airtable as a new row -->
<form action="https://XXXXXX.hooks.webhookrelay.com" method="post" target="form-i">
  <div>
    <label for="name">Name:</label>
    <input type="text" id="name" name="name" required>
  </div>
  <div>
    <label for="email">Email:</label>
    <input type="email" id="email" name="email" required>
  </div>

  <div>
    <label for="message">Message: </label>
    <br>
<textarea name="message" id="message" rows="10" cols="40">
</textarea>
  </div>
  <input type="submit" value="Submit">
</form>

I have made a simple CodePen example here https://codepen.io/defiant77/pen/gOQeRMm which you can open and edit the webhooks URL with the one from your bucket.

Testing it out

Fill in the form and click submit. In your Airtable you should be able to see the new record. If you can't, check out webhook logs in Webhook Relay's bucket details.

Troubleshooting

There are very few moving parts here but if something doesn't work, things to check:

Form action URL is pointing at the bucket that you have created.
Function is attached to the output. It's required to have it as it needs to do the transformation.
The destination URL should be pointing at the correct base ID and table ID in Airtable.
Personal access token needs to have permissions to write (data.records:write) and access is allowed to the base that you are working with.

That's it! 😃 You can also make a very similar integration for Google Sheets as the process is the same - using webhook as an API call.

Originally posted on Webhook Relay blog:
https://webhookrelay.com/blog/2023/08/07/airtable-integrations/

Easy and secure Jenkins Operator deployment on Kubernetes

Karolis — Mon, 06 Jul 2020 10:23:45 +0000

In this tutorial, we will configure a Jenkins pipeline on Kubernetes that leverages Jenkins and Webhook Relay operators. Jenkins Kubernetes operator will be creating Jenkins instances with a predefined seed job. Webhook Relay operator will ensure that GitHub webhooks on push events trigger new Jenkins builds for a fast and efficient CI/CD experience.

Advantages of this setup:

Your Jenkins instance is only accessible through kubectl port-forward while maintaining the ability to receive webhooks from public destinations.
Jenkins pipeline configuration is stored in Git.
Webhook Relay routing configuration is stored in Git, the same as the Jenkins itself.

You can read about operator pattern in Kubernetes docs.

Prerequisites

Prerequisites:

Helm
Webhook Relay account
Kubernetes environment, Minikube, k3s, GKE, AKS, etc. are fine.
Configured kubectl
Git

Installation

The installation will consist of several steps:

Installing Jenkins operator
Installing Webhook Relay operator

Create a fresh namespace

Let's start by creating a new namespace where we will put our Jenkins instance and run builds. I will call it 'jenkins' but you can choose any other name:

kubectl create namespace jenkins

And then switch to it:

kubectl config set-context $(kubectl config current-context) --namespace=jenkins

Jenkins Operator

We will install Jenkins operator using Helm. First, add the repository:

helm repo add jenkins https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart
helm repo update

Once the repository has been added, install it:

helm install jenkins-operator jenkins/jenkins-operator

Official docs can be found here: https://jenkinsci.github.io/kubernetes-operator/docs/installation/. The operator is not the Jenkins itself so to get our Jenkins instance, we will have to create a Custom Resource.

Custom resources are extensions of the Kubernetes API. This page discusses when to add a custom resource to your Kubernetes cluster and when to use a standalone service. It describes the two methods for adding custom resources and how to choose between them.

Start Jenkins (using Custom Resource)

We will need to create a CR. You can either use Jenkins Operator docs to create one or you can fork this https://github.com/webhookrelay/jenkins-operator-example.git repository and clone it. Then:

Update jenkins_cr.yaml file https://github.com/webhookrelay/jenkins-operator-example.git to your own repository fork (it will usually be https://github.com/<your username or organization>/jenkins-operator-example.git).
Create it with kubectl:

  kubectl apply -f jenkins_cr.yaml

Main differences in this file from the stock example:

Added github plugin as we will need it to trigger jobs
Seed job got githubPushTrigger: true set as well

Creating this PR should result in two additional containers:

kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
jenkins-jenkins                           1/1     Running   0          7m11s
jenkins-operator-6dbbc458c9-gmx6p         1/1     Running   0          18m
seed-job-agent-jenkins-65cc4bc684-9ztr5   1/1     Running   0          6m21s

Let's connect to Jenkins. First, get username and password:

kubectl --namespace jenkins get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.user}' | base64 -d
kubectl --namespace jenkins get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.password}' | base64 -d

Then, in one terminal start port forwarding:

kubectl port-forward jenkins-jenkins 8080:8080

And then just open http://localhost:8080 in your browser.

Webhook Relay

Retrieve your access token key & secret pair from https://my.webhookrelay.com/tokens and set them as an environment variables:

export RELAY_KEY=xxxxxxxxxxxx
export RELAY_SECRET=xxxxx

Add Webhook Relay Operator Helm repository and install it:

helm repo add webhookrelay https://charts.webhookrelay.com
helm repo update
helm upgrade --install webhookrelay-operator --namespace=jenkins webhookrelay/webhookrelay-operator \
  --set credentials.key=$RELAY_KEY --set credentials.secret=$RELAY_SECRET

Operator doesn't forward webhooks on its own. Each created CR will ensure an agent deployment that is configured to route specific buckets.

From the operator example repository we will need to create Webhook Relay Custom Resource:

kubectl apply -f webhookrelay_cr.yaml

Note that if you have modified Jenkins CR name you will need to update webhookrelay_cr.yaml "destination" field from destination: http://jenkins-operator-http-jenkins:8080/github-webhook/ to whatever your current Jenkins service is. Typically it will be in a format jenkins-operator-http-<CR name>.

GitHub Configuration

This step could be automated by making Jenkins automatically configure Github repositories to forwarding to this endpoint, however for simplicity and so that it's more clear how it works, we will add this URL manually.

Get your Webhook Relay public URL

To get your public endpoint you can either visit https://my.webhookrelay.com/buckets page or get it via CR status:

kubectl get webhookrelayforwards.forward.webhookrelay.com forward-to-jenkins -o 'jsonpath={.status.publicEndpoints[0]}'

Result should look something like:

$ kubectl get webhookrelayforwards.forward.webhookrelay.com forward-to-jenkins -o 'jsonpath={.status.publicEndpoints[0]}'
https://k0yv9ip5sxxp55ncsu936k.hooks.webhookrelay.com

Add public URL to GitHub repository settings

Take the public endpoint URL and add it to your GitHub repository:

Using the pipeline

First, when the pipeline is created, trigger the build manually. After that, any push to your GitHub repository will send a webhook through Webhook Relay to your Jenkins instance that's running inside a Kubernetes cluster:

This tutorial was initially published on Webhook Relay website: https://webhookrelay.com/v1/tutorials/webhooks-jenkins-operator-kubernetes.html

Serverless webhook transformation (DockerHub to Slack)

Karolis — Fri, 28 Feb 2020 08:49:56 +0000

Many Docker registries provide a way to notify team in chat channels when new images are pushed (if you are waiting for a build complete). Let's add this capability to the official DockerHub registry! :)

We will use newly released Webhook Relay Functions to transform webhooks while they are passing through the service.

Prerequisites:

Create a bucket and configure DockerHub notification

Create a bucket here https://my.webhookrelay.com/buckets
Once you have it, in the inputs section you will find your public input endpoint, copy it:
Add a new DockerHub webhook setting pointing at our public input endpoint (DockerHub docs):

Get a sample of DockerHub webhook

Push a new Docker image:

$ docker push karolisr/demo-webhook:latest
The push refers to repository [docker.io/karolisr/demo-webhook]
48bd38e03c42: Mounted from karolisr/webhook-demo
fd9f9fbd5947: Mounted from karolisr/webhook-demo
5216338b40a7: Mounted from karolisr/webhook-demo
latest: digest: sha256:703f2bab2ce8df0c5ec4e45e26718954b09bf4a625ab831c6556fd27d60f1325 size: 949

We should be able to see a new incoming webhook. It looks like this:

{
    "push_data": {
        "pushed_at": 1582839308,
        "images": [],
        "tag": "latest",
        "pusher": "karolisr"
    },
    "callback_url": "https://registry.hub.docker.com/u/karolisr/demo-webhook/hook/242ii4ddc2jji4a0cc44fbcdbcdecj1ab/",
    "repository": {
        "status": "Active",
        "description": "",
        "is_trusted": false,
        "full_description": "",
        "repo_url": "https://hub.docker.com/r/karolisr/demo-webhook",
        "owner": "karolisr",
        "is_official": false,
        "is_private": true,
        "name": "demo-webhook",
        "namespace": "karolisr",
        "star_count": 0,
        "comment_count": 0,
        "date_created": 1524557040,
        "repo_name": "karolisr/demo-webhook"
    }
}

Create a Function to transform the webhook

Go to the Functions page and click on a "Create Function" button. Enter a name, for example "dockerhub-to-slack" and click "Submit".

For this example we are using Lua language which provides an easy way to parse, transform payloads. Webhook Relay Functions also support WebAssembly modules, however they are currently under development and can only be added/updated using relay CLI.

You can now copy/paste webhook payload into the "request body" area for later tests. In the code editor let's add a Lua function to get repository name and prepare a Slack webhook payload:

local json = require("json")

local body, err = json.decode(r.RequestBody)
if err then error(err) end

local message = "New image pushed at: " .. body["repository"]["repo_name"] .. ":" .. body["push_data"]["tag"]

-- Preparing Slack payload
local slack = {
    response_type= "in_channel", 
    text= message}

local result, err = json.encode(slack)

-- Set request header to application/json
r:SetRequestHeader("Content-Type", "application/json")
-- Set request method to PUT
r:SetRequestMethod("POST")
-- Set modified request body
r:SetRequestBody(result)

Click "Save" and then try testing it with the "Send" button:

Connect everything together

Navigate to https://api.slack.com/messaging/webhooks and click "Create your Slack app". Select your workspace, enter a name that you will remember.
Create a new incoming webhook configuration, copy "Webhook URL" (it starts with https://hooks.slack.com/services/T3...), we will need to supply it to Webhook Relay.
Open your bucket details (via https://my.webhookrelay.com/buckets)
Open "OUTPUT DESTINATIONS" tab and create a new output called "Slack" with the Slack URL from step 2:
Once created, click on the "code" symbol and from the dropdown select dockerhub_to_slack function:

Push new image to DockerHub, you should see a new notification in your Slack channel:

That's it, feel free to continue modifying Lua function to include pusher's name and message format. Following this process you can transform any webhook into any other webhook.

Have fun!

Originally released on https://webhookrelay.com/v1/examples/convert-dockerhub-webhook-to-slack.html

A simple way to keep your Vue page title in sync with the router

Karolis — Sun, 08 Sep 2019 22:11:10 +0000

I have noticed in quite a few projects that developers forget to keep page title updated with the router or maybe think that they will do it tomorrow as it is such a small feature :). It always makes sense to keep the title synchronized with the contents for several reasons:

helps users with more than one tab
important for website analytics

I will show you how to do it with the standard vue-router.

Step 1: Declare route meta in your router config

First things first, let's add some additional metadata to our standard routes:

  routes: [
    {
      path: '/',
      name: 'home',
      component: HomePage,
      meta: {
        auth: true,
        title: 'Dashboard'
      }
    }, {
      path: '/login',
      name: 'login',
      component: LoginPage,
      meta: {
        auth: false,
        title: 'Login'
      }
    }, {
      path: '/buckets',
      name: 'buckets',
      component: BucketsPage,
      meta: { auth: true, title: 'Buckets' }

More docs on the router package can be found in the official vue router website.

Step 2: Add $route watcher in your App.vue

Go to your Vue main .vue file (it should have a <router-view></router-view> component) and add a watcher:

    watch: {
      '$route' (to, from) {
        document.title = to.meta.title || 'Your Website'
      }
    },

This will set a title as your page title (if you specified a meta field for that route). I have tried to achieve the same with router navigation guards but $route watcher seemed like the simplest solution.

Looks easy, right? :) Feel free to experiment and define more fields in the router meta like page description and anything else you want to be set for that page. Then modify the watcher to also use them.

Hope this helps!

Creating a simple Vue pluralize filter

Karolis — Sat, 07 Sep 2019 17:50:14 +0000

There are many options how to create a pluralize function but in Vue you should probably use filters. Let's create one as it's always handy to have one.

Step 1: Add pluralize package

Let's use https://github.com/blakeembrey/pluralize to do the heavy lifting:

yarn add pluralize

Step 2: Register filter

Our pluralize filter will take two arguments - first one is the left variable in the filter and second is the filter function argument (depends on your code structure, but it's usually main.js or a dedicated filters file):

// .. your other imports
import pluralize from 'pluralize'

// .. your other code

Vue.filter('pluralize', function (value, number) {
  return pluralize(value, number)
})

Step 3: Use the filter!

Now, to use the filter on the left we give it the word we want to pluralize and as an argument we pass the count:

{{ tunnels }} {{ 'tunnel' | pluralize(tunnels) }}

That's it :) Now, based on tunnels variable it will be either '1 tunnel' or '50 tunnels'.

I hope this will be useful to you once you need it!

Docker Compose update on GitHub release webhook

Karolis — Mon, 02 Sep 2019 17:46:52 +0000

Last year I wrote a blog post about combining several tools to automate simple NodeJS app updates on git push. Many users were solving similar problems by writing local web servers in Ruby, Python or PHP to receive webhooks and then do async processing. I am happy to announce that we finally decided to add this feature to the relay CLI. Now, to execute a bash script, you can:

relay forward --bucket my-bucket-name --relayer exec --command bash update.sh

And to launch a Python app on webhook:

relay forward --bucket my-bucket-name --relayer exec --command python my-app.py

This opens up some interesting posibilities to create pipelines that can react to pretty much anything that emits webhooks. In this article I will show you how to build a GitOps style pipeline that keeps a Docker Compose deployment in sync with a docker-compose.yaml hosted on a git repository.

Prerequisites

Docker & Docker Compose
Webhook Relay account with configured relay CLI
Github account

Repository with scripts that I used for this article can be found here: https://github.com/webhookrelay/docker-compose-update-on-git-push.

Step 1: Deploying containers through Docker Compose

First step is to do the initial deployment. We will create a simple dockerized Python application that you can find here that connects to Redis and deploy it:

version: '3'
services:
  web:
    image: "karolisr/python-counter:0.1.0"
    ports:
      - "5000:5000"
  redis:
    image: "redis:alpine"

Step 2: Setting up updates on Github tag

Since we only want to update on git tags and not just any pushes, let's configure a webhook and analyse the payload.

To achieve that, let's first create a bucket with an internal output:

$ relay forward --bucket docker-compose-update-on-git-push http://localhost:4000
Forwarding:
https://my.webhookrelay.com/v1/webhooks/a956a9f7-2260-4bc2-a54b-3d896acf4206 -> http://localhost:4000
Starting webhook relay agent...
2019-08-28 23:14:41.773 INFO    using standard transport...
2019-08-28 23:14:41.928 INFO    webhook relay ready...  {"host": "my.webhookrelay.com:8080", "buckets": ["8e977e70-09a6-464c-ad30-855e1cd5d9f9"]}

Here, bucket will be used later to subscribe to github requests while destination is just a mandatory argument that we don't have to use in this case.

Grab that https://my.webhookrelay.com/v1/webhooks/*** URL and go to your repository's settings -> webhooks section. Once there, set:

Payload URL to your unique https://my.webhookrelay.com/v1/webhooks/*** URL
Content type to application/json
Secret to a random secret name, for the sake of this example my secret will be 'webhooksecret'
Click Let me select individual events. and select Releases.

Now, go to your repository's releases page (for example https://github.com/webhookrelay/docker-compose-update-on-git-push/releases) and make a new release 1.0.0. Then, if you visit bucket details page or logs page - you should see webhook from Github. Open it and let's inspect the payload. It's quite lengthy but we should be able to see

 "action": "released",

in the top. To ensure that we only react on these events, create a rule:

If you tag another release, you should see now that only one webhook was forwarder

Step 3: Update script and starting relay background service

Our update script is:

#!/bin/bash

git pull
docker-compose up -d

It will pull the latest compose file and update containers. Now, let's update configuration in relay.yml file (access key & secret can be generated here):

version: v1
key: xxx     # your access key
secret: xxx  # your access secret
buckets:
- docker-compose-update-on-git-push # your bucket name where github webhooks are sent
relayer:
  type: exec
  command: bash
  commandArgs:
  - /full/path/to/docker-compose-update-on-git-push/update.sh # <-- should be full path to your update script
  timeout: 300

To start the relay, run:

relay run -c relay.yml

This will be running it through your terminal. For production use cases, please use background service mode. It will ensure that the daemon is launched on OS startup.

Let's try it out

Launch docker-compose:

docker-compose up -d

Check containers:

$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                    NAMES
26cd2219e18b        redis:alpine                    "docker-entrypoint.s…"   20 seconds ago      Up 3 seconds        6379/tcp                 docker-compose-update-on-git-push_redis_1
63c8cd1ae7bb        karolisr/python-counter:0.1.0   "flask run"              20 seconds ago      Up 18 seconds       0.0.0.0:5000->5000/tcp   docker-compose-update-on-git-push_web_1
$ curl http://localhost:5000
I have been seen 1 times.
$ curl http://localhost:5000
I have been seen 2 times.

Next step would be to build a new image 0.2.0 and push it to the registry. Once it's available, we can update our github repository docker-compose.yml and make a new release. For the sake of this example, let's do this through the Github UI.

In a few seconds you should see a new container running:

$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                    NAMES
27b2542423ec        karolisr/python-counter:0.2.0   "flask run"              9 seconds ago       Up 7 seconds        0.0.0.0:5000->5000/tcp   docker-compose-update-on-git-push_web_1
26cd2219e18b        redis:alpine                    "docker-entrypoint.s…"   10 minutes ago      Up 9 minutes        6379/tcp                 docker-compose-update-on-git-push_redis_1

Optional: Validating Github secret

Webhook Relay output rules can also validate Github signature:

This will ensure that only webhooks signed by Github will be processed.

Alternative solutions

I have somewhat successfully used these tools to update containers:

Both tools work if you have a simple setup that just needs to track latest tags. However, if you have multiple containers with semver (or any other versioning mechanism) - those solutions do not work. That's why it's useful to have an either .env or docker-compose.yml config with image tags committed into git and synchronized into the host that runs the services.

Conclusion

As with any code executed on your machine - you have to be careful when automating tasks. Webhook Relay will provide you with a unidirectional flow of webhooks into the machine. Your script/applications are on your machine and cannot be modified remotely through Webhook Relay. Coupled with authenticated webhook endpoints (you can configure it on a bucket level) or webhook payload checksum validation - you can build a secure update mechanism.

Originally released on https://webhookrelay.com/blog/2019/09/02/docker-compose-update-on-github-webhooks/

Using Google Firestore for a Golang backend application

Karolis — Mon, 26 Aug 2019 18:08:39 +0000

Usually, when I need a database I just pick Postgres or embedded key-value stores such as the excellent boltdb, badger from dgraph or Redis (if I need a KV store but shared between several nodes). With flexibility comes the burden of maintenance and sometimes additional cost. In this article, we will explore a simple Golang backend service that will use Google Firestore as storage.

When I started working on a simple project called bin.webhookrelay.com, I picked Badger as a key-value store, attached a persistent disk to a Kubernetes pod and launched it.

bin.webhookrelay.com is a free service that allows you to capture webhook or API requests for testing purposes. It also lets you specify what response body and status code to return, as well as set an optional response delay.

The data model was (and still is) simple:

Bin that has a certain configuration like what status code to return, response body, content-type header, and response delay.
Request which is the actual captures webhook request with bin ID, request body, headers, and query.

Most of the time, KV stores such as boltdb or badger are great for such use cases. Problems arise when you want to either scale horizontally or have a rolling update strategy meaning that a number of instances of your application would have to surge during the update. While Kubernetes is great for running pretty much any workload, an update where it has to detach a persistent disk and reattach it to a new pod can lead to downtime and just generally slow updates. I always try to avoid such scenarios, however, webhook bin service was suffering from it.

This time, I decided to try out Cloud Firestore. You probably have already heard about Firestore (previously known as Firebase) and that it is very popular amongst mobile app developers who need to have a database for their Android and iOS apps. Apparently, it can also provide a really nice developer UX for backend applications! :)

Setting it up

For authentication, Golang Firestore client uses a standard mechanism that relies on a service account. Basically, you need to go to:

GCP console
Click on IAM & admin
Go to service accounts (on the left navigation bar)
Create a new service account with Firestore permissions
Download the file, you can now use it for authentication

Docs can be found in the Google Cloud authentication section.

Application code is surprisingly simple. You get the client using Google application credentials, project ID and that's it:

func NewFirestoreBinManager(opts *FirestoreBinManagerOpts) (*FirestoreBinManager, error) {
    ctx := context.Background()

    var options []option.ClientOption
    if opts.CredsFile != "" {    
        options = append(options, option.WithCredentialsFile(opts.CredsFile))
    }

  // credentials file option is optional, by default it will use GOOGLE_APPLICATION_CREDENTIALS
  // environment variable, this is a default method to connect to Google services
    client, err := firestore.NewClient(ctx, opts.ProjectID, options...) 
    if err != nil {
        return nil, err
    }

    return &FirestoreBinManager{
        binsCollection: opts.BinsCollection,  // our bins collection name
        reqsCollection: opts.ReqsCollection,  // our requests collection name
        client:         client,
        pubsub:         opts.Pubsub,
        logger:         opts.Logger,
    }, nil
}

Adding and updating bins

When creating a document, you can specify document ID and just pass in the whole golang struct without first marshaling it into JSON:

func (m *FirestoreBinManager) BinPut(ctx context.Context, b *bin.Bin) (err error) {
    _, err = m.client.Collection(m.binsCollection).Doc(b.GetId()).Set(ctx, b)
    return err
}

Note that we supply collection name: Collection(m.binsCollection), ID: Doc(b.GetId()) and set the struct fields Set(ctx, b).

This really saves time! An alternative with a KV store would be something like:

func (m *BinManager) BinPut(ctx context.Context, b *bin.Bin) (err error) {
    b.Requests = nil
    encoded, err := proto.Marshal(b)
    if err != nil {

        return err
    }

    return m.storage.Store(ctx, "bins/"+b.Id, encoded, nil)
}

...
// store package


func (s *Storage) Store(ctx context.Context, id string, data []byte, metadata map[string]string) error {
    err := s.db.Update(func(txn *badger.Txn) error {
        // Your code here…
        return txn.Set([]byte(id), data)
    })
    return err
}

Deleting

To delete a bin in our case means deleting both the bin document and all associated webhook requests with it:


func (m *FirestoreBinManager) BinDelete(ctx context.Context, binID string) error {

    _, err := m.client.Collection(m.binsCollection).Doc(binID).Delete(ctx)
    if err != nil {
        m.logger.Errorw("failed to delete bin doc by ref",
            "error", err,
        )
    }

    // Now, get all the requests and delete them in a batch request
    iter := m.client.Collection(m.reqsCollection).Where("Bin", "==", binID).Documents(ctx)
    numDeleted := 0
    batch := m.client.Batch()

    for {
        doc, err := iter.Next()
        if err == iterator.Done {
            break
        }
        if err != nil {
            return fmt.Errorf("Failed to iterate: %v", err)
        }
        batch.Delete(doc.Ref)
        numDeleted++

    }
    // If there are no documents to delete,
    // the process is over.
    if numDeleted == 0 {
        return nil
    }
    _, err = batch.Commit(ctx)

    return err
}

Limitations

While it is very easy to store, retrieve and modify documents, some people will miss SQL type queries that can aggregate, count records and do other useful operations in the database. For example, to track document counts, you will have to implement a solution similar to one described here. My suggestion would be to spend more time planning data structure and what kind of operations are you planning to use before embarking on this journey :)

Conclusion

While being a bit skeptical at first, I quickly started liking Firestore. While running a managed Postgres would allow me to easier switch cloud providers, it would also make it more expensive to run. Keeping storage interface small means that you can implement Postgres (or any other database) driver in a matter of hours so then the most important things to look for are:

How much maintenance the solution requires
Cost
Performance

Useful resources:

This article was originally published on Webhook Relay blog.

Automated Jenkins builds on GitHub pull request

Karolis — Wed, 17 Apr 2019 14:34:50 +0000

In this short guide we will configure Jenkins to start builds on GitHub pull requests. Subsequent builds will be triggered on any new commits and GitHub pull request status will show whether build succeeded or failed. This setup will work without configuring router, firewall or having a public IP. It will also work behind a corporate firewall.

Getting a VM

In my case, I just grabbed a Vagrant box from https://app.vagrantup.com/ubuntu/boxes/xenial64:



Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/xenial64"
  config.vm.network "private_network", type: "dhcp"
end

Then:



vagrant up
vagrant ssh

And we have our VM. You can get the IP address by typing ifconfig in the terminal.

Installing Jenkins

I mostly followed this guide https://linuxize.com/post/how-to-install-jenkins-on-ubuntu-18-04/. The only caveat I encountered this time with Jenkins, was the jdk version mismatch.

Connecting to Jenkins

First, get your Jenkins token:



cat /var/lib/jenkins/secrets/initialAdminPassword
ce04d19270934633a7badcac3cfac316

Then, either open your node firewall (or check Vagrant port forwarding) or do the easy thing: connect with the relay:

Get CLI

To get the CLI, check instructions here. On a 64-bit Linux OS it's:



curl -sSL https://storage.googleapis.com/webhookrelay/downloads/relay-linux-amd64 > relay \
   && chmod +wx relay && sudo mv relay /usr/local/bin

Login

Go to https://my.webhookrelay.com/tokens, click CREATE TOKEN and copy/paste login command into the terminal, it should be something like:



relay login -k <your key> -s <your secret>

Start tunnel



$ relay connect :8080
Connecting: 
http://lsw7eq49jlhsuldvhpiyku.webrelay.io <----> http://127.0.0.1:8080
https://lsw7eq49jlhsuldvhpiyku.webrelay.io <----> http://127.0.0.1:8080

Now, open the browser. You should see a similar screen:

Follow the steps to configure your Jenkins initial admin user.

Installing the plugin

Plugin installation instructions can be found here.

Once you have it, add GitHub credentials - your username and GitHub token.

Forwarding configuration

Go to your bucket configuration and create a bucket called github-webhooks. Configure it to forward all webhooks to http://localhost:8080/. This will ensure that webhooks will reach Jenkins server.

Once you have the relay CLI on the machine where you run Jenkins, type:



relay forward --bucket github-webhooks

This will start forwarding webhooks. There are alternative options to run the forwarding daemon, such as Docker container.

If you are creating webhook configuration manually in GitHub, use http://localhost:8080/ghprbhook destination as it's the endpoint on which the plugin is listening for webhooks. In default case, Jenkins will automatically transform https://my.webhookrelay.com/v1/webhooks/21e13033-bd3d-47a2-bf15-6fd42d4b40a3 endpoints to https://my.webhookrelay.com/v1/webhooks/21e13033-bd3d-47a2-bf15-6fd42d4b40a3/ghprbhook and
Webhook Relay will preserve the extra /ghprbhook path.

Now, configure GitHub Pull Request Builder:

Creating a job

To create a new job, first select "Freestyle project", then:

Add the project's GitHub URL to the "GitHub project" field (the one you can enter into browser. eg: "https://github.com/rusenask/jenkins-test/"):

Configure Source Code Management section:

Select Git SCM.
Add your GitHub "Repository URL".
Under Advanced, set "refspec" to +refs/pull/*:refs/remotes/origin/pr/*
In "Branch Specifier", enter ${ghprbActualCommit}

Configure Build Triggers with a list of admins and tick the use github hooks for build triggering:

Add your Build step configuration. This can be anything you want, usually people tend to use either a bash script, Makefile target or something specific to your programming language such as go build:

Opening a PR

Now, whenever you open a new pull request in GitHub, you should see a build being triggered:

You can view build status in your Jenkins instance as well. This build indicator in GitHub will either turn red or green based on the build status.

Conclusion

As we can see, there are several required steps to make sure your PRs get automatically built and tested when using Jenkins. Those can be split into two groups:

System configuration that involves:
- setting Webhook Relay agent to forwarding webhooks
- installing and configuring GitHub pull request builder plugin
Whenever you create a new project in Jenkins, setting up few settings. The first time I did this it took me some time to go through the configuration options, but second and third time didn't take more than 1 minute :)

I hope you will find this guide useful!

P.S. Bonus troubleshooting below:

When we are talking about Jenkins, there are many ways for things to go wrong. Multiple plugin versions, corporate proxies and different operating systems contribute to all of this. I have compiled a short list of items for you to check if you encounter problems.

Ensuring webhook configuration

Make sure there's an automatically created GitHub repository webhook configuration:

Ensuring Webhook Relay agent can connect

Normally, connected agent should look like this:



relay forward --bucket github-webhooks
Filtering on bucket: github-webhooks
Starting webhook relay agent... 
1.55523552627511e+09    info    using standard transport...
1.5552355264042027e+09  info    webhook relay ready...  {"host": "my.webhookrelay.com:8080"}

If you are behind a corporate proxy, try adding --ws flag to change default transport type from GRPC to WebSocket:



relay forward --ws --bucket github-webhooks
Filtering on bucket: github-webhooks
Starting webhook relay agent... 
1.5552387754607065e+09  info    using websocket based transport...
1.5552387754607568e+09  info    authenticating to 'wss://my.webhookrelay.com:443/v1/socket'...
1.5552387754608495e+09  info    websocket reader process started...
1.555238775470567e+09   info    subscribing to buckets: [github-webhooks

Check logs

There will be several sources of logs you can check out:

Jenkins logs under /log/all:

Webhook Relay forwarding logs:

CLI logs:



relay forward --bucket github-webhooks
Filtering on bucket: github-webhooks
Starting webhook relay agent... 
1.55523552627511e+09    info    using standard transport...
1.5552355264042027e+09  info    webhook relay ready...  {"host": "my.webhookrelay.com:8080"}
1.555236773074343e+09   info    webhook request relayed {"destination": "http://localhost:8080/ghprbhook/", "method": "POST", "bucket": "github-webhooks", "status": 200, "retries": 0}
1.5552368184301443e+09  info    webhook request relayed {"destination": "http://localhost:8080/ghprbhook/", "method": "POST", "bucket": "github-webhooks", "status": 200, "retries": 0}
1.5552368215106862e+09  info    webhook request relayed {"destination": "http://localhost:8080/ghprbhook/", "method": "POST", "bucket": "github-webhooks", "status": 200, "retries": 0}
1.555236829308788e+09   info    webhook request relayed {"destination": "http://localhost:8080/ghprbhook/", "method": "POST", "bucket": "github-webhooks", "status": 200, "retries": 0}
1.5552368314174337e+09  info    webhook request relayed {"destination": "http://localhost:8080/ghprbhook/", "method": "POST", "bucket": "github-webhooks", "status": 200, "retries": 0}
1.555236920064973e+09   info    webhook request relayed {"destination": "http://localhost:8080/ghprbhook/", "method": "POST", "bucket": "github-webhooks", "status": 200, "retries": 0}
1.5552369202506151e+09  info    webhook request relayed {"destination": "http://localhost:8080/ghprbhook/", "method": "POST", "bucket": "github-webhooks", "status": 200, "retries": 0}

Originally published on Webhook Relay blog: https://webhookrelay.com/blog/2019/04/17/automated-github-pull-request-builds-on-jenkins/

Automating testing, building and publishing of TypeScript libraries

Karolis — Sun, 17 Feb 2019 12:36:00 +0000

It doesn't matter whether you are working on a side project, small open source library or your full-time job project, automating builds, tests and releases can greatly improve your life. You can then concentrate on code quality, features or just have a small break when you finish a task instead of trying to remember all the required steps to make a release.

In my previous article I demonstrated how to set up a self-hosted CI/CD solution with Drone. You don't need a powerful CI server or expensive VMs to run it, you can easily get one running on your laptop to perform these tasks in the background a lot faster than the free alternatives while also getting much greater flexibility.

Now, I would like to share some practical pipelines that I have recently implemented.

A short disclaimer: I don't identify myself as an experienced TypeScript/JavaScript developer, I always lean to Go but in this case I needed to write some JavaScript so it was a great opportunity to finally try out TypeScript :) The package itself can be found here, it's a simple library that allows you to receive webhooks inside your app without exposing it to the internet.

Testing the library

My library tests were probably not what you find in a standard library. Since they rely on the SaaS service (to receive those public webhooks), it has to get some credentials from environment and perform asynchronous actions. That's where I learnt about done callback:

it('should be able to forward the webhook', (done) => {
    var payload = "payload-" + Math.floor((Math.random() * 100000) + 1);
    // creating a handler
    var handler = function (data: string) {
      var msg = JSON.parse(data);
      if (msg.type === 'status' && msg.status == 'subscribed') { // <---- once received, send a webhook
        var dispatchWebhook = function() {
          axios.post('https://my.webhookrelay.com/v1/webhooks/9c1f0997-1a34-4357-8a88-87f604daeca9', payload)
          .then(function (response) {          
            expect(response.status).to.equal(200)
          })
        }
        setTimeout(dispatchWebhook, 1000)

      }
      if (msg.type === 'webhook' && msg.body === payload) {
        expect(msg.method).to.equal('POST');
        done(); // <---- once webhook received, end the test case
        client.disconnect();
      }
    }

    var client = new WebhookRelayClient(key, secret, [testBucket], handler)
    client.connect(); // <---- connecting so our handler will be called
  });

While this not really related to automation, it might be useful for someone :)

Building the library

When using Drone, everything runs in a Docker container. The main benefit of this is that it becomes trivial to get a reproducible builds. In our case, the first step includes:

Install dependencies
Build with tsc (TypeScript needs to be converted back to JavaScript)
Run tests

Our Drone file looks like:

kind: pipeline
name: default

steps:

- name: build
  image: node:latest
  environment: # supplying environment variables for testing
    RELAY_KEY:
      from_secret: relay_key
    RELAY_SECRET:
      from_secret: relay_secret
    RELAY_BUCKET: ws-client-tests
  commands:
    - npm install
    - npm run build
    - make test

here, npm run build is actually just:

"scripts": {
    "build": "tsc"
  },

And in the Makefile make test:

test:
    ./node_modules/mocha/bin/mocha --reporter spec --compilers ts:ts-node/register src/*.test.ts

Publishing to npm registry

It's always good to automate publishing packages as well. This way you will get a good release process for almost zero effort. When you are happy with the package functionality, you just tag a Github release and Drone will build, test and publish your package to npm registry:

- name: publish
  image: node:latest  
  environment:
    NPM_TOKEN:
      from_secret: npm_token
  commands:
    - make drone-publish
  when:
    event: [ tag ]

environment variable NPM_TOKEN is a token that you can generate for your account.

make drone-publish command looks like:

drone-publish:
    echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc 
    npm publish

It is important to set that .npmrc file as the publishing won't work without it. Strange? yes.

Bonus: Notifications

This last step is repeated across all my Drone pipelines, it's a notification to a Slack channel:

- name: slack
  image: plugins/slack
  when:
    status: [ success, failure ]
  settings:
    webhook:
      from_secret: slack_url
    channel: general
    username: drone

For this to work, get your Slack's webhook URL and create a slack_url secret.

Wrapping up

It takes 30-90 minutes to set up everything initially and once you have a CI system running, subsequent repositories can be added in seconds. Even if you think that running npm run build and npm publish takes only 1 minute of your time every time you release, automating this process will greatly improve your developer experience and life in general :) Combining automated builds and releases with tests will ensure that there is only one path to getting your package published. I have seen many cases where a TypeScript package build step was missed and the previous version was released. Or, after a 'quick fix' and push to registry the package was broken because someone forgot to run the test. Or, just consider that in the next year you might do 200 releases that would end up in many hours saved by automation!

Setting up simple, self-hosted & fast CI/CD solution with Drone for homelab or office

Karolis — Mon, 11 Feb 2019 18:37:55 +0000

Continuous Integration and Continuous Delivery have been very trendy topics in the DevOps world for the last several years. As more and more organizations strive to improve their test and release processes, we have observed many new tools appear in this ecosystem. All major cloud providers are offering their automation systems (some are very unsuccessful) so it's getting quite hard to pick the right one that fits your use-case.

There are quite a lot of services (Travis, CircleCI, Google Cloud Builder) and self-hosted solutions to build, test and deploy your code, but actually, a few are free, open-source and self-hosted.

The most well-know CI/CD tools are Jenkins and GitLab CI (I have recently asked this on dev.to post). However, Jenkins has a huge memory footprint since it runs on Tomcat (Java) and the workflows are defined inside Jenkins itself, not in the repository that will be used to run those tests/build jobs.

As for GitLab CI, it's great but requires you to run your own GitLab (which is huge) or to be on gitlab.com (although now you can mirror your Github repos to Gitlab to start builds automatically, however, it adds complexity). You can run your own runner independently though, they seem to be quite reliable. One of the organizations that I am helping with backend development is using Gitlab CI to great success, but as I mentioned before, business pricing and server running costs could be significant.

Little disclaimer: I have tried using Drone several years ago and I wasn't happy with it. Not sure whether it was the actual product fault or just documentation but I couldn't achieve what I wanted. This time seems very different. The product is stable, documentation is good and some very good examples are lying inside their own .drone.yml file: https://github.com/drone/drone/blob/master/.drone.yml. Builds are fast and reliable.

Drone

I have looked into many alternatives and decided that at least for me, the best CI/CD solution right now is Drone. It's an open source, very lightweight application written in Go that is available as a SaaS or as a self-hosted server. Drone consists of 2 components:

Server is responsible for authentication, repository configuration, users, secrets and accepting webhooks.
Agent are receiving build jobs and actually run your workflows.

Both server and agent are extremely lightweight services and only uses around ~10-15MB RAM. It basically means that you will not even feel that it's running on your laptop, desktop or even a Raspberry Pi.

My setup

I am running Drone server on a Raspberry PI while the agent is running locally on my desktop which has lots of RAM and 16 CPU cores. This means that builds are a lot faster than any free public services could offer. What is more, since agents are connecting to the server over the Webhook Relay tunnels, I can deploy them on Kubernetes, spare laptops or other machines running anywhere:

Benefits of such configuration:

No need to configure NAT, Drone server barely uses any resources so it can run on cheap hardware (hence Raspberry Pi with a bunch of other services).
Agents are decoupled and can be running on your home/office machines. By connecting to them over the tunnels, agent configuration remains the same, no matter where they are deployed (external cloud provider or internal office).

Future plans include transitioning from Raspberry Pi for Drone server to an Intel NUC (or some other low power alternative) and running some additional Drone agents on the same server.

Preparing configuration

Initial configuration is very straightforward. First, we create a tunnel on Webhook Relay to get our public subdomain. Then, we use that subdomain to create a Github OAuth app.

Once the server and tunneling daemon are running, you can connect/disconnect Drone agents without any hassle.

Setting up the tunnel for Github and remote access

I like dogfooding, therefore we are using our own tunnels throughout our stack (CI/CD, monitoring dashboards, various other automations). In this case, Webhook Relay tunnels provide the two most important things:

Access to Drone server no matter where it is (on your laptop, desktop or on some Raspberry Pi), no need to configure a firewall or router to get remote access.
Webhooks from Github will trigger the workflows.

Go to your tunnels page and create a new tunnel. If you are on a free tier, you will get a generated subdomain, it's fine.

Here:

name of the tunnel can be anything, but it has to match whatever you will set to the webhookrelayd container in the docker-compose.yaml. It acts as a filter.
subdomain is your public subdomain under *.webrelay.io. If you are on a free plan, this will be auto-generated.
destination is the address of the drone-server container. Since docker-compose configures virtual networks, it has to match the service name from the docker-compose.yaml.
crypto type is available for all paid plans, basically HTTP/TLS. I would recommend subscribing to get encryption! Otherwise, ping me a message and I can set up a free trial.

Configuring Github OAuth application

In this guide, we will be setting up Drone with Github. You can follow official docs. In short:

Create a GitHub OAuth application. The Consumer Key and Consumer Secret are used to authorize access to Github resources. The Authorization callback URL must match the below format and path, and must use your exact server scheme and host. So if your Webhook Relay tunnel address is my-drone-subdomain.webrelay.io then the Authorization callback URL will be https://my-drone-subdomain.webrelay.io/login.

Deploying Drone server & agent

I have initially started using Drone with both server and agent running on my desktop since it was only for my personal use, it was fine. However, later I detached server and moved it into a Raspberry Pi.

The all-in-one configuration looks like:



version: '3'
services:
  drone-server:
    container_name: drone-server
    image: drone/drone:1.0.1   
    volumes:
      - db-data:/var/lib/drone/
    restart: always
    environment:
      # - DRONE_OPEN=false
      - DRONE_SERVER_HOST=my-drone-subdomain.webrelay.io # tunnel hostname       
      - DRONE_GITHUB_SERVER=https://github.com
      - DRONE_ADMIN=<your username>
      - DRONE_GITHUB_CLIENT_ID=<github-client-id>
      - DRONE_GITHUB_CLIENT_SECRET=<github-client-secret>
      - DRONE_SERVER_PROTO=http # tunnel adds https on top
      - DRONE_RPC_SECRET=<shared secret>
      - DRONE_USER_CREATE=username:<your github username>,admin:true
      - DRONE_AGENTS_ENABLED=true
  drone-agent:
    container_name: drone-agent
    image: drone/agent:1.0.1
    command: agent
    restart: always
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_RPC_SERVER=https://my-drone-subdomain.webrelay.io
      - DRONE_RPC_SECRET=<shared secret>
      - DRONE_RUNNER_CAPACITY=8
      - DRONE_RUNNER_NAME="local"

  webhookrelay:
    container_name: drone-webhookrelay
    image: webhookrelay/webhookrelayd:latest
    command:
      - --mode 
      - tunnel 
      - -t
      - drone
    restart: always
    depends_on:
      - drone-server
    environment: 
      - KEY=<whr-key>
      - SECRET=<whr-secret>

volumes:
  db-data:

If you need another agent on a different machine, agent-only configuration:



version: '3'

services:
  drone-agent:
    container_name: drone-agent
    image: drone/agent:1.0.0-rc.5
    command: agent
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_RPC_SERVER=https://my-drone-subdomain.webrelay.io
      - DRONE_RPC_SECRET=<shared secret>
      - DRONE_RUNNER_CAPACITY=8
      - DRONE_RUNNER_NAME="local"

Once you have the config:



docker-compose up

It will ask you to authenticate with your Github credentials.

Working with Drone pipelines

One of the best things about Drone is their pipeline configuration. It seems like after the Drone showed the way, industry followed and now most of the CI/CD services implement the pipelines through the Docker containers. It means that your builds and tests are decoupled from physical hosts and the cleanup happens by design. An example pipeline, that

builds
unit tests
runs end-to-end tests with a Postgres database
sends a webhook to Slack
builds and pushes an image to DockerHub

looks like this:



kind: pipeline
name: default

workspace:
  base: /go
  path: src/github.com/webhookrelay/app

steps:
- name: build
  image: golang:1.11.4
  commands:
    - cd cmd/app && go build

- name: unit-test
  image: golang:1.11.4
  commands:
    - go test -v `go list ./... `

- name: end-to-end
  image: golang:1.11.4
  environment:    
    POSTGRES_HOST: database
    POSTGRES_USER: pguser
    POSTGRES_PASSWORD: pgpass
    POSTGRES_DB: pgdb     
  commands:
    - make install-transponder
    - make e2e

- name: slack
  image: plugins/slack
  settings:
    webhook:
      from_secret: slack_url
    channel: general
    username: drone
    icon_url: https://i.pinimg.com/originals/51/29/a4/5129a48ddad9e8408d2757dd10eb836f.jpg

- name: docker  
  image: plugins/docker
  settings:
    repo: namespace/repo
    auto_tag: true
    username:
      from_secret: docker_username
    password:
      from_secret: docker_password

services:
- name: database
  image: postgres
  environment:
    POSTGRES_USER: pguser
    POSTGRES_PASSWORD: pgpass
    POSTGRES_DB: pgdb

By always using containers to run your pipelines, you will make your builds and tests truly portable. Whenever you push to Github, a webhook will hit Drone and it will start a build:

Tips and tricks

I would also like to share some very useful commands. To install drone CLI, follow official docs.

Executing pipelines directly

You can also run pipelines directly with the Drone CLI:



drone exec --secret-file drone_secrets.yaml .drone.yml

A template for Drone secrets:



slack_url: https://hooks.slack.com/services/xxxxxxxxxxxx

Adding secrets through CLI

You can either add secrets through the web UI or use drone directly from your terminal:



drone secret add -repository username/repository-name --name foo --data bar --allow-pull-request

Wrapping up

I have used extensively Jenkins, Gitlab runners, CircleCI and Google Cloud Builder. I guess I still like a lot Cloud Builder for GKE related things as it builds images fast and authenticates to GCR registry without any additional work. However, you do get stuck with their pipeline configuration files and can't really move to any other vendor. As for Jenkins, it's just hard to imagine running it efficiently and smaller servers. Drone, at least for me, is an ideal example of a modern service that focuses on efficiency and doing one thing right.

So, just to recap, main pros of Drone:

Very easy to setup
Out of the box it uses SQLite as a database but you can use Postgres or Mysql.
Lightweight agents that can run anywhere
Available either as a self-hosted solution or as a SaaS
Pipelines defined as code
Run pipelines locally with just drone CLI
Integration with Github, Gitlab, Bitbucket

Originally published on Webhook Relay blog

What's your favourite CI/CD tool and why?

Karolis — Fri, 01 Feb 2019 16:06:01 +0000

Throughout the years I have seen many companies using different tools to achieve same/similar things in CI/CD area. There's no denying that Jenkins is probably the most widely used solution due to the huge amount of available integrations and plugins. There are, however, some downsides such as resource consumption that can become very expensive at scale. I always remember this great article from Monzo Building a Modern Bank Backend that was mostly about migrating from Mesos to Kubernetes, but it does have some important points such as:

Previously, we ran several very beefy Jenkins hosts dedicated to the task, which were inefficient and expensive. Now, build jobs run under Kubernetes, using spare capacity in our existing infrastructure, which is basically free. - Oliver Beattie

Similar issues I have noticed in other companies, such as disks getting full due to growing workspaces or just agents getting quite slow due to insufficient host resources. Most of these issues could be labelled as user configuration problem, but then it seems that they don't exist in other systems.

My favourite tools to automate tasks:

Google Cloud Build - powerful and cheap alternative to CircleCI/Travis. If you are running on GCP, I would definitely recommend looking into it as for most users it will be running for free :0
Gitlab CI/CD + runners. A great model, probably designed based on drone.io server/agent model. Nevertheless, works great most of the time :)
Drone - very lightweight solution that you can run even on your desktop or on any server. Pipelines are defined in the same repo as your code and all builds happen in containers, so no issues with dirty workspaces or tying them to individual hosts.

It would be interesting to know what tools other people are using and why, please write your answer in the comments! :)

Controlling gadgets with Google Home, IFTTT and Node-RED

Karolis — Tue, 29 Jan 2019 18:29:32 +0000

For a long time I have been using my Google Home mini just for setting up timers during cooking. Finally, I have found some more ways to use it! :)

In this article, I will show you how easy and quick it is to add voice controlled commands to your home/office or any other environment. Once you set up your first flow, any other features on top of that won't take even 1 minute to add.

We will be using:

Node-RED our main tool to wire everything together.
Google Home - I am using Google Home Mini to launch tasks.
IFTTT will be transforming commands coming from Google Home into webhooks.
Webhook Relay is going to act as a broker to deliver webhooks to our Raspberry PI running Node-RED without exposing it to the internet.

Webhook Relay node removes a lot of work that is required to securely expose your Node-RED to the internet. It is especially useful when you can't receive webhooks in your local network due to:

ISP blocks incoming connections
Double NAT due to using 4G
No static IP
Lack of knowledge how to set up HTTPS and reverse proxy

In short, it provides an encrypted, one-way transport to your Node-RED through a single node. And we do have a free tier.

If you Node-RED is exposed to the internet, you can skip Webhook Relay part and just use HTTP node to accept the webhooks from IFTTT :)

1. Preparing to receive webhooks from IFTTT

Webhook Relay will be acting as a message broker between Google Home with IFTTT and Node-RED. Naturally, let's configure it first. Got to buckets page and create a new bucket called 'gactions':

In the bucket details page, you should see 'Default public endpoint' URL that starts with https://my.webhookrelay.com/v1/webhooks/.... Keep this tab open, you will need to copy that URL into IFTTT.

2. Set up Google Home with IFTTT

Head to IFTTT, then to your applets and click on "New Applet". Search for "Google Assistant":

IF THIS (Google Assistant)

When choosing a trigger pick 'Say a simple phrase' for our scenario. You can try other ones later for different automation. Now, in 'What do you want to say?' section type 'turn the TV on' or something similar. Whatever you want basically. Populate other fields and pick your response phrase. Click "Create trigger".

THEN THAT (Webhook)

For the action service, choose webhook:

In the URL part, take the 'Default public endpoint' from step 1 (that starts with https://my.webhookrelay.com/v1/webhooks/...). Choose method to "POST", set Content Type to 'application/json' and set body to:

{
  "action": "tv_on"
}

Once done, click "Create Action". You can now repeat the same process for more commands such as turning the TV off, mute, lower the sound and so on. I have configured three applets in total to send webhooks to the same endpoint:

To turn it off:

{
  "action": "tv_off"
}

To mute it:

{
  "action": "tv_mute"
}

Adding new commands is very fast, takes less than a minute once you have finished with the first one.

3. Configure Node-RED

Our Node-RED flow consists of three steps:

Receive webhooks through node-red-contrib-webhookrelay node.
Extract body from the webhook and parse it using simple function and JSON node.
Launch actions based on switch node.

To control TV I am using node-red-contrib-tv-bravia node. Pretty much any internet connected device will also listen to node-red-node-wol (Wake on LAN).

The flow can be found on GitHub gist here. You can either import it or add nodes one by one. For the learning purposes, I would advise to add them manually so you can better understand how it works.

Let's get started. First, get authentication tokens and set them to the node-red-contrib-webhookrelay node. In the "Buckets" field, add our "gactions" bucket which we created previously:

Extracting & parsing body

Now, we need to extract body and parse it. Create a function node and in the function body add this:

return {
    payload: msg.payload.body
};

This will extract the webhook body from the whole webhook message (it includes input, bucket metadata, as well as request method and headers). Then, add a json node and configure it with:

Action: Convert between JSON String & Object
Property: msg.payload

Launching actions based on payload

Finally, time to add the switch and main control nodes. Remember the payloads that we configured in IFTTT? Time to read that action value:

For the control I am using node-red-contrib-tv-bravia node. Follow their instructions to set up the TV. In short - you need to know the IP address of your TV and MAC (for the Wake on LAN node). You can either find it from your router or go to your TV network settings and pick it up from there. Each switch wire goes to a different action, this makes it simple and robust.

Future work

Once you have the flow ready up to the switch node, feel free to add more gadgets behind it. As you can see, IFTTT makes it super easy to issue commands and act based on those payloads in your flow. If you have your Node-RED exposed to the internet, you can even skip Webhook Relay node and receive webhooks directly through an http node.

I have originally published this as a blog post here: https://webhookrelay.com/blog/2019/01/29/google-home-ifttt-node-red/