Forem: Pooria A

Automatic deployment from Github to your server with no third party app

Pooria A — Sun, 24 Nov 2019 09:34:41 +0000

When it comes to setting up a deployment pipeline I believe the number of solutions in the wild are countless. If we are on AWS we can use Code Deploy, Heroku, Zeit Now and Netlify provide their own solutions and of course one of the popular ones is to delegate the task to your CI/CD server (Travis, Circle CI etc) to handle it. If you are using Docker containers the best orchestrator tools to use are AWS ECS, Swarm and Kubernetes. Most probably if you are working with a bigger team you have Dev-Ops team mates to handle it and you might take your infrastructure as granted! 😐 However, if like me you joined a startup and the deployment process was manual (SSH to server, git pull, etc) and you wrote some bash scripts to do it for them you might want to embrace your inner nerd 🤓 and level up your deployment game. In this case, running your bash scripts automatically upon merging a Pull Request on Github would make everyone happy and that's not rocket science so let's do it!

Goal

To automatically deploy your code after merging pull requests to branches dev and master.

Things we use

Node JS
Bash Scripts
Github Webhooks
SSH command line

Getting Started

Let's say we have two versions of our website that we need to automatically deploy. One is called stage and represents the latest merged commits. These changes are often buggy and not reliable so we want only the internal team have access to it. "stage" represents our dev branch in Github. The second version of the website is called "prod" and will rerpresent the master branch in Github. This branch is (hopefully) stable and has passed the QA team and is considered safe for end-users. This version is the one that everyone outside the company know as our website URL.

Step 1: Clone your Git repositories

If you already don't have your github repositories cloned in the server you have to do it.

Make two directories called : prod and stage.

mkdir live stage
git clone git@github.com:p0o/your_repo.git
cp -a your_repo/. ./prod/your_repo
cp -a your_repo/. ./stage/your_repo
rm -rf ./your_repo

Make sure to add the extra . after your_repo, this a special cp syntax that allows copying hidden files and folders inside your folder as well (we need it to copy .git folder as well).

Wild Assumption: I assume you are familiar with the basics of managing a server and your can run your websites in the URL you want with a proper SSL certificate. I use Nginx for this purpose but I'm not gonna explain these steps in the post. You might search if you are not sure.

Step 2: Make bash scripts

We need to have two bash scripts to handle the deployment for each of these cases. If you need to build your files and Let's create a directory in our server's home directory and start from there. I call this directory scripts:

cd ~/
mkdir scripts
cd scripts

Okay let's proceed with making the bash files:

touch ./deploy_stage
touch ./deploy_prod

Give them execution permission:

chmod +x ./deploy_stage
chmod +x ./deploy_prod

(Thanks to darksmile92 for pointing this out)

I will put the sample code for one of them, the other one is just a different folder and might have different environment variables according to your project dependencies.

#!/bin/bash
echo "Deploying stage your_repo"

cd ~/stage/your_repo \
&& git checkout dev \
&& git pull \
&& npm i \
&& npm run build \
&& (pm2 stop your_repo_stage || true) \
&& echo 'Installing:  done.' \
&& (pm2 delete your_repo_stage || true) \
&& NODE_ENV=development pm2 --name your_repo_stage start npm -- start \
&& echo "your_repo deployed successfully"

This bash script will basically fetch the latest code from github, install dependencies, build the script (if required) and run it using PM2. If you are not familiar, PM2 is a very useful process management tool and you can easily install it using NPM.

Also it's good to mention that I chained my whole process with logical AND (&&) because I wan to exit the execution in case one of the processes failed.

Step 3: Write code to handle the webhook events

In order to get notified anytime something happens in Github we have to subscribe to their Webhook API which essentially means giving some URLs to github so they would send some information to it. These URLs have to be public and they can run scripts that would lead to deployment of your code so having them accessible to anyone except Github servers would have serious security implications (e.g Denial Of Service attack).

Github is using a SH1 HMAC signature to verify the JSON object it's sending you. We will have this signature hash in the X-Hub-Signature value of header. Since taking care of all this is a bit complicated, we can use github-webhook-handler package which is created exactly for the same purpose.

We also need to run our bash script files from node. We can do it using native functions but I prefer to use shellJs for the sake of simplicity.

Okay enough ranting, here is the code you need:

const http = require('http');
const createHandler = require('github-webhook-handler');
const shell = require('shelljs');

// We avoid to hardcode the secret in the code, you should provide it with an ENV variable before running this script
const { MY_SECRET } = process.env;
// You might use the same script for multiple repositories, this is only one of them
const REPO_NAME = 'my_repo';
// port is default on 6767
const PORT = process.env.PORT || 6767;

var handler = createHandler({ path: '/', secret: MY_SECRET })

http.createServer(function (req, res) {
  handler(req, res, function (err) {
    res.statusCode = 404
    res.end('no such location')
  })
}).listen(PORT);

handler.on('error', function (err) {
  console.error('Error:', err.message)
})

handler.on('pull_request', function (event) {
  const repository = event.payload.repository.name;
  const action = event.payload.action;

  console.log('Received a Pull Request for %s to %s', repository, action);
  // the action of closed on pull_request event means either it is merged or declined
  if (repository === REPO_NAME && action === 'closed') {
    // we should deploy now
    shell.cd('..');
    shell.exec('~/scripts/deploy_stage');
  }
});

Save it in a folder somewhere in your server and install the dependencies:

npm init
npm i github-webhook-handler shelljs --save

And then just run it with environment variable forever using PM2:

MY_SECRET=MyGithubWebhookSecret pm2 --name github-deployer start node -- ./index.js

That's all!

Step 4: Configure github webhook

Now we just need to go to Github and introduce our webhook to github. But pay attention that in the previous step we ran the webhook on the port 6767 with no HTTPS. So you need to setup nginx and give it a proper domain with HTTPS. You can just put it on a path in your main domain but explaining that process is not in the scope of this article. Furtunately there are multiple articles in the web for you to look for.

Go to the Setting tab of your repository and click on the Webhooks. In the right side of the page, click on the "Add Webhook" button.

Enter the URL you introduced in your Nginx for the Node JS app we ran. Let's say it is https://yourdomain.com/webhook

Choose application/json for the content-type and enter the secret we used to run our service with. In my example it was "MyGithubWebhookSecret" up there.

In the section "Which events would you like to trigger this webhook?" click on "Let me select individual events" and find Pull Requests and check it:

Make sure everything else is unchecked and click "Add Webhook" to save it. We are all set now 🦸

Step 5: Test and verify

Use PM2 to monitor the logs for the node js app we made just now. Enter:

pm2 log github_deployer

Now you can see if any changes happens! Go to your repository and change something in a new branch. Submit a pull request and merge it. You should see your bash script in the log would perform a deployment and after that your changes should be reflected to the website. If you had any error, you can see it here in the log and well... do something about it 😂

Conclusion

Even though I think the proposed solution in this article is fairly straightforward, it is not the best solution out there for this particular problem. Even my personal blog is using Zeit Now Github Integration to get deployed! However other solutions rely on third party apps and sometimes not accessible to certain teams according to their resources. In my case, the deployment scripts were already there, repositories were not using docker and I had very limited time to spend on this problem. Advance with it if you also happen to be in the same boat!

This article is originally published on my blog with the title Automatic deployment From Github to your server feel free to check it out for more posts! 👀
You can also find me on twitter => @p0oker

Why is my browser sending an OPTIONS HTTP request instead of POST?

Pooria A — Sat, 09 Nov 2019 20:49:08 +0000

This issue caught my attention a few days ago that my colleagues were facing difficulty in using a new API developed in-house using Flask. The problem was that no matter what, the front-end developer couldn't make a call with correct content-type. Even though that Axios uses JSON as the default content type, the call was always going with a text/html format and everyone were getting frustrated 🤨.

In the other hand, the back-end developer was showing her the result from Postman (an application for developers to send HTTP calls) and everything was working fine there!

I first tried to test if the end point is working fine or not. Me being a CLI guy, used my favorite HTTP client HTTPie to do the basic call. It's something like CURL but looks better for the eyes!

Nothing is wrong here if we test the API standalone with a HTTP client, but the axios request below would result in nothing.



axios.post('https://ENDPOITN_URL', {
  field1: 'something',
  field2: 'something'
});

My colleague moved forward and tried to enforce a application/json content-type to axios. It's a bit weird but maybe somewhere else in the code the default for the axios is changed?



const customHeaders = {
  'content-type': 'application/json',
};

axios.post('https://ENDPOITN_URL', {
  field1: 'something',
  field2: 'something'
}, customHeaders);

Still no practical results. I asked for a screenshot and this is how it was looking like in the browser:

Okay let's take a closer look, there are two things to consider here:

As you can see, the POST method is never sent and only a method called OPTIONS is sent to the endpoint. The response headers from this call has a content-type of 'text/html' which is the reason for all this evil here. So... what's going on?

What is a preflight request?

A preflight request, is a mechanism in CORS by the browser to check if the resource destination is willing to accept the real request or not. Afterall, why would a request be sent when the target host is not willing to receive it anyway?

This mechanism works by sending an OPTIONS HTTP method with Access-Control-Request-Method and Access-Control-Request-Headers in the header to notify the server about the type of request it wants to send. The response it retrieves determine if the actual request is allowed to be sent or not. This is a sample of a preflight request:



OPTIONS /resources/post-here/ HTTP/1.1 
Host: bar.other 
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 
Accept-Language: en-us,en;q=0.5 
Accept-Encoding: gzip,deflate 
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 
Connection: keep-alive 
Origin: http://foo.example 
Access-Control-Request-Method: POST 
Access-Control-Request-Headers: X-PINGOTHER, Content-Type

I highlighted the last three lines, because they are important fields in this call. Most developers are familiar with the Origin method because if it's not allowed from the backend API, you are not able to make AJAX calls to fetch the data. The other two parameters are overlooked 🧐 because most frameworks and libraries would take care of them anyway. For example any backend developer using express can simply add a middleware called CORS and make sure all the calls in his express app are providing those parameters for the OPTIONS method to the browsers.



var cors = require('cors')

app.use(cors()) // cool now everything is handled!

Whenever the server received that request, it should responds with Access-Control-Allow-Methods and some other meta data to identify if the original request is acceptable or not! A sample response would look something like this (but it varies):



HTTP/1.1 204 No Content
Date: Mon, 01 Dec 2008 01:15:39 GMT 
Server: Apache/2.0.61 (Unix) 
Access-Control-Allow-Origin: http://foo.example 
Access-Control-Allow-Methods: POST, GET, OPTIONS 
Access-Control-Allow-Headers: X-PINGOTHER, Content-Type 
Access-Control-Max-Age: 86400 
Vary: Accept-Encoding, Origin
Keep-Alive: timeout=2, max=100 
Connection: Keep-Alive

It's important to mention that, not all requests would preflight. As far as I know, only requests that are meant to be sent to a different origin and are not a form content-type are preflighted (excluding GET and HEADER methods).

So what was the problem?

I tried to send a normal OPTIONS request to the endpoint to check the rules. I used the --headers in HTTPie to only receive the header of the request.

Turned out that the value of the content-type here is text/html and that's why browser wouldn't push through with the actual POST method, however with a normal client it's acceptable.

But we originally mentioned that most of the frameworks would handle this out of the box, so why here Flask is giving us wrong content-type? It's sort of a tricky situation... I figured if I send a normal POST request to the API without the required body parameters, the endpoint will throw an error which is not properly handled!
Well it's an obvious bug on the backend but probably they didn't care because it was an internal API and it was working fine with correct parameters. However, the OPTIONS method contains no body parameters within and since the original API without params is returning a text/html content (the web server error page) the OPTIONS method was also returning the same, mistakenly thinking that this API does not accept a JSON request 🤦

I really enjoyed learning about this mechanism better through this article. If you like to learn more about this HTTP method and the process of preflight feel free to scavenge these links further:

Learn more

I originally published this article in my blog!

Using GraphQL queries to separate pages and blog posts in Gatsby

Pooria A — Wed, 05 Jun 2019 16:58:39 +0000

When I started my blog with Gatsby one of my main problems was to filter out my "about me" or any other pages from the list of blog posts. If you don't use Markdown files for your static pages, this task is as simple as creating separate React components for each page. However, if like me, you wanted to use Markdown for your static pages you would have the problem of seeing your pages in the list of blog posts!

In the beginning, I used my JavaScript skill to filter out the results but I wasn't happy with that. Recently I learned a better approach using GraphQL and that's what I'm describing in this post.

Using frontmatter

First, we need a mechanism to tag markdown files that we are using for our pages. Frontmatter (the fields specified on top of your markdown files) is a good place to tag some posts as pages. This is how I do it:

---
title: "My page title"
date: 05/06/2019
page: true
---

As you can see, in this markdown file I tagged the page property (could be anything else) as true. Now in my index.js which I am listing all my blog posts, I have the possibility to filter out the markdown files that their page property is false.

Excluding pages from blog posts

At this moment that I have my page field ready in the frontmatter, removing the pages from blog posts results is as simple as one extra line to your GraphQL query:

// index.js (or wherever you are showing the list of your blog posts)
export const pageQuery = graphql`
  query {
    site {
      siteMetadata {
        title
      }
    }
    allMarkdownRemark(
      sort: { fields: [frontmatter___date], order: DESC }
      filter: { frontmatter: { page: {ne: true}}}
    ) {
      edges {
        node {
          excerpt
          fields {
            slug
          }
          frontmatter {
            date(formatString: "MMMM DD, YYYY")
            title
          }
        }
      }
    }
  }
`

Pay attention that I'm using the filter argument on allMarkdownRemark query. ne is a reserved keyword that would stand for "Not Equal". If you are not sure about the options available for filtering, you can always use the graqhiQL (accessible at localhost:8000/__graphql) to check them out.

Querying the correct page

Now we need to make a new component in our /pages directory and query the page correctly. However, for every page, we probably want to make a new component and there is no way to figure out which page, belongs to which component from the frontmatter.

In order to simply serve a specific page in a component according to the name of the markdown file, we can simply use the slug field which is available in our GraphQL data:

export const pageQuery = graphql`
query {
  site {
    siteMetadata {
      title
      author
    }
  }
  markdownRemark(
    fields: { slug: { eq: "/about-me/" }}
  ) {
    excerpt
    html
    frontmatter {
      title
      date
    }
  }
}
`

With this query, you can access to all the data of your post in this.props.data.markdownRemark on the very same component you are executing this query with!

Conclusion

This solution is appropriate for the use case of having limited pages with different designs. If you need to have a dynamic number of pages, you don't want to create new components for each of them and you need to make some changes in your createPage function of you gatsby-node.js file to render Page components differently.

Deploying a static Gatsby app using Now.sh

Pooria A — Mon, 03 Jun 2019 05:26:04 +0000

If you are making your new static website with Gatsby, you are probably looking for an easy way to build and deploy it. There are two generic ways for you to do so, using Github integration with Now.sh platform or manually using CLI. In this article, I explain both of these methods and reflect my experience of using Now.sh for the deployment of the very same blog you are reading.

My assumption for this article is that you are familiar with Gatsby and recently started using Now.sh. Hopefully, you registered a domain with this service as well to create an alias for your deployment but it's not necessary.

Using CLI

Now.sh has a very handy and powerful command line tool. No matter how much you like the idea of integration with Github or using the Desktop App, you would always need to go back to cli for advanced functions. I also hate to say it but it's much more reliable than Github integration since sometimes those services just stop working for technical reasons.

In order to deploy you need to first download the command line tool from npm:

npm install -g now

Now you need to use this command in the terminal to log in to your now.sh account:

now login youremail@gmail.com

Enter the password and you're all set. If you enter now in every folder, it will upload all the files in that folder into a new address which is assignable to your domain. However, if you do it with a Gatsby blog you end up uploading all the files in your source directory including package.json but not the static files. If you want to run a build in the cloud, you need to specify a configuration file for now.sh to understand how to build your files.

Add a file with the name now.json in the root of your Gatsby directory (next to package.json). This is a sample of how this file should be:

// now.json
{
  "version": 2, // using version 2.0 of now.sh
  "name": "Your Gatsby App Name",
  "alias": "yourdomain.com", // remove it if you don't own a domain with now.sh
  "builds": [
    {
      "src": "package.json",
      "use": "@now/static-build",
      "config": {
        "distDir": "public" // default folder for Gatsby builds
      }
    }
  ]
}

One more step and we are good. Go to your package.json and add a new script called: now-build which should basically point to gatsby build like this:

// package.json
"scripts": {
    "dev": "gatsby develop",
    "now-build": "gatsby build",
    // other scripts ....
}

We are all set. Now you can simply run the command below in your Gatsby root folder:

now --target production

This command will prepare a new build and deploy your Gatsby app into your domain with Now.sh. Life is beautiful!

Using Github

Well yes, it's a great idea to use Github editor to write a new blog post or just edit the previous posts and Now.sh can trigger a new deployment immediately after every new commit. The rule of the thumb here is that if you already did set an Alias to your domain, every commit to master branch of your Github will trigger a deployment to your website URL and other branches will only deploy a version to a new address. This address is accessible inside your Github Environment tab or in your Now.sh dashboard.

In order to do it using Github you need to first connect your Github account to Now.sh. Afterward, all we need is to add a now.json file similar to the previous step:

// now.json
{
  "version": 2, // using version 2.0 of now.sh
  "name": "Your Gatsby App Name",
  "alias": "yourdomain.com", // remove it if you don't own a domain with now.sh
  "builds": [
    {
      "src": "package.json",
      "use": "@now/static-build",
      "config": {
        "distDir": "public" // default folder for Gatsby builds
      }
    }
  ]
}

Also make sure to add now-build script to your package.json file:

// package.json
"scripts": {
    "dev": "gatsby develop",
    "now-build": "gatsby build",
    // other scripts ....
}

from now on, every commit to the master branch of your Github will trigger a deployment to your website URL. Great job, you're not a lazy developer!

This post is originally written for my personal blog fsociety 🙃