Forem: metacollective

New Survey

metacollective — Fri, 25 Mar 2022 09:25:41 +0000

Hello everyone,

I and my friend are running a small survey, and would be great to get some feedback.

Can you please have a look at this when you have 2 minutes? I promise it won't take more than that.

https://docs.google.com/forms/d/e/1FAIpQLSe9ZOnLGNfN1byEMwwt-5m-8o_VBEpGD-sf2IkkxwC8ZhDSrg/viewform
Have a nice Friday you all :)

Terminalizer: Record your terminal and generate animated gif images or share a web player

metacollective — Thu, 24 Feb 2022 09:57:05 +0000

I have just stumbled upon it and seems very easy to use - https://terminalizer.com/
Just did a quick test on my FISH shell and it works like a charm.
Has anyone tried it and do you have any other alternative to suggest?

Minimum Viable Product vs. Minimum Lovable Product

metacollective — Fri, 18 Feb 2022 16:34:11 +0000

MVP vs MLP

Recently I have started working with a client who uses the MLP principle for their product management.
I have always worked on the MVP style and never thought of challenging it. But now that I have been introduced to this way of thinking (I am not calling it new because MLP could be around for some time and I only now I have heard of it 🙄)

MVP: The idea is to come up with a minimum set of features/functionalities in a product with which you can take it into the market and get instant feedback aka to fail-fast and build upon it.

MLP: Here the idea is to focus on what your end-user will love. So, you set out to build a product with the least possible number of features, yet lovable by your users/customers. Hence, it's not essentially the fail-fast approach, but the opposite of it where you get them to like your product as soon as it hits the market.

I am still going through the adoption & understanding of MLP, but my take is that it is a lot better than MVP in most cases (I say most because there are some cases where MVP works better, please read on…)

MLP focuses on customers first and to me, that is a winner from the get-go. How many times have we used products which are half-baked and leave us frustrated?
Because the focus is customer, focus on customer research is done beforehand. Unlike in MVP, we expect customers to tell us on product launch what they don't like or what they expect from it.

I come from a software background, and I think having MLP in place forces us to think about the architecture well in advance. We know our customer needs and challenges are and we can plan our architecture accordingly.

However, I do see why MLP is not for everyone. First of all, it's expensive to do in-depth market research. The other issue is what if you are building a brand new product with a very little know consumer base? In this case, trialing an MVP based on some business analysis makes sense.

This is a very subject post based on my opinion, but I would love to hear from you all out there.
:

How to use AWS EventBridge to schedule tasks

metacollective — Wed, 16 Feb 2022 16:17:07 +0000

In my last blog, I wrote about how you can create your own Twitter bot using the serverless framework and deploy it on AWS. And, now that I have that function already deployed on AWS, I wish to have an automated process of running this lambda function periodically (like a scheduled job or corn tasks)

One way of doing it on AWS is by using their EventBridge service. This is how AWS describes EventBridge

Amazon EventBridge is a serverless event bus that makes it easier to build event-driven applications at scale using events generated from your applications, integrated Software-as-a-Service (SaaS) applications, and AWS services.

To get started, log in to your AWS console and go to the EventBridge page.

Click on the "Create rule" button and that should take you to a new rule form. Fill in the name and description field. Then select the "Schedule" option from the Define pattern filed like this -

There are 2 ways of scheduling your event. One is at a fixed rate of every x minute/hours/days where x is a numeric value or you can also set them up as a cron expression. You can read all about their pattern rules - https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html#CronExpressions

I am going to select cron expression because I want to send 2 tweets every day. One at 9 am in the morning and the other at 5 pm in the evening.

If you are new to corn expression, then AWS has given a very nice and easy table to understand each entry

Since I want to trigger my event twice a day at a particular hour of the day, my corn job will look like this - (0 9,17 * * ? *)
Note the comma between 9 & 17

AWS will also give you the next 10 triggers so you can test your corn setting before saving them. Yes, AWS can be helpful at times 😉

Next is to select your target for it to be called by event bridge. Choose Target as Lambda function and then search for your lambda from the dropdown of functions

Click create at the bottom of the page and your event bridge is ready to call lambda function.

Please note that earlier (and even now) you could do this from Cloudwatch but AWS has moved it to the EventBridge service. At the moment, this is what they show you when you go to the Cloudwatch's event

After reading all this, if you are intrigued/excited to receive developer jokes like this then you can follow me on Twitter

// Detect dark theme var iframe = document.getElementById('tweet-1493248927361032198-411'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1493248927361032198&theme=dark" }

Twitter BOT to send joke and image (lambda + typescript)

metacollective — Tue, 15 Feb 2022 14:28:21 +0000

In this blog post, I am going to show you how you can build your own twitter bot which will periodically send out tweet (joke) with images

How will it work?

We will get a random joke from https://jokeapi.dev/ API. It's free with some limitation on hit rate etc.
We will get a random but relevant GIPHY image using https://developers.giphy.com/ API
We will create and send a tweet using twitter api and twitter-api-client node package

End Result will look like this
// Detect dark theme var iframe = document.getElementById('tweet-1493178651692646400-138'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1493178651692646400&theme=dark" }

Prerequisite

You should have a twitter account to register with twitter's developer portal
AWS access key & secret key to deploy on AWS

Step 1: Register with twitter's developer portal and get relevant keys

Getting keys from twitter's developer account is pretty easy and straight forward process. You can get a step by step guide from this video -

Step 2: Register with giphy's developer account and get their access key

Again, this is very easy and self explanatory (https://developers.giphy.com/), but in case you need some detailed information on this API and how to get its access keys, you can watch this video -

Step 3: Now you have your twitter & giphy keys, so let's get started with writing the bot code

We will be using the serverless framework to write this lambda function.

Install serverless globally

npm install -g serverless

Create an typescript project from the serverless template library

sls create --template aws-nodejs-typescript

Install required node packages for this project

npm install --save axios twitter-api-client
npm install --save-dev serverless-offline

axios: Promise based HTTP client for the browser and node.js
twitter-api-client Node.js client for Twitter API
serverless-offline This Serverless plugin emulates AWS λ and API Gateway on your local machine to speed up your development cycles.

Define your function in serverless.ts like this

import type { AWS } from '@serverless/typescript';

const serverlessConfiguration: AWS = {
  service: 'twitter-bot',
  frameworkVersion: '2',
  custom: {
    webpack: {
      webpackConfig: './webpack.config.js',
      includeModules: true
    }
  },
  // Add the serverless-webpack plugin
  plugins: ['serverless-webpack', 'serverless-offline'],
  provider: {
    name: 'aws',
    runtime: 'nodejs14.x',
    apiGateway: {
      minimumCompressionSize: 1024,
    },
    environment: {
      AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
    },
  },
  functions: {
    twitterJokesBot: {
      handler: 'handler.twitterJokesBot',
      events: [
        {
          http: {
            method: 'get',
            path: 'twitterJokesBot',
          }
        }
      ]
    }
  }
}

module.exports = serverlessConfiguration;

Here we have defined a function named twitterJokesBot and it will be of type GET

Make sure that you add your serverless-offline plugin to the list of plugins in serverless.ts

Define twitter-api-client

import { TwitterClient } from 'twitter-api-client';

export const twitterClient = new TwitterClient({
    apiKey: process.env.TWITTER_API_KEY,
    apiSecret: process.env.TWITTER_API_SECRET,
    accessToken: process.env.TWITTER_ACCESS_TOKEN,
    accessTokenSecret: process.env.TWITTER_ACCESS_SECRET,
});

Make sure that you have added these keys and tokens into your environment

Define joke functions

import Axios, { AxiosResponse } from "axios";

/**
 * Get a joke from jokeapi.dev. If the length of tweet string is greater than 280 character then recurse till you find one.
 * @returns string
 */
export const getJokeTweet = async () => {
  //List of hashtags for the tweet
  const hashTags =
    "@metacollective9 #jokes #javascript #programming #dev #linux #java #programming #python #reactjs #DataScience #infosec #gamedev #BigData #serverless";

  const footerText = "From https://jokeapi.dev";

  //Get a random joke from jokeapi
  let joke: AxiosResponse = await Axios.get(
    "https://v2.jokeapi.dev/joke/Programming?blacklistFlags=nsfw,religious,political,racist,sexist,explicit&type=single"
  );

  // \n will add line breaks to the tweet
  const tweet = `${joke.data.joke} \n\n ${hashTags} \n\n ${footerText}`;

  console.log(tweet.length);

  if (tweet.length <= 280) {
    return tweet;
  }

   return await getJokeTweet();

};

/**
 * Get a gif image from giphy.com
 * @returns image buffer
 */
export const getJokeImage = async () => {
  //get a giphy to go with this joke
  let giphy: AxiosResponse = await Axios.get(
    `https://api.giphy.com/v1/gifs/random?api_key=${process.env.GIPHY_API_KEY}&tag=bad+joke&rating=g`
  );

  //Get that giphy image as a buffer.
  const image: AxiosResponse = await Axios.get(giphy?.data.data.images.downsized_medium.url, {
    responseType: "arraybuffer",
  });

  return image;
};

There are two parts of this

getJokeTweet is to get a random joke from jokeapi.dev, then adds your hashtags and other texts that you want to attach with this tweet. It also checks that it is not more than 280 characters (tweet character limit as of Feb 2022), if it is then recurse and gets another joke Lookout for blacklistFlags in the API call. This makes sure that you get a nice & clean joke
getJokeImageis to get a random giphy image using the giphy api keys we got earlier. You have to get it as an arraybuffer so that we can upload it to twitter as a base64 image

Put them all together in your handler.ts

import { APIGatewayProxyHandler } from "aws-lambda";
import "source-map-support/register";
import { twitterClient } from "./src/util/twitterApiClient";
import { getJokeImage, getJokeTweet } from "./src/util/joke";
import { AxiosResponse } from "axios";
import { MediaUpload } from "twitter-api-client";
export const twitterJokesBot: APIGatewayProxyHandler = async (_event, _context) => {
  let tweet: string = "";

  try {
    //get Joke
    tweet = await getJokeTweet();

    const image: AxiosResponse = await getJokeImage();

    //Upload media to twitter
    const media: MediaUpload = await twitterClient.media.mediaUpload({
      media: Buffer.from(image.data, "binary").toString("base64"),
    });

    //Send a tweet with joke and media
    await twitterClient.tweets.statusesUpdate({
      status: tweet,
      media_ids: media.media_id_string, 
    });
  } catch (error) {
    console.log(error);
  }

  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        joke: tweet,
        length: tweet.length
      },
      null,
      2
    ),
  };
};

This is also a 2 step process

First is to upload a media asset to your twitter account
Then use the media_id_string from the response of uploaded media and send the tweet with it

That's it, now you are all set to test it -

Run sls offline --stage local in your terminal and you should have your endpoint ready to test with like this

🎉🎉🎉 Your bot is ready to launch.

If you wish to deploy it to your AWS account, then set your AWS Access Key & AWS secret Key in the environment and run sls deploy --stage dev

You can get the entire example from this repository https://github.com/appletreeat56/twitter-bot

Lambda to scrape data using typescript & serverless

metacollective — Sat, 12 Feb 2022 19:13:41 +0000

In this blog post we are going to do the following -

Write a lambda function in nodeJS/typescript to extract the following set of data from a website
- Title of the page
- Any image on the page
Store that extracted data on AWS's S3

We will use the following node packages for this project -

serverless (This must be installed globally): This will help us write & deploy Lambda function
cheerio: This will help us parse the content of webpage into a jQuery object
Axios: Promise based HTTP client for the browser and node.js
exceljs: To read, manipulate and write spreadsheet
aws-sdk
serverless-offline: To run lambda functions locally

Step 1: Install serverless globally

npm install -g serverless

Step 2: Create a new typescript based project from serverless template library like this

sls create --template aws-nodejs-typescript

Step3: Install the required node packages for this lambda project

npm install axios exceljs cheerio aws-sdk

Step 4: Add serverless-offline to plugins list in serverless.ts

plugins: ['serverless-webpack', 'serverless-offline']

Step 5: Add S3 bucket name in the environment variable in serverless.ts like this

environment: {
      AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
      AWS_BUCKET_NAME: 'YOUR BUCKET NAME'
}

Step 6: Define your function in serverless.ts file like this

import type { AWS } from '@serverless/typescript';

const serverlessConfiguration: AWS = {
  service: 'scrapeContent',
  frameworkVersion: '2',
  custom: {
    webpack: {
      webpackConfig: './webpack.config.js',
      includeModules: true
    }
  },
  // Add the serverless-webpack plugin
  plugins: ['serverless-webpack', 'serverless-offline'],
  provider: {
    name: 'aws',
    runtime: 'nodejs14.x',
    apiGateway: {
      minimumCompressionSize: 1024,
    },
    environment: {
      AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
      AWS_BUCKET_NAME: 'scrape-data-at-56'
    },
  },
  functions: {
    scrapeContent: {
      handler: 'handler.scrapeContent',
      events: [
        {
          http: {
            method: 'get',
            path: 'scrapeContent',
          }
        }
      ]
    }
  }
}

module.exports = serverlessConfiguration;

Step 7: In your handler.ts file define your function to do the following

Receive the url to scrape data of from query string
Make a get request to the url using axios
Parse the response data using cheerio
Extract data from the parsed response object and store them in a JSON file and all the image urls in an excel file
Upload the extracted data up to S3

import { APIGatewayEvent } from "aws-lambda";
import "source-map-support/register";
import axios from "axios";
import * as cheerio from "cheerio";


import { badRequest, okResponse, errorResponse } from "./src/utils/responses";
import { scrape } from "./src/interface/scrape";
import { excel } from "./src/utils/excel";
import { getS3SignedUrl, uploadToS3 } from "./src/utils/awsWrapper";

export const scrapeContent = async (event: APIGatewayEvent, _context) => {

  try {

    if (!event.queryStringParameters?.url) {
      return badRequest;
    }

    //load page
    const $ = cheerio.load(await (await axios.get(event.queryStringParameters?.url)).data);

    //extract title and all images on page
    const scrapeData = {} as scrape;
    scrapeData.images = [];
    scrapeData.url = event.queryStringParameters?.url;
    scrapeData.dateOfExtraction = new Date();
    scrapeData.title = $("title").text();
    $("img").each((_i, image) => {
      scrapeData.images.push({
        url: $(image).attr("src"),
        alt: $(image).attr("alt"),
      });
    });

    //add this data to a an excel sheet and upload to s3
    const excelSheet = await saveDataAsExcel(scrapeData);
    const objectKey = `${scrapeData.title.toLocaleLowerCase().replace(/ /g, '_')}_${new Date().getTime()}`;
    await uploadToS3({
      Bucket: process.env.AWS_BUCKET_NAME,
      Key: `${objectKey}.xlsx`,
      ContentType:
        'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
      Body: await excelSheet.workbook.xlsx.writeBuffer()
    });

    //Get signed url with an expiry date
    scrapeData.xlsxUrl = await getS3SignedUrl({
      Bucket: process.env.AWS_BUCKET_NAME,
      Key: `${objectKey}.xlsx`,
      Expires: 3600 //this is 60 minutes, change as per your requirements
    });

    //Upload to S3 & give a link to download result as xslx
    await uploadToS3({
      Bucket: process.env.AWS_BUCKET_NAME,
      Key: `${objectKey}.json`,
      ContentType:
        'application/json',
      Body: JSON.stringify(scrapeData)
    });

    return okResponse(scrapeData);
  } catch (error) {
    return errorResponse(error);
  }
};

/**
 * 
 * @param scrapeData 
 * @returns excel
 */
async function saveDataAsExcel(scrapeData: scrape) {
  const workbook:excel = new excel({ headerRowFillColor: '046917', defaultFillColor: 'FFFFFF' });
  let worksheet = await workbook.addWorkSheet({ title: 'Scrapped data' });
  workbook.addHeaderRow(worksheet, [
    "Title",
    "URL",
    "Date of extraction",
    "Images URL",
    "Image ALT Text"
  ]);

  workbook.addRow(
    worksheet,
    [
      scrapeData.title,
      scrapeData.url,
      scrapeData.dateOfExtraction.toDateString()
    ],
    { bold: false, fillColor: "ffffff" }
  );

  for (let image of scrapeData.images) {
    workbook.addRow(
      worksheet,
      [
        '', '', '',
        image.url,
        image.alt
      ],
      { bold: false, fillColor: "ffffff" }
    );
  }

  return workbook; 
}

Step 8: Set your aws access key and aws secret key in your environment like this

export AWS_ACCESS_KEY_ID = YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY = YOUR_ACCESS_SECRET_KEY

Step 9: You are now ready to run this function on your machine like this sls offline --stage local

Now you should be able to access your function from your machine like this http://localhost:3000/local/scrapeContent?url=ANY_URL_YOU_WISH_TO_SCRAPE

Step 10: If you wish to deploy this lambda function on your AWS account then you can do it like this -
sls deploy

You can checkout this lambda function from here.

How to create a simple docker image with AWS CLI and Serverless installed

metacollective — Fri, 11 Feb 2022 17:17:24 +0000

A step by step guide on how to create a simple docker image and host it on the docker hub

A simple and easy step-by-step guide on how to create your own docker image and then host it on the docker hub. In this tutorial, we will create an image with the following software installed

nodejs (v14)
git
bash
curl
openssh
py3-pip
wget
aws cli
serverless

Step 1: Create an account on docker hub website by going here - https://hub.docker.com/signup

Step 2: Create a new repository on your docker hub account by going here https://hub.docker.com/repository/create?namespace=[you account name].

You can mark it private or public as per your requirement. If private then only those who you want can access it, with the public, it's for everyone.

Step 3: Install docker for desktop. You can download installables from - https://www.docker.com/products/docker-desktop

Step 4: Login into your desktop docker using your docker hub account

Step 5: Open your editor (e.g: visual studio code) and add a new file called Dockerfile. Add the following lines into your file



FROM node:14-alpine

# Install packages
RUN apk update && apk add --update --no-cache \
    git \
    bash \
    curl \
    openssh \
    python3 \
    py3-pip \
    py-cryptography \
    wget \
    curl

RUN apk --no-cache add --virtual builds-deps build-base python3

# Update NPM
RUN npm config set unsafe-perm true
RUN npm update -g

# Install AWSCLI
RUN pip install --upgrade pip && \
    pip install --upgrade awscli

# Install Serverless Framework
RUN npm install -g serverless

FROM node:14-alpine: This image is based on the popular Alpine Linux project, available in the alpine official image. Alpine Linux is much smaller than most distribution base images (~5MB), and thus leads to much slimmer images in general.

Install various packages you want to install in this image
like this -

RUN apk update && apk add — update — no-cache \
git \
bash \
curl \
openssh \
python3 \
py3-pip \
py-cryptography \
wget \
curl

Update NPM

RUN npm config set unsafe-perm true
RUN npm update -g

Install AWS CLI

RUN pip install — upgrade pip && \
pip install — upgrade awscli

Install Serverless framework globally. You must install the Serverless framework globally for it to work. If you install it via a project then it will not work and any of your serverless commands will fail in this image’s container

RUN npm install -g serverless

Now you are all set to push this image up to the docker hub.
Go to your terminal and run this command to build an image from this Dockerfile -

docker build -t [replace this with your username]/[replace this with the name of your repository you created in step2]

Once it's built, you can find your image on your local machine by running this command -

docker images

Now, simply push your image to the docker hub

docker push [replace with your account]/[replace with your image name]

Check your docker hub account and your image should be there

S3 helper functions in typescript

metacollective — Thu, 10 Feb 2022 17:43:18 +0000

Following is a list of some of the most commonly used s3 functions (aws-sdk) and how you can use them using typescript.

getObject
upload
deleteObject
listObjectsV2
headObject

getObject: Retrieves objects from Amazon S3

export async function downloadFromS3(
  params: S3.GetObjectRequest
): Promise<any> {
  console.info("---- DOWNLOADING FROM S3", JSON.stringify(params, null, 2));
  try {
    return await s3.getObject(params).promise();
  } catch (error) {
    console.log(error);
    throw error;
  }
}

This function expects a parameter of GetObjectRequest type and its definition can be found here - https://github.com/aws/aws-sdk-js/blob/master/clients/s3.d.ts

You can call this function like this

await downloadFromS3({
  Bucket: "bucketName",
  Key: "objectKey",
});

upload: Uploads objects to Amazon S3

export async function uploadtoS3(s3Data: S3.PutObjectRequest) {
  console.info(
    "---- UPLODAING TO S3",
    JSON.stringify(`${s3Data.Bucket} ${s3Data.Key}`, null, 2)
  );

  try {
    return await s3.upload(s3Data).promise();
  } catch (error) {
    console.log(error);
    return error;
  }
}

This function expects a parameter of PutObjectRequest type and its definition can be found here - https://github.com/aws/aws-sdk-js/blob/master/clients/s3.d.ts

You can call this function like this

await uploadtoS3({
  Bucket: "bucketName",
  Key: "objectKey",
  ACL: "public-read",
  CacheControl: "max-age=86400",
  Body: JSON.stringify(dataObject),
  ContentType: "application/json",
});

deleteObject: deletes an objects from Amazon S3

export async function deleteFromS3(
  params: S3.DeleteObjectRequest
): Promise<any> {
  console.info("---- DELETE FROM S3", JSON.stringify(params, null, 2));
  try {
    return await s3.deleteObject(params).promise();
  } catch (error) {
    console.log(error);
    throw error;
  }
}

This function expects a parameter of DeleteObjectRequest type and its definition can be found here - https://github.com/aws/aws-sdk-js/blob/master/clients/s3.d.ts

You can call this function like this

await deleteFromS3({
  Bucket: "bucketName",
  Key: "objectKey",
});

listObjectsV2: Returns some or all (up to 1,000) of the objects in a bucket with each request. You can use the request parameters as selection criteria to return a subset of the objects in a bucket.

//only returns a max of 1000 keys hence we must check for more. Look for IsTruncated value and then recurse.

export async function listFilesFromS3(s3Request: S3.ListObjectsV2Request, allKeys: Array<string>) {
  console.info("---- LISTING S3 BUCKET", JSON.stringify(s3Request, null, 2));

  try {
    const data:S3.ListObjectsV2Output = await s3.listObjectsV2(s3Request).promise();
    let contents = data.Contents;
    contents.forEach(function (content) {
      allKeys.push(content.Key);
    });
    if (data.IsTruncated) {
      s3Request.ContinuationToken = data.NextContinuationToken;
      console.log("get further list...");
      return listFilesFromS3(s3Request, allKeys);
    } else {
      return allKeys;
    }
  } catch (error) {
    console.log(error);
    return error;
  }
}

This function expects a parameter of ListObjectsV2Request type and its definition can be found here - https://github.com/aws/aws-sdk-js/blob/master/clients/s3.d.ts

You can call this function like this

await listFilesFromS3(
  {
    Bucket: "bucketName",
  },
  []
);

headObject: The HEAD action retrieves metadata from an object without returning the object itself. This action is useful if you're only interested in an object's metadata. To use HEAD, you must have READ access to the object.

export async function getHeadObect(s3Data: S3.HeadObjectRequest) {
  console.info(
    "---- GET HEAD OBJECT",
    JSON.stringify(`${s3Data.Bucket} ${s3Data.Key}`, null, 2)
  );

  try {
    return await s3.headObject(s3Data).promise();
  } catch (error) {
    console.log(error);
    return error;
  }
}

This function expects a parameter of HeadObjectRequest type and its definition can be found here - https://github.com/aws/aws-sdk-js/blob/master/clients/s3.d.ts

An example of how you can call this function can be like this

await listFilesFromS3(
  {
    Bucket: "bucketName",
  },
  []
);

You can see an example of how I use these wrapper functions here.

Bitbucket pipeline for deploying Lambda using serverless

metacollective — Wed, 09 Feb 2022 14:55:38 +0000

A step by step guide on how to deploy your AWS's lambda function developed using the ServerLess framework using bitbucket's pipeline.

Prerequisite:

A lambda function project built using the ServerLess Framework
AWS access key & AWS secret access key with enough rights to deploy lambda function

Step 1: Enable pipelines in bitbucket

For this, head over to your bitbucket repository and enable the pipelines by going to Repository settings -> Pipelines settings

Step 2: Add your AWS credentials to Repository variables by going to Repository settings -> Repository variables

You can name your variables whatever you like, but make sure that they are different for each environment you would like to deploy to

Step 3: Add a file named bitbucket-pipelines.yml in the root of your serverless project

Since this is a serverless framework project, this pipeline will require a container with serverless installed globally. For this I already have a docker image with the following tools installed

nodejs (v14)
git
bash
curl
openssh
py3-pip
wget
serverless

You can get this image from here - https://hub.docker.com/repository/docker/metacollective/serverless_ci
or if you wish to push your own image, then you can use this Dockerfile



FROM node:14-alpine

# Install packages
RUN apk update && apk add --update --no-cache \
    git \
    bash \
    curl \
    openssh \
    python3 \
    py3-pip \
    py-cryptography \
    wget \
    curl

RUN apk --no-cache add --virtual builds-deps build-base python3

# Update NPM
RUN npm config set unsafe-perm true
RUN npm update -g

# Install AWSCLI
RUN pip install --upgrade pip && \
    pip install --upgrade awscli

# Install Serverless Framework
RUN npm install -g serverless

With the container image sorted, all we have to do is add the following lines to our bitbucket-pipelines.yml



image: metacollective/serverless_ci

pipelines:
  branches:
    default:
      - step:
          script:
            - npm install
    dev:
      - step:
          caches:
              - node
              - pip
          script:
              - npm install
              - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_NON_PROD
              - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_NON_PROD
              - sls deploy --stage dev --region eu-west-1
    qa:
      - step:
          caches:
              - node
              - pip
          script:
              - npm install
              - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_NON_PROD
              - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_NON_PROD
              - sls deploy --stage qa --region eu-west-1          
    master:
      - step:
          caches:
              - node
              - pip
          script:
              - npm install
              - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
              - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
              - sls deploy --stage prod --region eu-west-1

As you can see, we can now manage the pipeline for each branch from this one file.

Let's go through some of the important steps -
branches: It's like a switch case where it picks your deployed branch name and executes whatever is under it
npm install : Install node packages
export : Set environment variables. You can pick and choose different AWS accounts for each branch if you like and add their details in the repository variables under pipeline settings
sls deploy : Serverless command to push your Lambda function to AWS. Not only will it push to AWS, but it will also attach API Gateway with it.

After pushing your changes to bitbucket, you can monitor your pipeline building like this -

Now you have a working pipeline 🎉🎉🎉

p.s: If there is an error that says something along the lines of access denied, then check your AWS credentials and make sure that it has the required lambda execution role (https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html)

One feature I love about Fish (the friendly interactive shell)

metacollective — Tue, 08 Feb 2022 14:11:28 +0000

As programmers, we all have our specific choices of tools that we stick by, especially when it comes to using command-line tools. And, as the title of this post says, I am a big fan of FISH shell (integrated with iterm2).
You can find all about FISH and how cool it is in its documentation, but what really impressed me is there functions feature.

At any given time I am working on multiple projects which means I have to switch environments all the time. Each environment comes with its own set of environment variables and managing them was a lot harder till I found functions in fish.

All I have to do is set functions in fish’s config file (usually found here — ~/.config/fish/config.fish) like this -

function setPROJECT1ENV_LOCAL
  set -g -x CMS_AUTH_SECRET secretvalue
  set -g -x AWS_ACCESS_KEY_ID secretkey
  set -g -x AWS_SECRET_ACCESS_KEY moresecretkey
  set -g -x DB_PASSWORD secretpassword
  set -g -x API_KEY moresecretkey
end

function setPROJECT2ENV_LOCAL
  set -g -x CMS_AUTH_SECRET secretvalue2
  set -g -x AWS_ACCESS_KEY_ID secretkey2
  set -g -x AWS_SECRET_ACCESS_KEY moresecretkey2
  set -g -x DB_PASSWORD secretpassword2
  set -g -x API_KEY moresecretkey2
end

function setPROJECT3ENV_LOCAL
  set -g -x CMS_AUTH_SECRET secretvalue3
  set -g -x AWS_ACCESS_KEY_ID secretkey3
  set -g -x AWS_SECRET_ACCESS_KEY moresecretkey3
  set -g -x DB_PASSWORD secretpassword3
  set -g -x API_KEY moresecretkey3
end

Now whenever you have to switch environments, just call function from your fish shell like this -
setPROJECT1ENV_LOCAL or setPROJECT2ENV_LOCAL

Happy fishing everyone 🐟 😄

How to take backup of a database through SSH tunneling

metacollective — Mon, 07 Feb 2022 12:35:30 +0000

Step by step guide on how to take a database backup through an ssh tunnel

There are situations where we have to take backup of databases that are only accessible via ssh and that can be a challenge if you don’t know the steps. One example of that can be a database only accessible via a jump box (think AWS’s RDS in a VPC only accessible via an ec2 instance, where this ec2 instance has its port 22 open to the world)

For this tutorial my example setup is like this -

The Database is AWS’s RDS (Relational database system) Aurora within a VPC (virtual private cloud)

Step 1: Open a terminal and connect to your database instance via the ec2 instance (or any other server that you have)



sudo ssh -N -v -L 9001:[replace with host url for the database]:5432 -i key_file.pem [replace with user for the server]@[replace with ec2/server ip/url]

In the above command, we are connecting to the server using the key file and username via port 22 (default), and once that is done, then we are connecting to the database server. In the example above, I am connecting to a Postgres database on port 5432 via my local machine’s port 9001. You can adapt it to the port of your database (3306 for MySQL and so on… ) and any other available port on your local machine.

Step 2: Open a new tab on the same terminal and run your db backup command over port 9001



pg_dump -U [replace with you db username] -v --format=c -h localhost -p 9001 --file=db_backup.psql [replace with your database name]

Now you are running commands on your connected database server (and it’s not just limited to taking backups, you can do whatever you are allowed with your database user).

Once this is complete you will have a backup file on your local machine 🙌 🎆 🎉

A simple serverless architecture on AWS ecosystem for data ETL and visualization

metacollective — Sun, 06 Feb 2022 15:36:53 +0000

A simple step by step architecture for building an ETL process and then displaying that data back to the enduser by only using serverless service from AWS.

End user uploads data file into the S3 (This can be done manually by logging onto the console or via an API accessible through a frontend application)
S3 triggers a Lambda function on receiving a create object event
Lambda function receives S3 event object and processes the data from file into a relational database on AWS's Aurora database (ETL)
End user can now access this data from database by accessing the web application statically hosted on S3 & served via the Cloudfront
This web application will make relevant API calls via ApiGateway (attached to Lambda functions) to fetch data