Forem: Ted Ngeene

How to bypass captcha with 2captcha and Selenium using Python

Ted Ngeene — Fri, 28 Jan 2022 20:02:54 +0000

When you go to any modern website, they ask you to solve a challenge before submitting a form, such as identifying all firehoses, a slider to solve a puzzle or typing out some texts. This is called CAPTCHA(short for Completely Automated Public Turing tests to tell Computers and Humans Apart). They usually add a layer of security to prevent bots and malicious accounts from accessing a website's resources.

The squiggly lines and words of the past are less common nowadays and have been replaced by Google's version 2 of CAPTCHA, known as reCAPTCHA or key puzzles from keyCAPTCHA. However, bots are becoming more advanced and can bypass almost any captcha. In this tutorial, we'll make use of the captcha bypass software, 2captcha.

Why are we doing this?

Okay, I know some of you might be wondering, why would we need to bypass captcha? and most importantly; is this even legal? No, it is not illegal to bypass 2captcha.
This article is intended for;

Curious about how you can bypass captcha
Building web scrapers that would need to bypass a captcha

*Disclosure: I only recommend products I would use myself and all opinions expressed here are my own. This post may contain affiliate links that at no additional cost to you, I may earn a small commission.

What is 2captcha?

2Captcha.com is a captcha solving service that automates the captcha solving process. They have an API and several packages that wrap around different programming languages. All you need is to register on their website, get an API key and make requests to their API.

Note: They charge a small rate for every captcha solved, however, the fee is quite minimal at $0.05 to $ 1.00 for every 1000 captchas solved.

Signup Process and supported languages

Supported captchas the software solves include;

Requirements and Setup

For this tutorial, we'll utilize the 2captcha API. You'll need to have a developer account to use. You can head over to this link and sign up
The source code we'll be using can be found here. Setup instructions have been linked in the project README file. Feel free to clone the repository and tinker around.

Let's dive in.

Automating with Selenium

Before we get carried away by CAPTCHA, we need to understand the process our program will follow. I'm going to use selenium and python. Selenium is a browser automation service that provides extensions that allow programs to emulate browser interactions. It supports an array of browsers; from chrome, firefox, safari, and so forth, by integrating with their web drivers.

Captcha Solving Pseudocode

Now that we have our account setup, we'll do the actual capture bypassing, which brings us to the next part.

For this tutorial, i'll be bypassing the capture on this sample form

In order to achieve our objective, we'll need our program to follow these steps.

Launch a browser tab.
Fill in the required fields in the form.
Solve the captcha.
Send the contact message with the solved captcha.
Close the browser window.

All this will be achieved automatically!! How cool is that?

For this tutorial, I'll be using chrome driver since chrome is my default browser. To setup selenium, chromedriver, and all project dependencies run the following command

pip install -r requirements.txt

Always ensure you've activated a virtual environment to prevent the packages from being installed globally in your machine.

There are several packages that you'll see in the main.py file. I'll go over them in brief

2captcha-python - the 2captcha python SDK. It does most of the heavy lifting of interacting with the 2captcha API
Selenium with python - contains selenium python bindings.
webdriver-manager - manages our chromedriver installation rather than manually setting path configurations. It's main idea is to simplify management of binary drivers for different browsers.
python-decouple - enables storing of variables in .env or .ini files. This comes in handy when we have sensitive information we might not want exposed to the unauthorized parties, such as API keys.

Project Code

Open an main.py file at the root of your project directory and paste the following code. We'll go over the code in-depth in the next section.

from random import randint
from time import sleep

from decouple import config
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from twocaptcha import TwoCaptcha
from twocaptcha.api import ApiException, NetworkException
from twocaptcha.solver import ValidationException
from webdriver_manager.chrome import ChromeDriverManager

two_captcha_api_key = config('CAPTCHA_API_KEY')
website_url = config('WEBSITE_URL')
site_key = config('GOOGLE_CAPTCHA_KEY')

driver_options = webdriver.ChromeOptions()
driver_options.add_argument('--start_maximized')

# * use 2captcha python client, read docs in https://pypi.org/project/2captcha-python/
solver_config = {
    'apiKey': two_captcha_api_key,
    'defaultTimeout': 120,
    'recaptchaTimeout': 600,
    'pollingInterval': 10,
}

solver = TwoCaptcha(**solver_config)


class SolveCaptcha:

    def launch_selenium(self, response):
        if response:
            print('Captcha Solved! Launching Browser...')
            # initiate chrome webdriver
            # * check webdrivers for firefox and others in https://selenium-python.readthedocs.io/installation.html#drivers
            driver = webdriver.Chrome(service=Service(
                ChromeDriverManager().install()),
                                      options=driver_options)

            # open browser window and navigate to Url
            driver.get(website_url)

            driver.find_element(By.ID, 'name').send_keys('Ted')
            driver.find_element(By.ID, 'phone').send_keys('000000000')
            driver.find_element(By.ID,
                                'email').send_keys('tngeene@captcha.com')
            driver.find_element(By.ID,
                                'comment-content').send_keys('test comment')

            google_captcha_response_input = driver.find_element(
                By.ID, 'g-recaptcha-response')

            # make input visible
            driver.execute_script(
                "arguments[0].setAttribute('style','type: text; visibility:visible;');",
                google_captcha_response_input)
            # input the code received from 2captcha API
            google_captcha_response_input.send_keys(response.get('code'))
            # hide the captch input
            driver.execute_script(
                "arguments[0].setAttribute('style', 'display:none;');",
                google_captcha_response_input)
            # send text
            driver.find_element(By.ID, 'send-message').click()

            sleep(randint(5, 10))

            driver.quit()
            balance = solver.balance()
            # show 2captcha Balance.
            print(f'Your 2captcha balance is ${round(balance, 2)}')
        else:
            print('No response.')

    def initiate_captcha_solver(self):
        try:
            print('Solving captcha...')
            result = solver.recaptcha(sitekey=site_key, url=website_url)
            # launch browser window upon successful
            # completion of captcha solving.
            self.launch_selenium(result)
        except ValidationException as e:
            # invalid parameters passed
            print(e)
            return e
        except NetworkException as e:
            # network error occurred
            print(e)
            return e
        except ApiException as e:
            # api respond with error
            print(e)
            return e
        except TimeoutException as e:
            # captcha is not solved so far
            print(e)
            return e


SolveCaptcha().initiate_captcha_solver()

Importing Packages

from random import randint
from time import sleep

from decouple import config
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from twocaptcha import TwoCaptcha
from twocaptcha.api import ApiException, NetworkException
from twocaptcha.solver import ValidationException
from webdriver_manager.chrome import ChromeDriverManager

These lines are essentially referencing the packages we'll be using. We're telling the program that we'll require to use; decouple, the twocaptcha, and selenium packages.
I've also imported the randint and sleep modules that come as part of the standard python package.
We'll be using these two as well.

Interacting with imported packages

two_captcha_api_key = config('CAPTCHA_API_KEY')
website_url = config('WEBSITE_URL')
site_key = config('GOOGLE_CAPTCHA_KEY')

The first package we need to use is the python-decouple package. Here, we define the environment variables we'll be using for the program. You can obtain the 2captcha api key from the website's dashboard.

Environment variable.

An environment variable is a dynamic-named value that can affect the way running processes will behave on a computer. They are part of the environment in which a process runs. For example, a running process can query the value of the TEMP environment variable to discover a suitable location to store temporary files, or the HOME or USERPROFILE variable to find the directory structure owned by the user running the process.

View on Wikipedia

It's advisable to use environment variables to store sensitive information that we wouldn't want getting into unauthorized hands, for example, our 2Captcha API key. Create a env file at the root of your project folder. Modify it to this.

 CAPTCHA_API_KEY=<api_key_from_2_captcha.com>
 WEBSITE_URL=<website_we'll_be_accessing_captcha>
 GOOGLE_CAPTCHA_KEY=<google_captcha_site_key>

Paste the api key value from 2captcha dashboard on the CAPTCHA_API_KEY field.

GOOGLE_CAPTCHA_KEY is a special identifier found in all web forms having captcha, we can retrieve it by also opening dev tools, and searching the data-sitekey keyword.

Retrieve the value and paste it in the .env, GOOGLE_CAPTCHA_KEY value. python-2captcha sends this key under the hood to the 2captcha API, which then returns an API response of the solved captcha. We'll be seeing this further ahead.

The WEBSITE_URL is the webpage of our captcha form. Fill it in the .env file as well.

solver_config = {
    'apiKey': two_captcha_api_key,
    'defaultTimeout': 120,
    'recaptchaTimeout': 600,
    'pollingInterval': 10,
}

solver = TwoCaptcha(**solver_config)

Next we initialize the twocaptcha instance using the TwoCaptcha constructor. The constructor takes in a few arguments which we have defined inside the solver_config dictionary.
We then pass the configs into the constructor by desctructuring the dictionary.

These arguments are;

apikey: The API key we obtain from 2Captcha.
defaultTimeout: Time (seconds) to wait before giving up on waiting for a captcha solution.
recaptchaTimeout: Polling timeout for ReCaptcha in seconds. Defines how long the module tries to get the answer from res.php API endpoint
pollingInterval: Time (seconds) between polls to 2captcha server. 2Captcha documentation suggests this time to be at least 5 seconds, or you might get blocked.

Retrieving Dom elements

As you can tell from the form, we will need selenium to access the DOM form elements as it automatically fills the inputs. These are; the name, email, phone, comment, and finally there's a hidden google captcha field that takes the solved captcha as an input.

For the visible form fields, all we need to do is open up the dev tools and retrieve the individual ids of the form fields.

This section does just that.

   def launch_selenium(self, response):
        if response:
            print('Captcha Solved! Launching Browser...')
            # initiate chrome webdriver
            # * check webdrivers for firefox and others in https://selenium-python.readthedocs.io/installation.html#drivers
            driver = webdriver.Chrome(service=Service(
                ChromeDriverManager().install()),
                                      options=driver_options)

            # open browser window and navigate to Url
            driver.get(website_url)

            driver.find_element(By.ID, 'name').send_keys('Ted')
            driver.find_element(By.ID, 'phone').send_keys('000000000')
            driver.find_element(By.ID,
                                'email').send_keys('tngeene@captcha.com')
            driver.find_element(By.ID,
                                'comment-content').send_keys('test comment')

What we're telling selenium is, to launch chrome browser, and visit the specified url. Once it does so, find the DOM elements that matches the provided ids. Since these are form inputs, auto-populate those field with the data inside the send_keys function.

All 2captchas have a hidden text-area that autofills with the solved captcha code once you click the i'm not a robot check. Usually, this has the id set as g-recaptcha-response

Since selenium simulates browser human inputs, it'll be required to make the field visible. We achieve this with this code snippet

  google_captcha_response_input = driver.find_element(
                By.ID, 'g-recaptcha-response')

            # make input visible
            driver.execute_script(
                "arguments[0].setAttribute('style','type: text; visibility:visible;');",
                google_captcha_response_input)
            # input the code received from 2captcha API
            google_captcha_response_input.send_keys(response.get('code'))
            # hide the captcha input
            driver.execute_script(
                "arguments[0].setAttribute('style', 'display:none;');",
                google_captcha_response_input)
            # send text
            driver.find_element(By.ID, 'send-message').click()

This section typically makes the field visible, auto-populates the field with the solved captcha, hides the field again, and finally, the button click is simulated to send the comment with the solved captcha.

Finally, we'll close the browser tab a few seconds after the captcha has been solved. We're using the randint function to close the browser anytime between 5 and 10 seconds

            sleep(randint(5, 10))

            driver.quit()
            balance = solver.balance()
            # show 2captcha Balance.
            print(f'Your 2captcha balance is ${round(balance, 2)}')

The last section queries your 2captcha balance and returns the account balance of your account so as to give you a small report on the cash balance. As mentioned before, each captcha solved attracts a small fee.

All the fore-mentioned functionality resides in the launch_selenium() function. We need to tie it all together with the 2captcha service.

From the main.py file, you can see we have a initiate_captcha_solver() function.


    def initiate_captcha_solver(self):
        try:
            print('Solving captcha...')
            result = solver.recaptcha(sitekey=site_key, url=website_url)
            # launch browser window upon successful
            # completion of captcha solving.
            self.launch_selenium(result)
        except ValidationException as e:
            # invalid parameters passed
            print(e)
            return e
        except NetworkException as e:
            # network error occurred
            print(e)
            return e
        except ApiException as e:
            # api respond with error
            print(e)
            return e
        except TimeoutException as e:
            # captcha is not solved so far
            print(e)
            return e

We are calling the python_two_captcha sdk we'd initialized before, by
result = solver.recaptcha(sitekey=site_key,
url=website_url)

We're also catching some exceptions that may be returned by the 2Captcha API.

This result will be used as an argument in the launch_selenium function as we'll need the output of the response as an input to our form.

Selenium will only be launched upon a successful captcha solve, which usually takes a few seconds. For Recaptcha version 2, the ETA is usually anytime from 15 seconds to 45seconds. Recaptcha version 3 takes a shorter time.If a request timed out, we log the api response.

Demo

Okay, now your application is set up! It may feel like a lot 😅 but we did a lot of installation. We will now test our application.

Conclusion

It’s always a dilemma, should websites have a better experience and have simple to bypass the CAPTCHA or should websites aggressively protect themselves from bots and have a bad user experience. The war between websites and bots is never over. Whatever verification method websites pull out, it’s just a matter of time when someone figures out how to bypass it. ~ Filip Vitas

In this guide, we were introduced to 2captcha API, selenium, and a few concepts in 2captcha using python. By the end of it, I hope you can apply the knowledge gained to build your own captcha bypass service. I mean, if bots can do it, then so should we! A few next steps would be to add a User interface to input our values. You can also look into using the 2captcha API using your preferred programming language and other tools such as puppeteer, i've also previously written an article on the same using javascript. You can find this by visiting this link

Finally, if you liked the content and would like to use 2captcha, sign up with this link.

If you have any questions, you can always leave a comment below, or reach out on these channels;

Source code of demo project can be accessed here.

Use 2captcha responsibly.

Sponsors

Scraper API is a startup specializing in strategies that'll ease the worry of your IP address from being blocked while web scraping. They utilize IP rotation so you can avoid detection. Boasting over 20 million IP addresses and unlimited bandwidth. Using Scraper API and a tool like 2captcha will give you an edge over other developers. The two can be used together to automate processes. Sign up on Scraper API and use this link to get a 10% discount on your first purchase.
Do you need a place to host your website or app, Digital ocean
is just the solution you need, sign up on digital ocean using this link and experience the best cloud service provider.
The journey to becoming a developer can be long and tormentous, luckily Pluralsight makes it easier to learn. They offer a wide range of courses, with top quality trainers, whom I can personally vouch for. Sign up using this link and get a 50% discount on your first course.

Automating Django Deployment workflow with Github Actions

Ted Ngeene — Thu, 13 Jan 2022 11:53:08 +0000

If you’ve worked on a project that has constant changes, tests, and needs to be deployed to a server every time the changes occur, then you’re aware how the process of logging in to the server via ssh, redeploying, and testing is quite repetitive and frankly annoying sometimes. Luckily, with the advent of Github actions, this can be automated, leading to a happy developer.

The purpose of this article is to get you introduced to the concept of Github actions for a typical Django app deployment, but it can easily apply to any other framework. So...let’s see some Github actions in action.

Introduction to Github actions

According to the official docs, the definition of GitHub actions is;

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can create workflows that build and test every pull request to your repository or deploy merged pull requests to production.

GitHub Actions goes beyond just DevOps and lets you run workflows when other events happen in your repository.

GitHub provides Linux, Windows, and macOS virtual machines to run your workflows, or you can host your own self-hosted runners in your own data center or cloud infrastructure.

What this basically means is that they provide a Continuous Integration and Continuous Delivery pipeline, all in one platform. This is a powerful tool for dev teams and Individual Contributors and is usually a key step in the software development life cycle.

Key Terms and Concepts In Github Actions

Before we begin, let’s familiarize ourselves with a few terms and concepts that you’ll be seeing along with the post.

Workflow

A workflow is a configurable automated process that will run one or more jobs. Workflows are defined by a YAML file in your repository and will run when triggered by an event in your repository, or they can be triggered manually, or at a defined schedule.

Your repository can have multiple workflows each of which can perform a different set of steps. For example, you can have one workflow to build and test pull requests, another workflow to deploy your application every time a release is created, and still another workflow that adds a label every time someone opens a new issue.

Workflows are contained in the .github/workflows directory.

Events

An event is a specific activity in a repository that triggers a workflow run. For example, an event can be a pull request, push, comment, and so on. For more explanation on this, see events that trigger workflows.

Jobs

A job is a set of steps that execute on your workflow in the same runner. An example of a job would be running unit tests. A step can either be a shell script that will be executed or an action to be run. Steps are executed in order and are dependent on each other. Since each step is executed on the same runner, you can share data from one step to another. For example, you can have a step that builds your application followed by a step that tests the application that was built. which is what we’re going to be doing.

Actions

An action is a custom application for the GitHub Actions platform that performs a complex but frequently repeated task. Actions help reduce the amount of repetitive code that you write in your workflow files. An action can pull your git repository from GitHub, set up the correct toolchain for your build environment, or set up the authentication to your cloud provider.

You can write your own actions, or you can find actions to use in your workflows in the GitHub Marketplace.

Runners

A runner is a server that runs your workflows when they're triggered. Each runner can run a single job at a time. GitHub provides Ubuntu Linux, Microsoft Windows, and macOS runners to run your workflows; each workflow run executes in a fresh, newly-provisioned virtual machine.

Secrets

Secrets are environment variables that your jobs depend on. These are usually sensitive information that you wouldn’t want to be exposed to the public. Such secrets include; secret keys, API keys, access tokens, and passwords.

What you need for this config

Before we set up our first workflow, you’ll need a few things.

A Github repository - this is the repo you want to set up your actions. For this demo, you can reference this repo
Nektos/act package - it’s always advisable that you test your actions locally before pushing them to Github. This package enables us to do so.
A virtual private server - where we’ll be deploying our application. I recommend using digital ocean. They provide the best servers at affordable tiers. Sign up using my affiliate link to get $100 credits for 60 days.
1. Docker and docker-compose- the nektos/act package runs on docker since we’ll be virtualizing our runners. Install them both on your machine and the server.

Setting Our first workflow

For this tutorial, we need to first understand what our objective is before we write any workflow files.

We want our workflow to;

Run our tests every time a pull request is made to the main branch.
Deploy to a server every time a push is made to the main branch.

For this setup, we’ll need two workflow files, i.e, tests.yml, and deploy.yml files.

Let’s get started.

Running Tests

On your repo, switch to a new branch, in my case, I have a workflows branch.

Run

mkdir .github && mkdir ./github/workflow && touch ./github/workflows/tests.yml

Open the tests.yml file and paste the following code.

name: Django Tests CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-18.04
    strategy:
      max-parallel: 4
      matrix:
        python-version: [3.8, 3.9]

    steps:
      - uses: actions/checkout@v2
      # this fixes local act bug of python setup
      - name: local act python setup fix
        run: |
          # Hack to get setup-python to work on act
          # (see https://github.com/nektos/act/issues/251)
          if [ ! -f "/etc/lsb-release" ] ; then
            echo "DISTRIB_RELEASE=18.04" > /etc/lsb-release
          fi
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install Dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Run Tests
        env:
          DEBUG: ${{ secrets.DEBUG }}
          SECRET_KEY: ${{ secrets.SECRET_KEY }}
          DB_ENGINE: ${{ secrets.DB_ENGINE }}
          DB_NAME: ${{ secrets.DB_NAME }}
          BASE_WEATHER_API_URL: ${{ secrets.BASE_WEATHER_API_URL }}
          WEATHER_API_KEY: ${{ secrets.WEATHER_API_KEY }}
        run: |
          python manage.py test core.tests

Starting from the top, we specify our workflow name

name: Django Tests CI

This can be whatever name you want but always use a name that’s easy to remember and descriptive of what the file does.

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

The on keyword is the entrypoint for our events, in this file, we’d like the action to run every time a push and a pull request is made to the main branch. This way, tests will always run if these scenarios are triggered.

jobs:
  test:
    runs-on: ubuntu-18.04
    strategy:
      max-parallel: 4
      matrix:
        python-version: [3.8, 3.9]

This bit has a few configurations.

We create a job called test.
we specify that the job will run on ubuntu 18.04 since my server is running on that OS.
Inside the test job, we specify a strategy matrix. This allows running our tests on a few different python variations. In this case, we want the tests to be run in python 3.8 and python 3.9 environments.
max-parallel The maximum number of jobs that can run simultaneously when using a matrix job strategy

steps - Next, we define the series of steps the job will follow. Steps are usually sequential, that is, they’re executed one after the other.

 - uses: actions/checkout@v2
      # this fixes local act bug of python setup
      - name: local act python setup fix
        run: |
          # Hack to get setup-python to work on act
          # (see https://github.com/nektos/act/issues/251)
          if [ ! -f "/etc/lsb-release" ] ; then
            echo "DISTRIB_RELEASE=18.04" > /etc/lsb-release
          fi

The first step is a caveat to note when working with act in a local environment (for mac users). It’ll likely throw an error that there’s a problem setting up python. The command is a bash script that resolves this issue. You can ignore it if you won’t face such a headwind.

 - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install Dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

This next step sets up the python versions specified in the matrix. We install the dependencies and update pip in the bash script. (defined in the run keyword)

    - name: Run Tests
        env:
          DEBUG: ${{ secrets.DEBUG }}
          SECRET_KEY: ${{ secrets.SECRET_KEY }}
          DB_ENGINE: ${{ secrets.DB_ENGINE }}
          DB_NAME: ${{ secrets.DB_NAME }}
          BASE_WEATHER_API_URL: ${{ secrets.BASE_WEATHER_API_URL }}
          WEATHER_API_KEY: ${{ secrets.WEATHER_API_KEY }}
        run: |
          python manage.py test core.tests

Finally, we run our tests.

You have noticed the environmental variables in this action. To create environmental variables in your github repo go to your repository > settings > left-sidebar > secrets

Add whatever secrets you might require.

Since we’ll need to test our actions locally first, create a .secrets file in your project root. I decided to name mine ‘act.secrets’.

For a more detailed explanation of how secrets work, check out the official documentation.

Finally, let’s run the tests workflow by using;

act —secret-file act.secrets

Drumrolls 🥁🥁.... if you set up everything correctly, the output should be like this;

Congratulations, you’ve made your first GitHub action! Next, push the code to GitHub, make a PR to the main branch and check if it’s been triggered. Finally, merge the PR to the main branch. This should also trigger the workflow.

Setting Up AutoDeploy Pipeline

Now that we’ve set up our tests action, we’ll need to pass deploy to our server when we merge into the main branch.

For this, make a deploy.yml file under the workflows folder and paste the following snippet.

name: Django Deploy CD
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-18.04
    steps:
      - name: Deploy to server
      # don't run locally
        if: ${{ !env.ACT }}
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.LIVE_SERVER_IP }}
          username: ${{ secrets.SERVER_USERNAME }}
          password: ${{ secrets.SERVER_USER_PASSWORD }}
          port: ${{ secrets.DEPLOY_PORT }}
          script: |
            cd ${{ secrets.PROJECT_PATH }}
            git pull ${{secrets.REPO_URL}}
            docker-compose down 
            docker-compose up --build -d

Like the first file, this folder follows the same structure, just different commands. I will cover the new parts.

if: ${{ !env.ACT }} - this command instructs the nektos/act package to ignore this job. This is because, in most cases, we don’t want to deploy to our server from our local environment.

uses: appleboy/ssh-action@master - this instructs our job to use the publicly available GitHub action, “appleboy/ssh-action@master” which is an ssh client. Check out their official docs.

   with:
          host: ${{ secrets.LIVE_SERVER_IP }}
          username: ${{ secrets.SERVER_USERNAME }}
          password: ${{ secrets.SERVER_USER_PASSWORD }}
          port: ${{ secrets.DEPLOY_PORT }}

This action requires us to define a few secrets. The final expected GitHub secrets for this job are;

LIVE_SERVER_IP - server ip address
SERVER_USERNAME - user logging into the server
DEPLOY_PORT - ssh port, default is 22
SERVER_USER_PASSWORD - password of user, can using public key as well in place of this
USERNAME= github username
PROJECT_PATH - path the project is located on server
REPO_URL=https://{USERS_GITHUB_TOKEN}@github.com/{USERNAME}/{repo-name}.git

Finally, we run a series of commands once the action has managed to ssh into the server. Since I’m using docker and docker-compose, my pipeline is as follows.

   script: |
            cd ${{ secrets.PROJECT_PATH }}
            git pull ${{secrets.REPO_URL}}
            docker-compose down 
            docker-compose up --build -d

For the final piece, push the code to Github, make a PR to the main branch(remember this will trigger the test workflow). Once the tests have run, merge the PR, and if you followed along fine, the tests and deployment actions will be set in motion.

If you configured everything well, the output should be similar to this.

Test workflow

Deploy Action

Resources

The content of this post can be nerve-wracking and seem a lot to bite, however, if you feel you need further reading on some of the concepts explained, do check out these curated resources I used while doing my research.

The official Github actions documentation is a treasure trove on all things Github actions.
Colby Fayock’s article on Freecodecamp.
This tutorial on Brad Traversy’s Youtube channel.

Conclusion

Github actions offer a way for DevOps teams to quickly set up a CI/CD pipeline. In this post, we set up a Django application, with some automated tests. We also set up two GitHub actions; one that automatically runs our tests when a pull request is made to the main branch, and another that automatically deploys the application to a virtual private server.

You can read further on working with different environments, setting up restrictions on certain actions and deploying your own actions to the Github marketplace.

Thanks for following along this far and I hope this goes a long way to aid in saving your precious time by avoiding manually doing this.

If you have any thoughts, extra pointers, or just a comment, feel free to reach out by leaving a comment, or shooting up a dm on Twitter

Sponsors

Scraper API is a startup specializing in strategies that'll ease the worry of your IP address from being blocked while web scraping. They utilize IP rotation so you can avoid detection. Boasting over 20 million IP addresses and unlimited bandwidth. Using Scraper API and a tool like 2captcha will give you an edge over other developers. The two can be used together to automate processes. Sign up on Scraper API and use this link to get a 10% discount on your first purchase.
Do you need a place to host your website or app, Digital ocean is just the solution you need, sign up on digital ocean using this link and experience the best cloud service, provider.
The journey to becoming a developer can be long and tormentous, luckily Pluralsight makes it easier to learn. They offer a wide range of courses, with top quality trainers, whom I can personally vouch for. Sign up using this link and get a 50% discount on your first course.

When are microservices appropriate?

Ted Ngeene — Tue, 30 Nov 2021 23:52:25 +0000

Lately, I've been tending to lean more towards employing a monolithic approach in system architecture. On paper, using micro-services seems like an ideal solution. The fact that you can decouple a project into smaller manageable bits is good. However, from experience and a couple of blog posts and videos I've watched, they do introduce a lot of complexities. Just to mention a few would be; giving database access to a microservice poses major security risks if not properly implemented, error handling, and especially if the said microservice is down would essentially make the system fail, connection error, read timeout, resource not found...you have to deal with many error scenarios.

I'm not completely against them, for example I believe a microservice should do one thing and do it really well. A good example would be sending sms notifications. This wouldn't really need to interact with any database. Just accept a list of phone numbers and send out the messages.

So my question is; when are my microservices appropriate and do we really need them as much as folks preach?

AdonisJs - Introduction to database migrations

Ted Ngeene — Mon, 15 Nov 2021 06:48:08 +0000

Introduction to Database Migrations

This article will introduce you to the concept of migrations in AdonisJs.

In the previous articles, we learned 2 approaches to working with events in AdonisJs. In this sixth installment of the everything you need to know about adonisJs series, we will familiarize ourselves with the concept of migrations. The aim is to have a broader understanding of how our database state will look like.

Source code for this tutorial can be found here

I will link the resources at the end of the article.

Let's get started.

Our Database Structure

Just to remind you how our database should look like, we'll reference this diagram.

Defining Model Relationships

Database migrations and modeling can sometimes be a pain point for many devs to fully understand. It's important to plan your database in the most optimal way that meets business requirements. Therefore, before we start coding, it's crucial to understand know just how our models will be related to each other. I'll attempt to explain this from a high-level view, then head over to the implementation.

We'll use the diagram above as a guide to modeling our database.

A user can own many stores and a store can only be owned by one user, hence this would be a many-to-one relationship.
A store can be followed by many users and a user can follow many stores. So this would be represented by a many-to-many relationship.
An item(product) can only belong to a single store, but a store can have many items, hence this is a one-to-many relationship.
An item can only be uploaded by one user, and a user can upload many items. This will also be represented as a one-to-many relationship.

many-to-one and one-to-many are the same things.

Migrations

Migration files allow us to create schema definitions using code. They are a way to alter a database programmatically. Adonis allows us to do so in a typescript-safe environment.
In most cases, an application will have a migration file for each table it needs. Later on, in the application's lifespan, you may have migrations for any alteration(s) you need to perform in a given table.

The flow of migrations

Migrations have two directions, up and down. We run migrations to go up when we have changes we'd like to persist in our database and down when we want to roll back (undoing changes) when the changes were done.

This flow is good for several reasons:

We can easily roll back changes that aren't in production to make alterations as needed, during development.
We can maintain a history of the changes made to our application database over time.
Alterations made to our migrations are enforced at the database level, hence when we deploy to production, the changes are always synced with our database state. This means that the database is always updated with the latest changes.

It's worth noting that rolling back migrations in production should be done with caution. This is because it poses the risk of losing data. Always have a backup of the database if you wish to roll back in production.

Migration commands and setting up our first migration file

The Adonis CLI ships with several commands that we can run in regards to migrations

make:migration - allows us to create a migration
migration:run - allows us to run all migrations that haven't been previously run, i.e, sync them with the database
migration:rollback - This allows us to roll back previously run migrations. (Return to a previous state)
migration:status - allows us to view the run status on our migrations, i.e, whether the migrations altered the database state.

Migration folders are contained in the database/migrations directory.

Setting up our store Migration

For this tutorial, I'll set up the store migration. All other migration folders will follow the same flow. You can recreate them following similar steps.
If you run into any errors, feel free to check out the source code or leave a comment.

We'll run the make migration command to set up our store schema.

  node ace make:migration stores

This will create a migration file in the migrations directory. Paste the missing values in the folder.


import BaseSchema from '@ioc:Adonis/Lucid/Schema'

export default class Stores extends BaseSchema {
  protected tableName = 'stores'

  public async up() {
    this.schema.createTable(this.tableName, (table) => {
      table.increments('id').primary()
      table.string('name', 255).notNullable()
      table.integer('owner').references('users.id').onDelete('CASCADE')
      table.string('description').notNullable()
      table.string('cover_photo').nullable()
      table.string('slug').notNullable().unique()
      table.string('email', 255).notNullable()
      table.string('phone_number', 30)
      table.boolean('is_active').defaultTo(true)
      /**
       * Uses timestamptz for PostgreSQL and DATETIME2 for MSSQL
       */
      table.timestamp('created_at', { useTz: true })
      table.timestamp('updated_at', { useTz: true })
    })
  }

  public async down() {
    this.schema.dropTable(this.tableName)
  }
}

As you will notice, the migration filename is prefixed with some numeric value. We add the current timestamp to the filename so that the migration files are sorted in the order created.
By default, migration includes an id and timestamps column.
The rest of the fields are defined by the developer. If you look at the migration folder above, you can see that it's self-descriptive,

Up

The up method defines the record initialization logic. The first line instructs our migration file to create a table called stores within our database. The following are then methods that enable us to define the column types or attributes of our records.

table -> means we're initializing a database column
table.string('description') -> the column is to be of type string with a name, 'description`. There are other column types, including; boolean, enum, integer, and so on.
notNullable/nullable -> defines whether the field is allowed to have NULL values.
primary -> defines the column as the primary key of that table
unique -> ensures that the value of the database columns is all unique. We will implement this functionality in a later article.

Down

The down method is used to roll back the actions executed by the up method. For example, if the up method creates a table, the down method should drop the same table.

`tsx

public async down() {
this.schema.dropTable(this.tableName)
}

Relationships and foreign keys

As you may have noticed from our migration and the database schema diagram, our stores will be owned by a user. So how do we go about this in a migration file?

We can use a couple of approaches.

One is exactly as I've defined in my migration file.

`tsx

table.integer('owner').references('users.id').onDelete('CASCADE')

Here we are telling adonis that the column named, "owner" is an integer that references the user's table, specifically the id column, which we know is an integer. the onDelete() method enforces certain rules for when a user is deleted and they have an existing store. In our case, we go with CASCADE, meaning that when a user is deleted, then recursively delete all their stores. Other options include; PROTECT and SET_NULL.

Another approach would be;

`tsx

table.integer('owner').index()
table.foreign('owner').references('users.id').onDelete('CASCADE')

`tsx

table.integer('owner').references('id').inTable('users')

As you can tell, there's more than one to achieve an objective in Adonis. The approach you choose is entirely up to you.

For many-to-many relationships, we'll look into it once we start working with models in another article.

Running and Rolling Back migrations

Now that we have our up and down methods defined, we can go ahead and run our migrations by running the command on our terminal.

node ace migration:run

This command executes the up method in all migration files.

SQL statements for every migration file are wrapped inside a transaction. So if one statement fails, all other statements within the same file will rollback.

Also, in case of failure, the subsequent migrations will be aborted. However, the migrations before the failed migration stay in the completed state.

If the migration was successful, the command line GUI should not throw any error.

That's it! We have other migration operations which I will not cover in-depth as they have been comprehensively covered in the official docs.

Resources

Some of the references that I used to cover this article were acquired from the following sources.

The official AdonisJS documentation on schema migrations.
This awesome article from Jagr

Sponsors

*Disclosure: I only recommend products I would use myself and all recommendations here are my own. This post may contain affiliate links that at no additional cost to you, I may earn a small commission.

Scraper API is a startup specializing in strategies that'll ease the worry of your IP address from being blocked while web scraping. They utilize IP rotation so you can avoid detection. Boasting over 20 million IP addresses and unlimited bandwidth. Using Scraper API and a tool like 2captcha will give you an edge over other developers. The two can be used together to automate processes. Sign up on Scraper API and use this link to get a 10% discount on your first purchase.
Do you need a place to host your website or app, Digital ocean
is just the solution you need, sign up on digital ocean using this link and experience the best cloud service provider.
The journey to becoming a developer can be long and tormentous, luckily Pluralsight makes it easier to learn. They offer a wide range of courses, with top quality trainers, whom I can personally vouch for. Sign up using this link and get a 50% discount on your first course.

I hope you've had a better understanding of database migrations.
If you have any questions on this topic, feel free to leave a comment or get in touch directly on twitter

Adonis Js - Events and Mailing Part 2

Ted Ngeene — Tue, 05 Oct 2021 17:00:14 +0000

Curiosity keeps leading us down new paths. ~ Walt Disney

It is often said that in programming there's always more than one way of doing things. The maxim is that to grow we must be open to new ideas...new ways of doing things.

As developers, we always try to explore different ways of doing things.

In this second piece of implementing events and mailing in Adonis, I'll demonstrate another technique we can use to capture different events in our applications.

A reminder that the entire source code for this project can be found here.
Let's start.

Using the built-in Adonis event emitter module

For this guide, we'll use the built-in Event emitter module that comes in AdonisJs. You can peruse the official event documentation to have a better understanding of this concept.

We'll implement the same flow we did in our previous article, where the user receives a notification email upon registration to activate their account, so be sure to check it out!

What is the AdonisJs event module?

According to the adonis documentation, "The AdonisJS event emitter module is built on top of emittery".

Emittery is a modern async event emitter for node.js.

Some basic knowledge of the node.js event loop would help you better understand it but is not necessary.

Usage

node ace make:prldfile events

This command creates a new events.ts file in the contracts directory. This file is the entry point for all events in our application. Select all options prompted by the CLI.

import User from 'App/Models/User'

declare module '@ioc:Adonis/Core/Event' {
  interface EventsList {
    'new:user': { newUser: User }
  }
}

The Event.on method registers a new event listener. It accepts the name of the event, in our case new:user, followed by a method to handle the events as the arguments.

Listener Classes

Listener classes define the logic for our events. This is similar to the model function we defined in the last post.
Conventionally event listeners are stored inside the app/Listeners directory. However, you can customize the namespace inside the .adonisrc.json file.

To make a new event listener class that will handle the emailing, run the following command.

node ace make:listener User

A new User.ts file will be created under the app/Listeners/ directory. Open the newly created file and paste this code.

import Mail from '@ioc:Adonis/Addons/Mail'
import Env from '@ioc:Adonis/Core/Env'
import { EventsList } from '@ioc:Adonis/Core/Event'
import Route from '@ioc:Adonis/Core/Route'

export default class User {
  public async onNewUser({ newUser }: EventsList['new:user']) {
    const appDomain = Env.get('APP_URL')
    const appName = Env.get('APP_NAME')
    const defaultFromEmail = Env.get('DEFAULT_FROM_EMAIL')
    const currentYear = new Date().getFullYear()
    const url = Route.builder()
      .params({ email: newUser.email })
      .prefixUrl(appDomain)
      .makeSigned('verifyEmail', { expiresIn: '24hours' })
    await Mail.send((message) => {
      message
        .from(defaultFromEmail)
        .to(newUser.email)
        .subject('Please verify your email')
        .htmlView('emails/auth/verify', { user: newUser, url, appName, appDomain, currentYear })
    })
  }
}

As you can see, the code above is very similar to what we'd defined earlier as the sendVerificationEmail() function. For a more detailed explanation, head over to that article and check out the description.

However, just as a recap, we're defining the mail sending capability and building a URL that will encode our user token. The token expires in 24 hours and is tied down to a named URL, verifyEmail. Now, onto the new stuff.

public async onNewUser({ newUser }: EventsList['new:user'])

We're defining an async function named, onNewUser inside the default User class that takes the newUser as an argument. The newUser argument is tied to the event we just defined before. There, it'll always ensure that the parameters passed will match the ones defined in the event declaration.

If you wish to pass more than one argument, you can always define them in the events.ts file by separating them with semicolons.


 'new:user': { newUser: User; <arg2>: <modelName; <arg3>: <modelName;... }

Then calling the same arguments on the function declaration

public async onNewUser({
newUser,
arg2,
arg3,
....
})

Finally, we can emit our event on the authController.

Import the events module

import Event from '@ioc:Adonis/Core/Event'

Then right below the validation in the register function


const data = await request.validate({ schema: validations })
const newUser = await User.create(data)

Event.emit('new:user', {
  newUser,
})

We're calling the new:user event we defined in the events.ts file as this is the event name, the second parameter is the arguments to take.

Note that the argument names must be named similarly to the arguments we passed to the event declaration. By this I mean that you have to name the variable newUser just as it's declared in the event declaration. The same follows when having multiple arguments.

Now, we'll test the event.

Testing

Remember to turn on the development server by running, node ace serve --watch`

Registration

Email

Account activated

Resources

The complete source code can be accessed on my Github profile, here.
Frontend code for the email template can be found in this GitHub gist
official adonis events documentation
Node.js event loop.

Conclusion

In this article, we've learned another way of working with events in AdonisJs. It's entirely up to you to decide which method best works for you. That being said, always employ the most optimal approach.
If you have any queries, comment and insight, don't hesitate to reach out on my Twitter, personal website or simply by leaving a comment below.

I'll be going over models, migrations, and relationships in my next article. Till then...cheers!

AdonisJs - Events and Mailing Part 1

Ted Ngeene — Fri, 24 Sep 2021 20:21:11 +0000

Welcome back to the fourth installment of the AdonisJs series!

A web framework's ability to handle events and mailing greatly add to its appeal. Luckily, Adonis provides these in an easily configurable way that will set you on your path to building robust APIs and web apps.

In this article, I'll show just how we go about these two crucial concepts in modern software development. In order to achieve our goal, we'll build upon the previous article, where we talked about user registration and login.

If you're not familiar with the concept of events in software development, worry not.

Events can be described as actions that are triggered by another action in a system, for example, you'd want your users to be emailed upon successful registration. In this case, the email sending is an event triggered by successful user registration.

What we'll be working on

For this tutorial, we're going to implement an event that emails our users an activation link for their accounts. The purpose of this is to enforce security in our application.
We definitely wouldn't have bots and fictitious accounts registering in our application.

We'll look at two approaches we can employ to achieve this, but before we do that, let's set up the mailer.

Installing Adonis Mailing package.

Since we'll need users to receive emails, we need a way to make our system be able to send them. Luckily, Adonis also has a mailer package for this. To install, run

 npm i @adonisjs/mail

As usual, we also need to configure the package preferences,

node ace invoke @adonisjs/mail

For my configuration, I'll be using SMTP

Open up the env.ts file and paste the following values.

SMTP_HOST: Env.schema.string({ format: 'host' }),
SMTP_PORT: Env.schema.number(),
SMTP_USERNAME: Env.schema.string(),
SMTP_PASSWORD: Env.schema.string(),
DEFAULT_FROM_EMAIL: Env.schema.string(),

These values are the environment variables for our SMTP configuration. I recommend validating them in the env.ts file.
The DEFAULT_FROM_EMAIL is the email that will appear as the sender from our application.
The main environment variables will reside in the .env file. We'll get there in a minute.

Setting up our SMTP service

The Simple Mail Transfer Protocol(SMTP) is an internet standard communication protocol for electronic mail transmission. Mail servers and other message transfer agents use SMTP to send and receive mail messages.

There are a number of SMTP providers; including, google, mailgun, SendGrid...and so on. However, since we are not building a production-ready application yet, I'll be using mailtrap.

Mailtrap is an email testing tool that Captures SMTP traffic from staging and dev environments. Simply put, it mocks real-world emails in a sandbox environment. Head over to their website, create an account, and navigate to the demo inbox section.

Next, open the .env file located in your project's root directory and paste these constants.

  SMTP_HOST=smtp.mailtrap.io
  SMTP_PORT=2525
  SMTP_USERNAME=<your_mailtrap_username>
  SMTP_PASSWORD=<your_mailtrap_password>
  DEFAULT_FROM_EMAIL=admin@fitit.com #varies

The username and password values are found on the mailtrap dashboard, under the integrations section.

Now that we have all that setup, we'll get straight right into actual coding.

Approach 1 - Using a function in a model

This approach will involve us having a function inside our user model that will handle email sending. We'll call this function in our authController, right after a successful registration.

Open the app/Models/User.ts. We'll import several packages at the top,

import Mail from '@ioc:Adonis/Addons/Mail'
import Env from '@ioc:Adonis/Core/Env'
import Route from '@ioc:Adonis/Core/Route'

Next, after all the model column definitions, we'll write a sendVerificationEmail function.


  public async sendVerificationEmail() {
    const appDomain = Env.get('APP_URL')
    const appName = Env.get('APP_NAME')
    const currentYear = new Date().getFullYear()
    const url = Route.builder()
      .params({ email: this.email })
      .prefixUrl(appDomain)
      .makeSigned('verifyEmail', { expiresIn: '24hours' })
    Mail.send((message) => {
      message
        .from(Env.get('DEFAULT_FROM_EMAIL')
        .to(this.email)
        .subject('Please verify your email')
        .htmlView('emails/auth/verify', { user: this, url, appName, appDomain, currentYear })
    })
  }

The code above is contains several building blocks that helps to achieve the email sending functionality.


    const appDomain = Env.get('APP_URL')
    const appName = Env.get('APP_NAME')

These are values which reside in our .env file. If you don't have them already, you can check out the env.example file of this project repo

My current values are

  APP_NAME=Fitit
  APP_URL=http://127.0.0.1:3333

 const currentYear = new Date().getFullYear()

we get the current year. We'll inject this as a variable in our html template for our email

const url = Route.builder()
      .params({ email: this.email })
      .prefixUrl(appDomain)
      .makeSigned('verifyEmail', { expiresIn: '24hours' })

We're using the Route package to encode our user data into an activation token. The makeSigned() function is tied to a verifyEmail route that we'll build in the next part. Finally, we set an expiry period for the token. For this case, I set the period to 24 hours. That way, if a user fails to verify their account within that duration, their account won't be activated.

Mail.send((message) => {
      message
        .from(Env.get('DEFAULT_FROM_EMAIL')
        .to(this.email)
        .subject('Please verify your email')
        .htmlView('emails/auth/verify', { user: this, url, appName, appDomain, currentYear })
    })

This section makes use of the adonis mailer package. We're giving instructions to the program to send an email to the created user's email. The email will appear as received from the value DEFAULT_FROM_EMAIL in our .env file.
The subject line will be, "Please verify your email".

The htmlView method contains the template our email will read from, that is, the html code that styles our verification email.
For this, create a folder named resources in the project root directory. The entry point for all Adonis html templates must be located within this folder.
Since we initialized this project as an API only, we'll install the package that enables us to have .edge templates

npm i @adonisjs/view

The first argument the function takes is the html template to read from.
Create a emails\auth\verify.edge file and paste the code from this github gist. The next argument it takes are the variables to be read in the html template. In our case,

{ user: this, url, appName, appDomain, currentYear }

Writing the Verify Email controller

We're going to write an email verification controller that will be responsible for validating and activating a user's account.

node ace make:controller users/EmailVerificationsController

Open the created file and copy this snippet.

import { HttpContextContract } from '@ioc:Adonis/Core/HttpContext'
import User from 'App/Models/User'
import { DateTime } from 'luxon'

export default class EmailVerificationsController {
  public async confirm({ response, request, params }: HttpContextContract) {
    if (request.hasValidSignature()) {
      const user = await User.findByOrFail('email', params.email)
      if (!user.isActivated) {
        user.email_verified_at = DateTime.local()
        user.isActivated = true
        user.save()
        return response.status(202).send({ message: 'Account verified and activated' })
      } else {
        return response.status(409).send({ message: 'Account was already verified' })
      }
    } else {
      return response.status(403).send({ error: { message: 'Invalid token' } })
    }
  }
}

Next, copy this into the users.ts file under the routes directory

Route.get('/verify-email/:email', 'users/EmailVerificationsController.confirm').as('verifyEmail')

We're mapping the controller to a route. In adonis, routes can be given custom names, by defining the named route to a function, as(). In our case, the route is called verifyEmail. If you're keen enough, you'll notice that we passed this named route as a parameter in the model function we defined in the User model.

    .makeSigned('verifyEmail', { expiresIn: '24hours' })
    })

From the email verification controller code, we can see we have the confirm() function.
This block contains the user activation logic, that is, immediately the link is hit, we verify the user and activate his account. It also contains constraints that check whether the URL pattern matches the valid the signature, hence the makeSigned() and hasValidSignatureSections()

Finally, we modify the AuthController to send the email after a successful registration. Right after we save our user in the database,

const user = await User.create(data)
    // send verification email
    user?.sendVerificationEmail()

    return response
      .status(201)
      .send({ success: 'Registration successful, check your email inbox for a verification email' })

Testing

For testing open up your postman and repeat the registration steps we used in the previous article. Create a new user, and head over to your mailtrap demo inbox.

If everything worked well, then the following images should be what you see.

There's a lot of content to cover in events. The method covered in this piece is just one way to go about it. To keep the article short, I've decided to cover the next approach in the next piece.

If you have any comments, queries, don't hesitate to leave a comment or email.
Until then, stay Adonis. It is the way!

How to bypass captcha with 2captcha API and Selenium using Javascript

Ted Ngeene — Sun, 12 Sep 2021 08:44:51 +0000

Spam is a big nightmare for website owners. With the dawn of bots, this challenge has never been more prominent. Completely Automated Public Turing tests to tell Computers and Humans Apart(or CAPTCHA as they are commonly known) were introduced to tackle this issue.

The squiggly lines and words of the past are less common nowadays and have been replaced by Google's version 2 of CAPTCHA, known as reCAPTCHA. However, bots are becoming more advanced and can bypass almost any captcha. With a little spare time and, a few resources, we can make a program that bypasses the all too annoying CAPTCHA. In this tutorial, we'll make use of the captcha bypass software, 2captcha.

Why are we doing this?

Okay, I know some of you might be wondering, why would we need to bypass captcha? and most importantly; is this even legal? No, it is not illegal to bypass 2captcha. Whatever we'll be building is within the confines of the law. Secondly, this article is intended for anyone;

Curious about how you can bypass captcha
Building web scrapers that would need to bypass a captcha

*Disclosure: I only recommend products I would use myself and all opinions expressed here are my own. This post may contain affiliate links that at no additional cost to you, I may earn a small commission.

What is 2captcha API and how does it work?

Note: They charge a small rate for every captcha solved, however, the fee is quite minimal.

Signup Process and supported languages

Supported captchas the software solves include;

Requirements and Setup

For this tutorial, we'll utilize the 2captcha API. You'll need to have a developer account to use. You can head over to this link and sign up
The source code we'll be using can be found here. Setup instructions have been linked in the project README file. Feel free to clone the repository and tinker around.

Let's dive in.

Automating with Selenium

Before we get carried away by CAPTCHA, we need to understand the process our program will follow. I'm going to use selenium and node.js. Selenium is a browser automation service that provides extensions that allow programs to emulate browser interactions. It supports an array of browsers; from chrome, firefox, safari, and so forth, by integrating with their web drivers.

For this tutorial, I'll be using chrome driver since chrome is my default browser. To setup selenium and chromedriver, run

 npm i selenium-webdriver
 npm i chromedriver

This will write to the package.json file initialized in your node application.

Installing 2Captcha Dependencies

Next, we'll need to install 2captcha's node.js package

npm i @infosimples/node_two_captcha

This package will do the heavy lifting in terms of interacting with 2Captcha API.

Finally, we'll install dotenv since we'll be having sensitive keys that we need stored in an environment variables file.

npm i dotenv

Project Code

Open an index.js file at the root of your project directory and paste the following code. We'll go over the code in-depth in the next section.

require("chromedriver");
require("dotenv").config();
const Client = require("@infosimples/node_two_captcha");
const { Builder, By, Key, until } = require("selenium-webdriver");

const client = new Client(process.env.CAPTCHA_API_KEY, {
  timeout: 60000,
  polling: 5000,
  throwErrors: false,
});

const initiateCaptchaRequest = async () => {
  console.log("solving captcha...");
  try {
    client
      .decodeRecaptchaV2({
        googlekey: process.env.GOOGLE_CAPTCHA_KEY,
        pageurl: process.env.WEBSITE_URL,
      })
      .then(function (response) {
        //   if captcha is solved, launch selenium driver.
        launchSelenium(response);
      });
  } finally {
    //   do something
  }
};

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

async function launchSelenium(response) {
  if (response) {
    console.log("Captcha Solved! Launching Browser instance...");
    let driver = await new Builder().forBrowser("chrome").build();
    // Navigate to Url
    await driver.get(process.env.WEBSITE_URL);

    await driver.findElement(By.id("name")).sendKeys("Ted");
    await driver.findElement(By.id("phone")).sendKeys("000000000");
    await driver.findElement(By.id("email")).sendKeys("tngeene@captcha.com");
    await driver.findElement(By.id("comment-content")).sendKeys("test comment");

    const gCaptchResponseInput = await driver.findElement(
      By.id("g-recaptcha-response")
    );
    await driver.executeScript(
      "arguments[0].setAttribute('style','type: text; visibility:visible;');",
      gCaptchResponseInput
    );

    await gCaptchResponseInput.sendKeys(`${response.text}`);

    await driver.executeScript(
      "arguments[0].setAttribute('style','display:none;');",
      gCaptchResponseInput
    );

    await driver.findElement(By.id("send-message")).click();

    // wait 8 seconds and close browser window
    await sleep(8000);

    driver.quit();
  } else {
    //   if no text return request time out message
    console.log("Request timed out.");
  }
}

(async function main() {
  const response = await initiateCaptchaRequest();
})();

Importing Packages

require("chromedriver");
require("dotenv").config();
const Client = require("@infosimples/node_two_captcha");
const { Builder, By, Key, until } = require("selenium-webdriver");

These lines are essentially referencing the packages we'll be using. We're telling the program that we'll require to use chrome driver, dotenv, the node_two_captcha, and selenium packages.

Interacting with imported packages

const client = new Client(process.env.CAPTCHA_API_KEY, {
  timeout: 60000,
  polling: 5000,
  throwErrors: false,
});

The first package we need to use is the node_two_captcha package. The first parameter of the TwoCaptchaClient constructor is your API key from 2Captcha. In our case above, we referenced an environment variable (CAPTCHA_API_KEY). More on this below. The other parameters are:

timeout: Time (milliseconds) to wait before giving up on waiting for a captcha solution.
polling: Time (milliseconds) between polls to 2captcha server. 2Captcha documentation suggests this time to be at least 5 seconds, or you might get blocked.
throwErrors : Whether the client should throw errors or just log the errors.

Environment variable.

An environment variable is a user-definable value that can affect the way running processes will behave on a computer. Environment variables are part of the environment in which a process runs. For example, a running process can query the value of the TEMP environment variable to discover a suitable location to store temporary files, or the HOME or USERPROFILE variable to find the directory structure owned by the user running the process.

View on Wikipedia>

 CAPTCHA_API_KEY=<api_key_from_2_captcha.com>
 WEBSITE_URL=<website_we'll_be_accessing_captcha>
 GOOGLE_CAPTCHA_KEY=<google_captcha_site_key>

Paste the api key value from 2captcha dashboard on the CAPTCHA_API_KEY field.

Captcha Solving Pseudocode

Now that we have our account setup, we'll do the actual capture bypassing, which brings us to the next part.

For this tutorial, i'll be bypassing the capture on this sample form

In order to achieve our objective, we'll need our program to follow these steps.

Launch a browser tab
Fill in the required fields in the form
Solve the captcha
Send the contact message with the solved captcha.

All this will be achieved automatically!! How cool is that?

Retrieving Dom elements

For the visible form fields, all we need to do is open up the dev tools and retrieve the individual ids of the form fields.

This section does just that.

   let driver = await new
    Builder().forBrowser("chrome").build();
    await driver.get(process.env.WEBSITE_URL);

    await driver.findElement(By.id("name")).sendKeys("Ted");
    await driver.findElement(By.id("phone")).sendKeys("000000000");
    await driver.findElement(By.id("email")).sendKeys("tngeene@captcha.com");
    await driver.findElement(By.id("comment-content")).sendKeys("test comment");

All 2captchas have a hidden text-area that autofills with the solved captcha code once you click the i'm not a robot check. Usually, this has the id set as g-recaptcha-response

Since selenium simulates browser human inputs, it'll be required to make the field visible. We achieve this with this code snippet

  const gCaptchResponseInput = await driver.findElement(
      By.id("g-recaptcha-response")
    );
    await driver.executeScript(
      "arguments[0].setAttribute('style','type: text; visibility:visible;');",
      gCaptchResponseInput
    );

  await gCaptchResponseInput.sendKeys(`${response.text}`);

  await driver.executeScript(
      "arguments[0].setAttribute('style','display:none;');",
      gCaptchResponseInput
    );
 await driver.findElement(By.id("send-message")).click();

Finally, we'll close the browser tab 8 seconds after the captcha has been solved.

// wait 8 seconds and close browser window
    await sleep(8000);

    driver.quit();

All the fore-mentioned functionality resides in the launchSelenium() function. We need to tie it all together with the 2captcha service.

From the index.js file, you can see we have a initiateCaptchaRequest() function.

const initiateCaptchaRequest = async () => {
  console.log("solving captcha...");
  try {
    client
      .decodeRecaptchaV2({
        googlekey: process.env.GOOGLE_CAPTCHA_KEY,
        pageurl: process.env.WEBSITE_URL,
      })
      .then(function (response) {
        //   if captcha is solved, launch selenium driver.
        launchSelenium(response);
      });
  } finally {
    //   do something
  }
};

We are calling the node_two_captcha client we'd initialized before.

The WEBSITE_URL is the webpage of our captcha form. Fill it in the .env file.

GOOGLE_CAPTCHA_KEY is a special identifier found in all web forms having captcha, we can retrieve it by also opening dev tools, and searching the data-sitekey keyword.

Retrieve the value and paste it in the .env, GOOGLE_CAPTCHA_KEY value. node_two_captcha sends this key under the hood to 2capthca API, which then returns an API response of the solved captcha.

Demo

Okay, now your application is set up! It may feel like a lot 😅 but we did a lot of installation. We will now test our application.

To do this, run

npm index.js

Conclusion

It’s always a dilemma, should websites have a better experience and have simple to bypass the CAPTCHA or should websites aggressively protect themselves from bots and have a bad user experience. The war between websites and bots is never over. Whatever verification method websites pull out, it’s just a matter of time when someone figures out how to bypass it. ~ Filip Vitas

In this guide, we were introduced to 2captcha API, selenium, and a few concepts in 2captcha. By the end of it, I hope you can apply the knowledge gained to build your own captcha bypass service. I mean, if bots can do it, then so should we! A few next steps would be to add a User interface to input our values. You can also look into using the 2captcha API using your preferred programming language and other tools such as puppeteer

Finally, if you liked the content and would like to use 2captcha, sign up with this link.

If you have any questions, you can always leave a comment below, or reach out on these channels;

Source code of demo project can be accessed here.

Use 2captcha responsibly.

Sponsors

Scraper API is a startup specializing in strategies that'll ease the worry of your IP address from being blocked while web scraping. They utilize IP rotation so you can avoid detection. Boasting over 20 million IP addresses and unlimited bandwidth. Using Scraper API and a tool like 2captcha will give you an edge over other developers. The two can be used together to automate processes. Sign up on Scraper API and use this link to get a 10% discount on your first purchase.
Do you need a place to host your website or app, Digital ocean
is just the solution you need, sign up on digital ocean using this link and experience the best cloud service provider.
The journey to becoming a developer can be long and tormentous, luckily Pluralsight makes it easier to learn. They offer a wide range of courses, with top quality trainers, whom I can personally vouch for. Sign up using this link and get a 50% discount on your first course.

AdonisJs - Understanding User Registration and Authentication

Ted Ngeene — Sun, 05 Sep 2021 19:47:01 +0000

In this third installment of the Everything, you need to know about AdonisJs series. we'll go over the basic setup of database models, using the user model. We'll also configure our registration and login controllers(authentication). Finally, I'll show you how to handle routing for endpoints.
This article will also briefly introduce you to basic lifecycle hooks in Adonis. Let's dive in.

Definitions

Authentication is the process of verifying who a user is, for example by making them enter a password.

If you're not familiar with the concept of database models, the following description succinctly defines it.

A database model is a type of data model that determines the logical structure of a database. It fundamentally determines in which manner data can be stored, organized and manipulated. The most popular example of a database model is the relational model, which uses a table-based format.

View on Wikipedia>

The model is essentially the data that will be manipulated in the system, it has attributes and relationships with other models.

Routes allow us to make HTTP requests to our application. The entry point for all Adonis routes is located in the start/routes.ts file. You can define all routes in this file or other files and import them into this file as we will do. For more detailed info on Adonis routes, head over to their official documentation.

HTTP methods

In most cases, you'll need your application to perform some business logic. This is where HTTP methods come in, these allow us to perform some actions on our models. Common HTTP methods include.

GET - Used to fetch data from a specified resource.
POST - Used to store new data or dispatch data to the server.
PUT/PATCH - Used to update existing data.
DELETE - Used to delete existing data.

Finally, controllers are files that all logic on the program that will be performed. A controller determines what response to send back to a user when a user makes a browser request. For example, we can have an authController that will handle all authentication logic.

Routes are tied down to controller functions. They are URL patterns that are tied down to a handler function, in this case, a function in a controller. Using the example above, we could have a login route mapping to a function in the auth controller.

From the above definitions, it's pretty clear that we're covering the MC in the MVC pattern, that is, the model and the controller.

Now we can actually get our hands dirty in setting up the user model.

Setting up the user model

A cool thing with Adonis is that it has a neat package called adonisjs/auth which handles authentication. It leverages the fully-fledged in-built authentication system of Adonis.
We'll begin by installing the package; npm i @adonisjs/auth

After successful installation, as earlier mentioned on the configuration of packages, we configure the package settings by running node ace configure @adonis/auth
This will lead the cli to prompt some questions. For my configuration, I followed the steps below.

If the configuration was successful, you'll notice that some new files will be added to the file tree of your application.

These are the user migrations and user model files. The package creates a basic user model which we can modify depending on the use case.
You'll also notice that for this particular configuration, since I decided to use API token guard, then a separate migration file for API tokens has also been created.

The configuration for the auth package is stored inside the config/auth.ts file. Inside this file, you can define one or more guards to authenticate users. If you'd like to use a combination of API tokens and sessions, for example, all you'd need to invoke the node ace configure @adonisjs/auth command and add the guard you'd like to use. This will then append to the config/auth.ts file.

Modifying the user migration

Every application's user model is different. The basic model provided makes a general assumption of the common user attributes of most systems, however, to modify it to our use-case, we need to open the database/migrations/....users.ts file. Don't mind the digits.

For our application, the user table will need to look like this




import BaseSchema from '@ioc:Adonis/Lucid/Schema'

export default class UsersSchema extends BaseSchema {
  protected tableName = 'users'

  public async up() {
    this.schema.createTable(this.tableName, (table) => {
      table.increments('id').primary()
      table.string('email', 255).notNullable()
      table.string('username', 255).notNullable().unique()
      table.string('avatar').nullable()
      table.dateTime('email_verified_at').nullable()
      table.boolean('is_activated').notNullable().defaultTo(false)
      table.string('password', 180).notNullable()
      table.string('remember_me_token').nullable()

      /**
       * Uses timestampz for PostgreSQL and DATETIME2 for MSSQL
       */
      table.timestamp('created_at', { useTz: true }).notNullable()
      table.timestamp('updated_at', { useTz: true }).notNullable()

      table.index(['id', 'username'])
    })
  }

  public async down() {
    this.schema.dropTable(this.tableName)
  }
}

From the configuration above, we can see which fields we'll need our users to have. Over and above their inputs, we'll require that users verify their accounts. This will prevent bots from using our system. The implementation for this will be covered in the next section.

We'll also need to index some fields, which adonis provides. All we have to do is indicate which fields we'd like to be indexed.
For those of you not familiar with the concept of database indexing, head over to this definition.

Finally, it's time to migrate the data



node ace migration:run

If you got a successful migration, you'll see this on the command line.

It's worth noting that you might experience an error in migration related to phc-argon not being installed as a dependency. If you come across this, install this library by running npm i phc-argon2, then try running the migration again.

Modify the user model

In most cases, we'll have separate model files for each table in our database. These model files describe the columns to lucid. They also contain relationship definitions, lifecycle hooks, computed properties, serialization behavior, and query scopes. We'll dig into this at a later time.

Under the app/models directory, open the User.ts. We'll adjust it to this format.




import { DateTime } from 'luxon'
import Hash from '@ioc:Adonis/Core/Hash'
import { column, beforeSave, BaseModel } from '@ioc:Adonis/Lucid/Orm'

export default class User extends BaseModel {
  @column({ isPrimary: true })
  public id: number

  @column()
  public email: string

  @column()
  public username: string

  @column()
  public avatar: string

  @column()
  public isActivated: boolean = false

  @column.dateTime()
  public email_verified_at: DateTime

  @column({ serializeAs: null })
  public password: string

  @column.dateTime({ autoCreate: true })
  public createdAt: DateTime

  @column.dateTime({ autoCreate: true, autoUpdate: true })
  public updatedAt: DateTime

  @beforeSave()
  public static async hashPassword(user: User) {
    if (user.$dirty.password) {
      user.password = await Hash.make(user.password)
    }
  }
}

The code above is quite self-explanatory; it defines all the fields that we'd need our user model to have. However, at this point, I'd like to mention on the last bit



@beforeSave()
  public static async hashPassword(user: User) {
    if (user.$dirty.password) {
      user.password = await Hash.make(user.password)
    }
  }

This is a brief introduction to adonis lifecycle hooks. What this hook does is essentially encrypt user passwords using a hashing algorithm. This operation is performed right before a user is saved into the database, hence the beforeSave() function. We wouldn't want to store user passwords as raw texts. You can perform other lifecycle operations using any of these hooks in adonis



beforeSave(), beforeCreate(), beforeUpdate(), beforeDestroy(), beforeFind(), afterFind(),beforeFetch(), afterFetch(), beforePaginate(), afterPaginate()

Creating our Auth Controller

For the next step, we will make a controller that will handle all user authentication. We do this by running



node ace make:controller Users/AuthController

You'll notice that a new directory has been created under the app/Controllers/Http.
Open up the AuthController file and paste the following code.




import { HttpContextContract } from '@ioc:Adonis/Core/HttpContext'
import User from 'App/Models/User'
import { rules, schema } from '@ioc:Adonis/Core/Validator'

export default class AuthController {
  public async register({ request, response }: HttpContextContract) {
    // validate email
    const validations = await schema.create({
      email: schema.string({}, [rules.email(), rules.unique({ table: 'users', column: 'email' })]),
      password: schema.string({}, [rules.confirmed()]),
      username: schema.string({}, [rules.unique({ table: 'users', column: 'username' })]),
    })
    const data = await request.validate({ schema: validations })
    const user = await User.create(data)
    return response.created(user)
  }

  //   login function
  public async login({ request, response, auth }: HttpContextContract) {
    const password = await request.input('password')
    const email = await request.input('email')

    try {
      const token = await auth.use('api').attempt(email, password, {
        expiresIn: '24hours',
      })
      return token.toJSON()
    } catch {
      return response
        .status(400)
        .send({ error: { message: 'User with provided credentials could not be found' } })
    }
  }

  //   logout function
  public async logout({ auth, response }: HttpContextContract) {
    await auth.logout()
    return response.status(200)
  }
}

So what does the code above do?

Registration

There are three functions within this controller;
The first one being the registration of users.

We have some validators that ensure that the data being input meets certain requirements, in our case, the email and username fields should be unique. The password field should also be entered twice, that is, have a password confirmation field that matches the password.
If the user input meets the set validations, then the system creates a record of the user in the database.

Login

The login functionality of our application will handle the authorization of users. We'll require that users enter an email and password. If the two match against a user in the database, then we return an API token that gives the user access to our system.
This token will validate all requests from the user and will only be valid for 24 hours.
In a case where the user enters the wrong credentials, the system will throw an error with an appropriate response message.

Logout

Finally, we will need users to also be able to log out when they need to. The logout() function helps us achieve this.

Defining User Routes

Next, navigate to the start directory and create a new directory called routes, under it make a file named users.ts . Therefore your start directory should be looking like start/routes/users.ts. Paste the following;




import Route from '@ioc:Adonis/Core/Route'

Route.group(() => {
  // registration logic
  Route.post('register', 'Users/AuthController.register').as('register')
  Route.post('login', 'Users/AuthController.login').as('login')
  Route.post('logout', 'Users/AuthController.logout').as('logout')
}).prefix('api/v1/users/')

The above defines the user-related URLs that our application will be having.

The prefix keyword means that all URLs within the Route group will be prepended with the api/v1/users pattern.

For now, all the routes use POST requests, but not to worry, we'll see how to use other HTTP methods in upcoming articles.

I'll take a dive into its functionality, but before then we need to inject the user routes into the entry point of all routes for our application. This is the start/routes.ts file.

Open the file and modify it such that its contents are like this;



import HealthCheck from '@ioc:Adonis/Core/HealthCheck'

import Route from '@ioc:Adonis/Core/Route'

import './routes/users.ts'

// check db connection

Route.get('health', async ({ response }) => {

  const report = await HealthCheck.getReport()

return report.healthy ? response.ok(report) : response.badRequest(report)

})

Testing

We will be using postman for testing, for my setup, I've made a collection and added a global URL variable called BASE_DEV_API_URL, which is basically, http://localhost:3333/api
Next, I've added the three requests we've just created above and tested them out. We'll cover different scenarios, using different inputs.

User registration

Successful registration

Unique fail for email and username

This error will occur when an email and username fail to meet the uniqueness validator.

Password Confirmation Missing

Successful Login

Wrong login credentials

Logout For the logout functionality, copy the bearer token of a logged in user and pass it as an authorization type of Bearer token under the authorization tab in Postman. Hit the http:/localhost:3333/api/v1/users/logout url. The result, if successful should be a status 200.

Conclusion.

Congratulations! You've made it to the end of the tutorial. I hope you're following along just fine. In this article, we've learnt to setup an authentication scheme in Adonis, gotten introduced to controllers and validators and finally, done some basic HTTP routing.

In case of any query, feel free to shoot a DM or comment on the post below.

All the source code of the above application can be found here

For the next part of the series, we'll cover Relationships, by setting up more models. See you on the next piece!

AdonisJs - Installation and Database Setup

Ted Ngeene — Mon, 16 Aug 2021 08:58:44 +0000

Now that you have all the requirements we need to be installed, let's set up the project.

If you're having any problems setting up or installing the dependencies, feel free to leave a comment or shoot me a DM.

What we'll be covering in this post.

Creating a new project
Adonis Project Structures
IDE Setup
Starting the Development Server
Database Setup

Create a new project

To initialize a new project, all we need is a simple command on the terminal.
Simply navigate to the directory you'd like the project to be located and type the following;

npm init adonis-ts-app fitit or yarn create adonis-ts-app fitit

Let's break down this command for context.

npm init - this is simply initializing the project. Since we're using npm as the package manager, we use npm.
adonis-ts-app - we're initializing the project as a typescript project. If you wish not to use typescript and stick to vanilla JavaScript, you can use the adonis new fitit. This would involve installing the adonis cli. For more details on this, head over to this link. However, I recommend using the typescript version.
fitit - this is the name of the application.

In general, the general structure of creating a new Adonis application is

npm init adonis-ts-app <project_name>

After you've typed the project initialization command, npm first installs the necessary adonis packages, which then prompts you to choose a preferred project structure.

Adonis Project structures

You can choose between one of the following project structures.

web project structure is ideal for creating classic server-rendered applications. We configure the support for sessions and also install the AdonisJS template engine(edge). Also, the support for serving static files like images is provided.
api project structure is ideal for creating an API server. This also configures the support for CORS.
slim project structure creates the smallest possible AdonisJS application and does not install any additional packages, except the framework core. This sort of allows for flexibility. It's up to the developer to build the application ground-up.

It's worth noting that apart from some minor differences, the overall project structure is almost similar.

For this project, we'll be using the api project structure. The reason for this is that I'm mostly going to be using Adonis as an API server, which I'll then hook to a vue.js frontend.

If you're using vs code, the command prompt will ask for additional configurations. These are prettier and eslint-related. I advise using both formatters.

After the project has been created, you'll get a message like this on the terminal.

Vs Code Setup

This section is intended for anyone using vs code as their IDE. If you use any other IDE, you can skip over to the next section.

Some extensions will make your adonis development easier in vs code. These are;

Prettier and eslint - As I'd mentioned earlier you need to have prettier and eslint installed. This will be set up during the installation process.
JavaScript and TypeScript Nightly
Adonis JS Snippets - provides insightful snippets.
Adonis Js Go to controller will come in handy when we start working with routes.

Starting the Development server

Navigate to the project directory and open it on your IDE now that the project has been created.
Run the following command to spin up your local development server

node ace serve --watch

Open up your browser, and visit localhost:3333. If you did everything right, then you should see a screen with hello world text. Congratulations on setting up your first Adonis project! 🎉🎊

From here, things start to get interesting, we'll set up the database but before we head there, you might be wondering what node ace serve --watch is.

ace is a command-line framework that is embedded within your app. It allows you to create project-specific commands and run them using node ace. We'll be using ace commands throughout the tutorial, each with it's own purpose. For a more detailed explanation on ace commands, head over to this article by @amanvirk1

For the above command, let's break down what it does.

The serve command starts the HTTP server and performs an in-memory compilation of TypeScript to JavaScript.
The -watch flag is meant to watch the file system for changes and restart the server automatically(hot reload).

Database Setup

Before we wind up, let's connect our application to a database.

I'll assume that you have some knowledge of setting up Postgres or SQL databases and already have one on your computer. If this is not the case, you can use sqlite, which will be a file created on your machine once we start defining the database configuration.

AdonisJS has first class support for SQL databases. The data layer of the framework is powered by Lucid(AdonisJs ORM) and the package must be installed separately.

Simply run npm i @adonisjs/lucid

Upon successful installation, we'll configure our database driver and create a database. For this tutorial, I'll be using postgres. So my configs will be postgres-related. However, if you're using SQL, then use relevant configuration.
If you choose sqlite, no configuration will be required.

Once done, run

node ace configure @adonisjs/lucid

node ace invoke @adonisjs/lucid

The ace configure and invoke commands executes the instructions Javascript file exposed by the package.

Since I decided to go with postgres, I'll copy the code for validating the environment variables to the env.ts file.

Environment variables are injected from outside. The env.ts file validates that they type match and that the app is always running with the correct set of configuration values.
Your env.ts file should now look like this.

import Env from '@ioc:Adonis/Core/Env'

export default Env.rules({
  HOST: Env.schema.string({ format: 'host' }),
  PORT: Env.schema.number(),
  APP_KEY: Env.schema.string(),
  APP_NAME: Env.schema.string(),
  PG_HOST: Env.schema.string({ format: 'host' }),
  PG_PORT: Env.schema.number(),
  PG_USER: Env.schema.string(),
  PG_PASSWORD: Env.schema.string.optional(),
  PG_DB_NAME: Env.schema.string(),
  NODE_ENV: Env.schema.enum(['development', 'production', 'testing'] as const),
})

The config/database.ts file holds all the configuration related to the database. For more details on this, check out the documentation.

Next, we'll configure our database on postgres.

You can name your database whatever name you wish.

Finally, go to your .env file and modify these values


DB_CONNECTION=pg
PG_HOST=localhost
PG_PORT=5432
PG_USER=postgres
PG_PASSWORD=<your_postgres_password>
PG_DB_NAME=<your_db_name>

Testing Database connection.

Adonis comes with a neat health checker that checks if the database connection is working.

Navigate to start/routes.ts and paste the following code.


import HealthCheck from '@ioc:Adonis/Core/HealthCheck'
import Route from '@ioc:Adonis/Core/Route'

// check db connection
Route.get('health', async ({ response }) => {
  const report = await HealthCheck.getReport()

  return report.healthy ? response.ok(report) : response.badRequest(report)
})

With your server still running, open a browser tab and type localhost:3333/health

If everything worked fine and your database is connected, it should display the following screen.

Closing Thoughts

Thank you for following along, if you like the content and would like to know more about Adonis Development, bookmark this series, head over to my personal website or follow me on Twitter. You can also leave a comment in case you need any clarification or would like to point out an addition.

For the next piece, I'll be covering database models and relationships.

You can also follow the series on my personal website.

Stay tuned!

AdonisJs - Introduction

Ted Ngeene — Thu, 12 Aug 2021 19:43:32 +0000

When we talk about modern software development, it's hard to overlook the role of JavaScript. With so many frameworks to choose from, choosing the 'perfect one is often a conundrum that many newbie developers and experienced ones face when they want to learn new skills. In this series, I will highlight why you should go with AdonisJs. By the end of the tutorial, I believe you will be able to handle most, if not all of the common tasks involved in any backend application.

All source code of the application we'll be building can be found here.

Table Of Content

What is AdonisJs?
Why Use AdonisJs?
Series Outline
Prerequisites
Communities and Resources in AdonisJs

What is AdonisJs?

Some of you are wondering, "what in the world is AdonisJs in the first place?". AdonisJs describes itself as, "a backend framework for Node.js." However, you can do both back-end and front-end development using the framework. It follows the Model View Controller(MVC) architecture, therefore, you can create fully functional web apps using a single framework. It has its own templating engine called edge. It was created by Aman Virk.

Why AdonisJs

A couple of weeks ago, I started using the framework. Being torn between express and Adonis, I finally swayed the Adonis way. The reason for this is;

It has a robust Object Relational Mapper(ORM), which provides first-class support for SQL databases, Query builder with active records, seeds, migrations, and Redis support. This to me was the biggest factor in choosing it as I didn't have to go through the extra step of manually configuring the ORM.
Typescript support outside the box🎊
Well-written and thorough documentation. You can build a web app by just using the documentation.
Multi driver auth support, which lets you choose between JWT, session, and opaque API tokens.
It is easy to set up and use.
It follows good design patterns on what should encompass a web framework.
Strong emphasis on web security
A growing community. I personally see Adonis cementing itself as a top backend framework in the next couple of years.

I could go on and on about why I like the framework, but for that, I'll link some communities at the end of the post for you to have a look at the discussions around it.

Outline

Before we make any development I'll highlight what we'll be building and the requirements you'll need to set up an Adonis project.

We're going to be making a sports apparel store, called FitIt. The goal of the series is to highlight a couple of concepts that are important in your journey to becoming a top-notch Adonis developer. The store will be multi-tenant, meaning users can own different stores and post their gear for that particular shop.

The Database model is illustrated below;

Topics

These are the features we'll be going over, which are essential to understand while working on any project.

Installation and setup(using typescript)
Login and registration
- Different authentication schemes
- Account activation
- social signup
Relationships
- one to many
- many to many
CRUD operations
- Pagination
- Filters
- File uploads
- Validators
- Success and error messages
- Slug system
Lifecycle hooks
Events in AdonisJs
Mailing
Database Seeders
Hook up to vue.js frontend.
Deployment.

Prerequisites

Node.js - AdonisJS is a Node.js framework and hence it requires Node.js to be installed on your computer. To be precise, we require at least the latest release of Node.js 14.
A package manager. I will be using npm However, you can use yarn.
A code editor. My go-to editor is Visual Studio Code.
Postman for API testing.

Communities and Resources

If you've made it this far, then I'm sure I've picked your interest in learning this framework. AdonisJs has a growing community, where you'll get assistance in your journey. For more info on the framework, you can check out;

You can also follow me on dev as well as check out my personal website where I'll be writing more on Adonis and full stack development.

Handy Git Commands and practices to improve your workflow

Ted Ngeene — Sat, 16 Jan 2021 07:57:07 +0000

I have been in active software development for the better part of the last year and a half. For the majority of that period, I've come to appreciate the power of git, especially since the onset of the pandemic, where I've been working remotely 90% of the time, the technology has been very crucial in the development of my career.
The aim of this post is to highlight some of the common git commands that I use on a daily, which I'm sure most of you reading this will apply at some point or do also apply. The post is also meant to deconstruct the notion that git is complicated. Hopefully, you'll gain new insights into it. Let's dive in...

This article is divided into four parts:

Definition of terms
Commands
Best Practices
Resources

*Disclosure: I only recommend products I would use myself and all opinions expressed here are my own. This post may contain affiliate links that at no additional cost to you, I may earn a small commission.

Definition of terms

Before we move to the commands, it's important to familiarise ourselves with common terms used in the git world.
Let's have a look at some of them.

Git - it is a distributed version control system for tracking any set of files, for coordination among developers. Basically put, it is like a savings program for your project. It enables tracking and logging the changes you make to your file or file sets over time, a version-control system gives you the power to review or even restore earlier versions by taking snapshots of every revision in your project. You can access these versions to compare and revise them when the need arises.
Git repository hosting service - these are third-party web applications that wrap and enhance a version control system. An example of popular ones are; Github, Bitbucket, Gitlab. Many people confuse these to be one and the same thing with git. However, they are completely different. So, are they much more closely connected than Java and JavaScript? Yes, git is the underlying technology that platforms such as Github use to manage repositories. For a more detailed explanation on this, check out this article.
Git branch - a branch represents an independent line of development on a project. For example, if you're working on a feature that involves creating an admin dashboard, you may choose to create a branch called dashboard in which you work on all the dashboard logic.
HEAD - When working with Git, only one branch can be checked out at a time - and this is what's called the "HEAD" branch. Often, this is also referred to as the "active" or "current" branch. Git makes note of this current branch in a file located inside the Git repository, in .git/HEAD. (This is an internal file, so it should not be manually manipulated!). By now you may have noticed that the tree is the symbolic representation in git 😄.

Commands

-git init - This command creates a new Git repository. It can be used to convert an existing, unversioned project to a Git repository or initialize a new, empty repository.
-git remote add origin {remote_url} - connects a repo to a remote git URL. An example is;
git remote add origin https://github.com/tngeene/Dadjokes-app.git

git remote -v - checks the remote URLs set on a particular repo. A repo can have more than one remote, say for example you're pushing the code to two platforms such as Github and Bitbucket
git remote remove origin or git remote rm origin - removes a remote
- alternatively, you can use git remote set-url origin {new.url.here} this command automatically removes the old remote and assigns the new one. In our example above, you might realize you've made a mistake and set the wrong origin, using the second approach of removing an origin, (I won't delve into the first one since it's pretty straightforward), you can run this command git remote set-url origin https://github.com/tngeene/Bakery-template.git and this will remove the previously specified url and set the new one.
git rm -r --cached . - this command comes in handy when you want to untrack files, say you realized you would like to add it to the .gitignore file when it was already tracked.
- rm : remove command.
- -r : allows recursive removal.
- — cached : will only remove files from the index. Your files will still be there.
- . : untracks all files. You can untrack a specific file by git rm -r --cached {path_to_ file}
git clone - useful for getting the contents of a remote repo to your local machine. Usually used when you don't initially have the code. Example usage is git clone {url_of_repo}, for example git clone https://github.com/tngeene/fibonacci-sequence-react.git
git fetch - really only downloads new data from a remote repository - but it doesn't integrate any of this new data into your working files. Fetch is great for getting a fresh view of all the things that happened in a remote repository.
git pull - in contrast, is used with a different goal in mind: to update your current HEAD branch with the latest changes from the remote server. This means that pull not only downloads new data; it also directly integrates it into your current working copy files. This has a couple of consequences:
- Since git pull tries to merge remote changes with your local ones, a so-called "merge conflict" can occur.

Sytntax to use

git pull {remote_branch_name} {feature_branch_name} - for example git pull origin master

One can use git pull —set-upstream {remote_branch_name} {feature_branch_name}. This syntax avoids specifying the remote branch to update every time, for example, git pull —set-upstream origin master will allow you to just type git pull once you've checked out in the master branch.

git push - pushing the code that's in your local machine to a remote branch. The syntax is the same as the git pull command highlighted above.

A thing to note about this is that for an initial repo, especially if it was initialized with a README file, it might bring in errors such as the remote having contents you do not have. Another one could be that the repo has unrelated histories. So before you push code make sure to first pull from the remote by specifying git pull origin master —allow-unrelated-histories

git branch - allows one to create a new branch. The syntax for this is, git branch -f {branch} {start_point}.

git branch - f updates master

git checkout The git checkout command lets you navigate between the branches created by the git branch command. For example, if we would like to navigate to the updates branch created above, we would simply run git checkout updates. Something to note when we're checking out is to make sure we've committed the changes we were working on in the branch we're checking out from.
- Using the git checkout command, one can also create a new branch and checkout to it immediately, all using one command! So how do we go about this, you might ask. Well, simply type git checkout -b {branch_name} while in the current working branch. In our case above, git checkout -b updates. This will automatically merge all the existing changes in your current branch into the newly created branch.
git add - now that you've finished working on all the cool stuff you've been working on, you'll want to push them to the remote origin we set with the git add remote origin command. The git add command is part one of that process. This command adds a change in the working directory to the staging area. It tells Git that you want to include updates to a particular file in the next commit. There are two approaches to adding files that you want to push to the remote.
- git add . This command adds all the changes you've made to be included when you push them to the remote. The . at the end signifies that you want to stage all the files.
- git add {path_to_file_you_want_to_stage} - this command lets you specify which updates you'd like to be staged for commit. For example one might only want to update the settings file, you would only need to type git add ./settings.py
git commit - finally that you've added all your changes you will want to save them to be included once you decide to push to the remote repo. This command saves the changes to your local machine.
- The syntax for this commit involves including a message to describe what changes you've been working on, for example after working on a bug that fixes the login logic on a form, one might choose to commit it using this git commit -m "fixed email validation on login API" The -m flag denotes message

   It is important to write informative commit messages so 
   those other people who are working on the branch know 
   exactly what you were working on(just as it's important to 
   use descriptive variable names 😉).

git merge - after working on all the changes in your newly checked out branch, you might want to merge all the changes to the main branch before pushing them to the remote. After making a commit on the feature branch, run the command git checkout {branch_you_want_to_checkout_to} and then git merge {feature_branch_name}. In our example above, if we want to merge changes we've made into the main branch, we would go through these steps

        git checkout main
        git merge updates

git log - this command allows you to view information about previous commits. it simply shows the commits that lead up to the current state of the current working branch.

Reverting to a previous commit.

Often times you may realize that you made a mistake and would like to roll back to a more stable version of the project. Luckily, git offers solutions to this problem. We can choose to either use git revert or git reset commands. For the purposes of this guide, I'll go with git revert as I have more experience using it. However, for in-depth analysis and comparison of both commands, you can have a look at this article from atlassian.
The first step in rolling back is to identify at which point you'd like to go back. For this, we'll use our friend, git log

Notice the ordering of the commit history, with the "HEAD" being the first one on the list. If we want to go back to the commit with the message "reverted to ssr", we would run

git revert --no-commit 4eab63b2..HEAD
git commit

This will revert everything from the HEAD back to the commit hash, meaning it will recreate that commit state in the working tree as if every commit after 4eab63b2 had been walked back. You can then commit the current tree, and it will create a brand new commit essentially equivalent to the commit you "reverted" to.
The --no-commit flag lets git revert all the commits at once.

Note that we've used the commit hash of the commit above the one we'd like to roll back to.

There is of course more than one way to revert, but this is what works for me. Feel free to add to the discussion on best practices to revert in the comments.

Best Practices

While working with git, it vital to follow some practices, especially working with a team. The practices I'll mention are my personal preference and totally biased towards my own views.

Use Descriptive commit messages

It's important to let other developers working on the project on what exactly you've been working on. writing a commit message such as "fixed loader on landing page" instead of "landing page update" goes a long way in informing the code reviewers of what you've worked on.

Make Pull Requests

Once you've pushed your changes to Github if on a feature branch, remember to make a pull request(merge request in GitLab). This will notify project owners of a request to integrate your changes into the main branch and they can do a code review.

Write an informative pull request message.

Still, on the subject of pull requests, it's important to let code reviewers know why you'd like your code to be merged into the main branch. An example of a pull request message would go along the lines of;

This commit contains updates on the user registration logic
1. Fixed signup form validation by adding error handling
2. Added vee-validate on login and registration forms
3. Updated redirection on successful signup
4. Integrated social auth as a signup option (using google and Facebook)

Keep Commit Revisions minimal

As a minimalist, I believe that it's best to work on bits of the code and push them instead of making revisions on many files and pushing them at once. My case for this argument is that it provides for good tracking in case things break at some point, we can just roll back to the commit and undo the minor mistake made on a couple of files instead of multiple files, which can prove to be a nightmare.

That covers the end of this article. There's a ton of information on git, but for now, we'll only dive into the shallow end that is the very deep pool of git. I hope this piece helps someone and feel free to drop your thoughts. You can also check out my website or follow me on Twitter.

Include a `gitignore` file

When working on a project, there are some directories and files you'd prefer git not to track as well as some files which are not recommended to be tracked, an example being the node_modules files in node.js. This aids in reducing boilerplate code since the files are still going to be recreated in the other developer's machine. I found this interesting GitHub repo containing a list of popular .gitignore templates for most frameworks and languages.

Resources

This section highlights some useful resources to get you started with git. It includes videos, blog posts, and podcasts.

Freecodecamp's Git and GitHub youtube course
A collection of gitignore templates
Ladybug podcast episode on git and github
Devmountain's explanation on the difference between git and github
The official git documentation
Edureka's youtube git course

Sponsors

Do you need a place to host your website or app, Digital ocean is just the solution you need, sign up on digital ocean using this link and experience the best cloud service provider.
The journey to becoming a developer can be long and tormentuos, luckily Pluralsight makes it easier to learn. They offer a wide range of courses, with top quality trainers, whom I can personally vouch for. Sign up using this link and get a 50% discount on your first course.
Scraper API is a startup specializing in strategies that'll ease the worry of your IP address from being blocked while web scraping. They utilize IP rotation so you can avoid detection. Boasting over 20 million IP addresses and unlimited bandwidth. Using Scraper API and a tool like 2captcha will give you an edge over other developers. The two can be used together to automate processes. Sign up on Scraper API and use this link to get a 10% discount on your first purchase.

Thanks for going through the article. I hope in one way or another, you have a better understanding of git. Cheers!

What are your thoughts on strapi.js?

Ted Ngeene — Tue, 15 Dec 2020 04:06:07 +0000

I recently came across strapi.js. They describe themselves as, "the next-gen headless CMS, open-source, javascript, enabling content-rich experiences to be created, managed and exposed to any digital device."
So I went ahead and bootstrapped a simple blog application. From the looks of things, it seems like a pretty decent resource.
To anyone that has used Strapi before, would you recommend it? which customizations does it offer? features like extending the user models, authentication, scalability, ease of hosting? How does it fair security-wise? What tips would you give to a beginner learning the CMS?