Forem: Nitesh Thapliyal

Key Management in AWS: Exploring CloudHSM

Nitesh Thapliyal — Thu, 11 Jan 2024 18:46:40 +0000

In the ever-evolving landscape of cloud computing, securing sensitive data is paramount. As organizations increasingly migrate their workloads to the cloud, effective key management becomes a critical aspect of their overall security strategy. AWS, as a leading cloud services provider, offers two prominent solutions for key management: AWS CloudHSM and AWS Key Management Service (KMS).

In this blog we will explore CloudHSM

Before we deep dive into services, we should first understand about Cryptography

Cryptography

Cryptography is the practice of techniques for securing communication and data from adversaries. It involves the use of mathematical algorithms to transform information into a format that is unintelligible to unauthorized users, ensuring confidentiality, integrity, authentication, and non-repudiation

Key management is one of the concepts used in Key management is one of the concepts used in Cryptography

Key management

Key management refers to the process of generating, storing, distributing, and disposing of cryptographic keys in a secure and organized manner. Cryptographic keys are essential components in the field of cryptography, used to encrypt and decrypt information, authenticate users, and ensure the integrity of data.

The cryptographic keys are fundamental to ensuring the security of communication, protecting sensitive data, and maintaining the integrity of information. There are different types of cryptographic keys, and their roles can be broadly categorized into two main types: symmetric keys and asymmetric keys.

Symmetric Key Encryption

Symmetric-key cryptography is a term used for cryptographic algorithms that use the same key for encryption and for decryption.

The commonly used algorithms that are used for symmetric key encryption are:

AES (Advanced Encryption Standard)
DES (Data Encryption Standard)

(Source: The SSL Store)

Asymmetric Key Encryption / Public key Encryption

It is a cryptographic system in which keys come in pairs. The transformation performed by one of the keys can only be undone with the other key. One key (the private key) is kept secret while the other is made public.

The commonly used algorithms that are used for asymmetric key encryption are:

RSA(Rivest–Shamir–Adleman)
DSA (Digital Signature Algorithm)

(Source: appViewX)

If we store the cryptographic keys in our system it can be a security issue therefore the Hardware Security Modules (HSMs) are used for storing cryptographic keys due to their specialized design and features that enhance the security and protection of sensitive key material.

Hardware Security Module(HSM)

A Hardware Security Module is a specialized, highly trusted physical device that performs all major cryptographic operations, including encryption, decryption, authentication, key management, key exchange, and more. HSMs are specialized security devices, with the sole objective of hiding and protecting cryptographic materials

Types of HSMs

There are two main types of Hardware Security Module:

General Purpose

General Purpose HSMs can utilize the most common encryption algorithms and are primarily used with Public Key Infrastructures, cryptowallets, and other basic sensitive data.

Payment and Transaction

These types of HSM are created with the protection of payment card information and other types of sensitive transaction information. These types of Hardware Security modules are narrower in the types of organizations they can work within, but they are ideal to help comply with Payment Card Industry Data Security Standards (PCI DSS).

The main reason to use HSMs is that it is Tamper-resistant, tamper-evident, and tamper-proof systems to provide extremely secure physical systems.

(Source: cpl.thalesgroup.com)

We need our key to keep somewhere and manage somewhere and most of the industry uses HSM but it can be very costly to use, to overcome this problem AWS launched a service called CloudHSM where you can use a pay-as-you-go model, allowing you to pay for the resources you consume. This can be cost-effective for organizations with dynamic or unpredictable workloads.

CloudHSM

AWS CloudHSM provides cloud-based hardware security modules (HSMs) for generating and using your own encryption keys in the AWS Cloud. With CloudHSM, you can manage your own encryption keys using FIPS 140-2 Level 3 validated HSMs and integrate with your applications using industry-standard APIs.For more detail

*Let's create a CloudHSM cluster: *

Go to aws CloudHSM service and click on create cluster

Select the VPC and Availability zones based on how many devices you want and then click next

Add the backup retention period

After this time period, the cluster backup will be deleted

Click on Create cluster

After a few minutes cluster will be created

Here you can see that the cluster is uninitialized which means we need to add HSM appliance to it

Click on the cluster and then click on Initialize

select the AZ to create the HSM and click on create

Now you can see one HSM is added to the cluster

Now download the Certificate signing request

Download the CSR and sign it.

To sign the CSR follow the steps:

Generate the Private key

Use the command openssl genrsa -aes256 -out customerCA.key 2048 to generate the private key

Note Remember the pass phrase it will be use in next step

To sign the CSR use the command

openssl req -new -x509 -days 3652 -key customerCA.key -out customerCA.crt

Here customerCA.crt is a certificate that you downloaded

fill the details to sign the certificate

use the following command :

openssl x509 -req -days 3652 -in <cluster ID>_ClusterCsr.csr \ -CA customerCA.crt \ -CAkey customerCA.key \ -CAcreateserial \ -out <cluster ID>_CustomerHsmCertificate.crt

Now upload the certificates

*Note: * The cluster certificate will be the one with cluster id

Now you can see that your cluster is initiated

Now set the password for the cluster and activate it

use the following aws documentation to activate cluster

After following the above doc you will see that now your cluster is activated

That's how you can create a CloudHSM cluster and add HSM device to it and now you can start using it to store your keys.

Hope you find this blog insightful 🌟

Thank you!

Simplify your Infrastructure Management using AWS CloudFormation

Nitesh Thapliyal — Sat, 21 Oct 2023 13:55:07 +0000

In this blog, we'll explore what AWS CloudFormation is, why it's essential, and dive into a practical demonstration to showcase its capabilities.

From understanding the basics to creating your first CloudFormation stack, you'll gain valuable insights and hands-on experience, enabling you to automate your AWS infrastructure management effectively.

Let's explore the AWS CloudFormation and discover how to harness the power of Infrastructure as Code for your projects.

What is AWS CloudFormation?

It is an AWS service that allows you to define and provision AWS infrastructure as code using a template.

The template can be created either using JSON or YAML

When do we use AWS CloudFormation?

When you want to manage and provision your AWS infrastructure in a consistent and repeatable manner.

It allows you to automate the use of AWS services by defining infrastructure as a code.

This approach is particularly valuable in a DevOps environment where you need to create a large infrastructure and you prefer automation over manual things.

You can use AWS CloudFormation in the following scenarios:

Infrastructure as Code
When you want to manage your infrastructure in AWS in a repeatable manner
Templated Deployment
When you want to create, update, or delete a stack of AWS resources by defining a CloudFormation template
Complex Deployment
For managing multiple AWS resources such as EC2 instances, RDS databases, S3 buckets, and more. CloudFormation helps ensure that all resources are created, configured, and connected correctly.
Stack updates
CloudFormation handles updates by creating a change set, which helps prevent disruptions to your existing resources.

Let's understand practically how we can utilize CloudFormation...

For the demo purpose, I'm going to launch two EC2 instances by creating the CloudFormation Template.

Follow the steps for better understanding:

1. Open CloudFormation service in AWS

2. Create a CloudFormation Template file

For creating the template I'm using YAML you can use JSON also

Resources:
  NiteshOsUsingCF:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-099b3d23e336c2e83
      InstanceType: t2.micro
      AvailabilityZone: ap-south-1a

  NiteshOs2:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-099b3d23e336c2e83
      InstanceType: t2.micro
      AvailabilityZone: ap-south-1a

here you can see I'm defining that I want to launch 2 EC2 instances with instance type t2.micro and availability zone as ap-south-1a

You can get the ImageId from the AWS AMI catalog, select the image of your choice, and copy its AMI ID

Note - I'm defining EC2 instances for demo purposes you can define any AWS services like S3, EBS, etc... and their syntax you can check from the docs

To know more about how to create template and syntax you can check the aws doc here

3. Upload Template to CloudFormation

Make sure your YAML or JSON formatting is correct and the file name is with extension .yml or .json

4. Give a name to your stack

Like here I've given my name as NiteshStack

5. Configure Stack

Configure the stack according to your need

6. Review your stack and click submit

After you review your stack configuration click the submit button

Now you can see the resource creation process is initiated

After a few minutes, you can see the resources are created

Here we can see that new EC2 instances at ap-south-1a are created

If you want to delete your stack you can delete it

Once you delete the stack your resources will be deleted and in our case, the EC2 instances will be deleted

That's how you can utilize AWS CloudFormation service to simplify the Infrastructure Management

Hope you find this blog insightful, Do give it a like 🌟

Thank you!!

EC2 101: Understanding the Basics of Amazon's Elastic Compute Cloud

Nitesh Thapliyal — Wed, 11 Jan 2023 08:27:57 +0000

Learn how to use AWS EC2 efficiently with this blog post. In this post, we'll cover connecting to your instance using the AWS Management Console or using ssh which allows you to connect directly to your instance through its private IP address

What is EC2(Elastic Compute Cloud)?

AWS EC2 is a cloud service that allows you to run virtual servers in Amazon's elastic infrastructure. It's a powerful way to create your own server infrastructure without any need for configuration or management. With AWS EC2, you'll have access to fast and reliable computing power, backed with the service's 99.999999% uptime guarantee.

AWS EC2 is a cloud computing service for running applications that can be accessed from anywhere with internet access.

Features

Elastic

EC2 allows users to quickly and easily scale the number of virtual machines (known as "instances") they are using up or down, depending on the needs of their application.

Pay-as-you-go

Users are only charged for the resources they actually consume, with a granular per-second billing.

Customizable

EC2 instances can be customized with different combinations of CPU, memory, storage, and networking resources, allowing users to choose the configuration that best meets the needs of their application.

Secure

EC2 instances can be launched in a virtual network that the user defines, and security groups can be used to control inbound and outbound network traffic.

Support for multiple operating systems

including Amazon Linux, Ubuntu, Windows Server, and many others

Integration with other AWS services

EC2 instances can easily be configured to work with other AWS services, such as Amazon RDS for database services, Elastic Block Store (EBS) for persistent storage, and Elastic Load Balancing for distributing incoming traffic among multiple instances.

Auto Scaling and Load balancing

They can automatically increase or decrease the number of instances in response to changes in demand for the application, and automatically distribute incoming traffic among multiple instances.

Spot Instance

an option which allows to bid for spare EC2 instances capacity.

Components of EC2 service

There are several components that are used when launching an Amazon Elastic Compute Cloud (EC2) service:

Virtual private cloud (VPC): A VPC is a virtual network that allows you to launch AWS resources in a logically isolated area.
Subnet: EC2 instances are launched within a subnet, which is a range of IP addresses within a VPC.
Security groups: Security groups are used to control inbound and outbound network traffic to and from an EC2 instance.
Key pair: A key pair is used to securely connect to an EC2 instance using SSH.
Elastic IP: Elastic IP address is a static public IP address, that can be allocated, associated, and disassociated to instances as required.
Block Storage: It allows to create storage volumes that can be attached to instances as additional storage or it can be used to create snapshots for backups
Instance: An EC2 instance is a virtual server that runs on Amazon's infrastructure. The instance type determines the CPU, memory, storage, and network resources that are made available to the instance.
Images: They are pre-configured machine images that include the operating system, software, and any necessary configuration settings. These images can be used to launch new instances.
Auto Scaling group: This group allows you to automatically scale the number of instances in response to changes in demand for your application.
Load balancer: A load balancer distributes incoming traffic among multiple instances, helping to ensure that your application is highly available and can handle a large amount of traffic.
Elastic Container Service(ECS) or Elastic Kubernetes Service(EKS) : Used for launching containerized applications.

Use Case of EC2 service

Hosting web applications: You can use EC2 instances to host web applications, by installing a web server (such as Apache or Nginx) and deploying your application code on the instances.
Running batch jobs: You can use EC2 instances to run batch jobs, such as data processing or image rendering, by creating custom AMIs (Amazon Machine Images) with the necessary software installed and then launching instances from those AMIs.
Building a test and development environment: You can use EC2 instances to build a test and development environment, by creating custom AMIs with the necessary software and configuration, and then launching instances from those AMIs.
Creating a Virtual Desktop Infrastructure (VDI): With EC2 you can create a virtual desktop infrastructure that allows users to access their desktops remotely.
Building a High Performance Computing(HPC) clusters: EC2 instances can also be used to build high performance computing clusters to perform complex calculations.
Running containerized applications: you can use EC2 service to launch containerized applications using Elastic Container Service (ECS) or Elastic Kubernetes Service (EKS)
Hosting databases: You can use EC2 instances to host databases, such as MySQL or PostgreSQL, either by installing the database software on the instances directly or by using a service like Amazon RDS.
Building a data pipeline: You can use EC2 instances to build a data pipeline, using tools such as Apache Kafka or Apache NiFi, to collect, process, and analyze large amounts of data.

Steps to launch EC2 Service

Choose the region:

Search for EC2 service:

Click on Launch instance:

Give name to your instance and choose an image:

Select instance type and choose key pair if created:

Configure your network rules:

Launch the instance:

Connect to the instance:

Connect using SSH

Now you can see our instance is connected

Hope this blog helped you to learn something about AWS EC2

Happy Learning✨

AWS Automation using Terraform Micro Byte

Nitesh Thapliyal — Sun, 01 Jan 2023 05:53:12 +0000

In this blog we are going to discuss about the micro byte on the topic AWS automation using Terraform, You might be thinking what is micro byte?

Micro Byte

Micro Bytes are short learning experiences where readers can learn a topic by doing the Micro-Byte. It teaches an important concept very clearly with learn-by-doing exercises i.e. learning by executing tasks in a hands-on way. It Roughly takes 1 hour of learning time for learners to go through it. It Has custom illustrations/imagery to explain concepts. It is Entertaining and engaging with a unique spin/angle to learn the topic.

In this micro byte we will learn How we can automate the aws cloud by automating it by launching resources with the help of terraform.

What is openCV?

OpenCV is a library that works with almost all programming languages therefor known as cross-platform library which is mainly used for Image Processing that is used in Computer Vision. OpenCV is used for image processing, Video analyses with feature like face detection and Object detection.

What is Terraform?

Terraform is an Infrastructure as code tool that let's us build, change and version cloud. It let's us define both cloud and on-prem resources in human readable configuration files that we can reuse and share.

What OpenCV and Terraform will do?

OpenCV will allow us to detect our face and as soon as our face is detected Terraform configuration file will get executed which will launch the AWS services.

Microbyte contains various Activities that will help to perform the task:

Activity-1

Create a Dataset of face images

Why we need to create Dataset?

We need to create a Dataset of face images to train the face recognition machine-learning model
Here we are going to use the haarcascade_frontalface model for face detection

To create the dataset we will make use of OpenCV, it will capture your face image 100 times and store it in a directory, which will be used by the model for face detection.

use the following code below and fill the required data as mentioned

import cv2
import numpy as np

# Load HAAR face classifier
face_classifier = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

# Load functions
def face_extractor(img):
    # Function detects faces and returns the cropped face
    # If no face detected, it returns the input image

    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    faces = face_classifier.detectMultiScale(gray, 1.3, 5)

    if faces is ():
        return None

    # Crop all faces found
    for (x,y,w,h) in faces:
        cropped_face = img[y:y+h, x:x+w]

    return cropped_face

# Initialize Webcam
cap = cv2.VideoCapture(0)
count = 0
# Collect 100 samples of your face from webcam input
while True:

    ret, frame = cap.read()
    if face_extractor(frame) is not None:
        count += 1
        face = cv2.resize(face_extractor(frame), (200, 200))
        face = cv2.cvtColor(face, cv2.COLOR_BGR2GRAY)

        # Save file in specified directory with unique name
        #path
        file_name_path = 'Path_of_dir/' + str(count) + '.jpg'
        cv2.imwrite(file_name_path, face)

        # Put count on images and display live count
        cv2.putText(face, str(count), (50, 50), cv2.FONT_HERSHEY_COMPLEX, 1, (0,255,0), 2)
        cv2.imshow('Face Cropper', face)

    else:
        print("Face not found")
        pass

    if cv2.waitKey(1) == 13 or count == 100: #13 is the Enter Key
        break

cap.release()


cv2.destroyAllWindows()

Add your directory path
Execute the file by pressing key ctrl + enter or the run button in jupyter notebook.

output:

Activity 2 - Train the Model

In this activity, we are going to train the face recognition model so that it can recognize our face as soon as the web camera turns on

To train the model we will use the dataset that we have created in Activity-1

Now to train the model:

Use the code below to your jupyter notebook and fill the required data as mentioned.

import cv2
import numpy as np
from os import listdir
from os.path import isfile, join

# Get the training data we previously made
data_path_1 = 'dataset_path'
onlyfiles_1 = [f for f in listdir(data_path_1) if isfile(join(data_path_1, f))]


# Create arrays for training data and labels
Training_Data_1, Labels_1 = [], []
# Create arrays for training data and labels

# Create a numpy array for training dataset 1
for i, files in enumerate(onlyfiles_1):
    image_path = data_path_1 + onlyfiles_1[i]
    images = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    Training_Data_1.append(np.asarray(images, dtype=np.uint8))
    Labels_1.append(i)   

# Create a numpy array for both training data and labels for person 1
Labels_1 = np.asarray(Labels_1, dtype=np.int32)

# Initialize facial recognizer
#model = cv2.face.createLBPHFaceRecognizer()
# NOTE: For OpenCV 3.0 use cv2.face.createLBPHFaceRecognizer()
# pip install opencv-contrib-python
#model = cv2.createLBPHFaceRecognizer()

Nitesh_model  = cv2.face_LBPHFaceRecognizer.create()
# Let's train our model 
Nitesh_model.train(np.asarray(Training_Data_1), np.asarray(Labels_1))
print("Model trained sucessefully")

Add your dataset path

Output:

Activity 3 - Get the details from AWS

Before we start creating the configuration file, we should know the following things:

Make sure Terraform is installed in your system, if not install it by following the step in Pre-Requisite
The region where you are going to launch the service, in this case, EC2 service
The AMI-id of the instance which you will be using

If you are not able to find the region name in AWS then:

refer to Images/Activity-3/region.png

For getting the AMI-id, follow the following steps:

Login to your AWS account
Go to the EC2 service
Click on Launch Instance button

refer to Images/Activity-3/launch_instance.png
Select the AMI that you want to work on and you will find the AMI-id just below the AMI

Now we have the required information, we can add these details to our terraform configuration file

Activity 4 - Create Terraform Configuration file

With the help of Terraform Configuration file, you can launch the resources in the cloud platform and also attach the resource like EBS volume to the EC2, with the help of it you can launch the resources in the multiple cloud platform at once like you can launch resources in AWS(Amazon Web Service), GCP(Google Cloud Platform), etc.

To create the Terraform file to launch the AWS resources: > Copy the code below and add the detail as mentioned

provider "aws" {
region = "<your_region_name>"
profile = "Nitesh"
}

# Launching instance
resource "aws_instance" "os1"{
ami="<your_ami_id>"
instance_type = "t2.micro"
security_groups = [ "<your_scurity_group_name>" ]
tags={
Name="<your_ami_instance_name>"
}
} 

# To retrieve the detail information about the instance
output "os1"{
value = aws_instance.os1
}

#To retrieve the public ip of instance
output "os2"{
value = aws_instance.os1.public_ip
}

#To retrieve the availability zone of an instance
output "os3"{
value = aws_instance.os1.availability_zone
}

# Launching EBS Volume
resource "aws_ebs_volume" "st1"{
availability_zone= aws_instance.os1.availability_zone
size = 5
tags = {
name = "<Name_your_ebs_volume>"
}
}

# To check about volume in detail
output "os4"{
value=aws_ebs_volume.st1
}

# Attach Volume to EC2 instance
resource "aws_volume_attachment" "ebs_att"{
device_name="/dev/sdh"
volume_id=aws_ebs_volume.st1.id
instance_id=aws_instance.os1.id
}

# To check in detail about volume attachment
output "os5"{
value=aws_volume_attachment.ebs_att
}

Add your region name
Add your ami-id
Add your instance name
Add your Security group name that you have created in Activity-3 Task
Add your EBS volume name

Note: We don't have to run the terraform file now because we need to integrate the terraform file with the face recognition program in the next activity

Activity 5 - Integrate Terraform file with the Face Recognition Program

In this activity, we are going to integrate the terraform configuration file with the face recognition program so that our terraform file gets executed as soon as our face gets recognized.

To perform the activity:

Copy the file from and paste it to the location where your terraform file is present

import cv2
import numpy as np
import os


face_classifier = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

def face_detector(img, size=0.5):

    # Convert image to grayscale
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    faces = face_classifier.detectMultiScale(gray, 1.3, 5)
    if faces is ():
        return img, []


    for (x,y,w,h) in faces:
        cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,255),2)
        roi = img[y:y+h, x:x+w]
        roi = cv2.resize(roi, (200, 200))
    return img, roi


# Open Webcam
cap = cv2.VideoCapture(0)

while True:

    ret, frame = cap.read()

    image, face = face_detector(frame)

    try:
        face = cv2.cvtColor(face, cv2.COLOR_BGR2GRAY)

        # Pass face to prediction model
        # "results" comprises of a tuple containing the label and the confidence value
        results = Nitesh_model.predict(face)

        if results[1] < 500:
            confidence = int( 100 * (1 - (results[1])/400) )
            display_string = str(confidence) + '% Confident it is User'

        cv2.putText(image, display_string, (100, 120), cv2.FONT_HERSHEY_COMPLEX, 1, (255,120,150), 2)

        if confidence > 70:
            cv2.putText(image, "Hello Nitesh", (250, 450), cv2.FONT_HERSHEY_COMPLEX, 1, (0,255,0), 2)
            cv2.imshow('Face Recognitioned', image )
            if cv2.waitKey(1)==13:
                cap.release()
                cv2.destroyAllWindows()
                !terraform init
                !terraform apply --auto-approve


        else:
            cv2.putText(image, "Unrecognised Face", (250, 450), cv2.FONT_HERSHEY_COMPLEX, 1, (0,0,255), 2)
            cv2.imshow('Face Recognition', image )

    except:
        cv2.putText(image, "Face Not Found", (220, 120) , cv2.FONT_HERSHEY_COMPLEX, 1, (0,0,255), 2)
        cv2.putText(image, "Searching for Face....", (250, 450), cv2.FONT_HERSHEY_COMPLEX, 1, (0,0,255), 2)
        cv2.imshow('Face Recognition', image )
        pass

    if cv2.waitKey(1) == 13: #13 is the Enter Key
        break

cap.release()
cv2.destroyAllWindows()

Execute the file by pressing ctrl + enter or run button in the jupyter notebook

As soon as the file gets executed you will see that your WebCam gets opened your face gets recognized as shown below

Now as soon as our face gets recognized, the terraform configuration file gets executed and you will see the output as below

aws_instance.os1: Refreshing state... [id=i-07c3d200a11ce50e0]
aws_ebs_volume.st1: Refreshing state... [id=vol-0e4577c488a197d1f]
aws_volume_attachment.ebs_att: Refreshing state... [id=vai-991301012]
aws_ebs_volume.st1: Destroying... [id=vol-0e4577c488a197d1f]
aws_ebs_volume.st1: Destruction complete after 0s
aws_instance.os1: Creating...
aws_instance.os1: Still creating... [10s elapsed]
aws_instance.os1: Still creating... [20s elapsed]
aws_instance.os1: Still creating... [30s elapsed]
aws_instance.os1: Creation complete after 33s [id=i-0d1e951772b6e049d]
aws_ebs_volume.st1: Creating...
aws_ebs_volume.st1: Still creating... [10s elapsed]
aws_ebs_volume.st1: Creation complete after 11s [id=vol-06c89d52fc685bcea]
aws_volume_attachment.ebs_att: Creating...
aws_volume_attachment.ebs_att: Still creating... [10s elapsed]
aws_volume_attachment.ebs_att: Still creating... [20s elapsed]
aws_volume_attachment.ebs_att: Creation complete after 21s [id=vai-977186061]

Apply complete! Resources: 3 added, 0 changed, 1 destroyed.

Now you can check in the AWS that the resources that we have mentioned in the terraform configuration file are launched successfully

Instance launched successfully

EBS volume of 5 GB as mentioned launched successfully

To learn more about it then do checkout the Micro Byte

Happy Learning✨

Reverse Email Lookup Using Proxycurl

Nitesh Thapliyal — Thu, 15 Dec 2022 16:48:41 +0000

Ever wonder what information you can get from the email of a person?

The information that you can get only from the Email address are:

Email address owner information
Email address owner Country
Job profile and associated social media account linked with the Email address
You can also get the phone number, permanent address and profile picture of the Email address owner

Now you might be thinking HOW?

Using Reverse Email Lookup

Reverse Email Lookup:

Reverse email lookup is the process of finding the identity of an email sender by using their email address. This can be used to confirm the identity of an email sender or to find out more information about them.

Use case of Reverse Email Lookup:

Scam and Fraud Prevention
Verification of Ecommerce Transactions
Credit Risks Analysis
HR Talent Identification

To know more about the Reverse Email Lookup check the blog 3 Simple Methods To Do Reverse Email Lookup

How to perform Reverse Email Lookup?

Here comes Proxycurl into picture.

Proxycurl is an excellent tool for obtaining and enriching data about people and companies. One of its many services include a Reverse Work Email Lookup API which you can use to programmatically pull data into your applications.

It provides an API, using which a reverse email lookup process can be run into your system and you can integrate it with your application.

How to create High Availability Architecture using S3 and CloudFront?

Nitesh Thapliyal — Mon, 05 Dec 2022 06:02:14 +0000

What is High Availability?

A highly available architecture involves multiple components working together to ensure uninterrupted service during a specific period. This also includes the response time to users’ requests. Namely, available systems have to be not only online, but also responsive.

Implementing a cloud computing architecture that enables this is key to ensuring the continuous operation of critical applications and services. They stay online and responsive even when various component failures occur or when a system is under high stress.

Highly available systems include the capability to recover from unexpected events in the shortest time possible. By moving the processes to backup components, these systems minimize downtime or eliminate it. This usually requires constant maintenance, monitoring, and initial in-depth tests to confirm that there are no weak points.

Example:

If you want to deploy a website and want your website to be fast, secure and available all the time even in the time of some failure in the network then High Availabilty Architecture comes in play.

It is important to create High Availability Architecture to work efficiently without any delay due to failure.

In this Blog we will perform following activities:

We will Launch an instance using AWS CLI
We will configure Webserver
We will Create an EBS Volume
We will Attach EBS Volume
We will Create Partition
We will create S3 Bucket
We will Create Cloudfront Distribution

Prerequisites

You should have AWS account, you can create AWS free tier account.
Install AWS CLI in Windows,Mac or in Linux.
Configure AWS CLI with IAM user.
It is preferred that you are aware HTML basics and Linux

Step 1:

Launch an Instance using AWS CLI

First configure the AWS CLI in your terminal or window cmd, here I'm using window cmd.
To configure provide Access Key, Secret Key and region name.

Now to launch an Instance, we require:

Image-id
Instance-type
Count i.e how many instance we want to launch
subnet-id
Security-Group-id
Key-name

Syntax:

aws ec2 run-instances --image-id ami_ --instance-type _type_id --count no_of_instance --subnet-id -a57e77cd --security-group-ids group_id --key-name key_name

output:

Step 2:

Configure Webserver

To Configure Webserver we need to install httpd i.e Apache Webserver in instance that you have launched in Step 1
To install httpd use command: yum install httpd
After httpd installation, we need to check whether service is started or not.
To check the httpd service use command: systemctl status httpd

And to start the service use the command systemctl start httpd

Step 3:

Create an EBS Volume

To Create an EBS Volume we require:

Availability Zone name
Size of Volume that we want to create.

Syntax:

aws ec2 create-volume --availability-zone ap-south-1a --size 1

Here ap-south-1a is availability zone, you have to give availability zone name where you have launched instance and 1 is the size of volume i.e 1Gb, you can give desired size to your volume.

Step 4:

Attach EBS Volume to Instance

To attach the EBS Volume to an Instance we require:

Volume-id (The volume that we have created in step 3)
instance-id (The instance that we have launched in step 2)
device (Example: /dev/sdf)

Syntax:

aws ec2 attach-volume --volume-id (volume_id) --instance-id (instance_id) --device(ex: /dev/sdf)

Step 5:

Create and Format a Partition

Now open the instance that you have launched in step 1
run the command fdisk /dev/xvdf to create partition.
After running this command press n to create new partition and then press p and then enter.
To save the partition press w.

-Now we need to format the partition.

To format the partition use the command mkfs.ext4 device_name

Step 6:

Mount /var/www/html directory to partition
We need to mount /var/www/html directory to partition because we will create our html page in this directory and it will be safe in this partion even in the time of instance failure.
To mount the directory use the command mount /dev/xvdf /var/www/html
And to check it is mounted or not use command df -h

Now create HTML page, go inside directorye🌟 /var/www/html in your linux instance.
Create a HTML file.

Syntax:

<!DOCTYPE html>
<html>
<head>
<style>
img {
  width: 100%;
}
</style>
</head>
<body>
<img src="html5.png"  width="128" height="128">
<img src="html5.png" width="128" height="128">
</body>
</html>

Now add your own content to the syntax above and create your own html webPage.

Step 7:

Create S3 Bucket

To create S3 Bucket we require:

bucket name(you can give any name)
region name(region name should be same like your instance region)

Syntax:

aws s3api create-bucket --bucket --region --create-bucket-configuration LocationConstraint= region

Step8:

Upload images in S3 Bucket

The images that you will use while creating website or webpage, store that images in S3 bucket so that images in your website will open with less latency or delay.

Go to the AWS service in AWS WebUi
Select S3 service
Now select your bucket that you have created in step 7 upload images
Make the images that are uploaded public so that all can see your images.

Now copy the link of the images in the S3 Bucket and paste that link in your webpage.

Step 9:

Create Cloudfront Distribution

We create Cloudfront Distribution so that everyone in the world can access to our website with same speed like we access locally.

We will make Cloudfront Distribution of S3 bucket so that images will open without any delay when someone will access the website from other countries.

To create Cloudfront Distribution we require:

origin-domain-name(S3 Bucket domain)

Syntax:

aws cloudfront create-distribution — origin-domain-name

Output:

Now open the webpage using the Ip address of instance and file name in your browser .

By performing all steps you can deploy your website with High availability,secure your website and make your website manageable.

Hope you find this blog insightful, Do give it a like 🌟

Thank you!

MLOps Stack Requirements

Nitesh Thapliyal — Thu, 14 Apr 2022 18:09:35 +0000

In this Blog we going to discuss one the following topics:

MLOps Stacks requiremnt
AI Model Life Cycle
How we can use Katonics MLOps Platform

As we have discussed about the MLOps already Click to read
We should also know about what set of tools and technologies are used in MLOps stacks.
We will discuss about MLOps life cycle as it is not a single technology or platform rather than a set of technology working together in a systemize way to achieve goal.

MLOps Stack Requirements

- Data gathering and preparation

In data gathering and preparation, platform should be able to gather batches as well as streaming data to automate the process. You have to build pipeline that would engage any batch or streaming data, do the cleaning and transmission to prepare it for model to fit.

- Source Control

It is a tool which we need for saving the versions of many entities. In MLOps platform we experiment with lot of data, so we need to have version control of it so that reproducibility can happen i.e so, at any point of time we can go back and check what data you have used for what.

- Experimentation

The MLOps platform should be data scientist friendly. It should work out of the box with popular machine learning frameworks like PyTorch, Keras, Tensorflow, Scikit-learn and many more.

- Hyperparameter Tuning

Choosing the correct hyperparameter for machine learning and deep learning model is one of best ways to extract the last use of model, therefore we need Hyperparameter Tuning framework to select the best combination that delivers the best performance.

- Distributed Model Training

The MLOps platform should be able to automatically train the model using different sets of data. Training a model is resource intensive, so to serve this platform provide a service which can scale itself and do distributed model training. Platform also need CI/CD tools for scheduling, task queuing.

- Auto ML

The platform should have Auto ML, which help you to iterate over hundreds of algorithm and automatically gives you best accuracy model.

- Deployment

Automated deployment using container service like Docker and kubernetes should be there in a platform.

- Model Registry

It is a repository for hosting all the important artifacts for production ready models like training data, model type, key artifact etc.

- Feature Store

It is use to facilitate collaboration between teams, reduce duplication, reduce feature engineering cost per model, and in turn increases the productivity of ML engineers.

- Monitoring

In monitoring we check the real time performance view of deployed models. Alert mechanism upon seeing data drift or deviation in performance of model.

- Governance

We require Governance services to control access implementation policy and track model activity. Capability to track end-to-end trail of ML assests.

- Compilance and Audit Services

The compilance and audit service includes authorization and authentication. Detect and correct any biased or unethical outcomes.

The AI Model Life Cycle

The AI Model life Cycle represents how data from the source are trained to get insightful information from them.

The data from the sources like:

SaaS
Data Warehouse
External systems

The data is given to the automated pipeline, which includes the following processes to train data :

Data extraction
Data preparation
Feature engineering
Model Training
Model evaluation
Model Validation

The offline data processed through the experimentation and prototyping which includes:

Problem definition
Data selection
Data exploration
Feature engineering
Model prototyping
Model validation

After this the data is stored in the source repository which is connected to the automated pipeline.

The resultant model is stored in model registry and the end result is an API which we can use in our web applications.

How to use katonics MLOps Platform

Katonic is a collection of cloud-native tools for all of the stages of MDLC (data exploration, feature preparation, model training/tuning, model serving, model testing, and model versioning). Katonic provides a Unified UI and has tooling that allows these traditionally separate tools to work seamlessly together. An important part of this tooling is the pipeline system, which allows users to build integrated end-to-end pipelines that connect all components of their MDLC.

Katonic is for both data scientists and data engineers looking to build production-grade machine learning implementations. Katonic can be run either locally in your development environment or on a production cluster. Often pipelines will be developed locally and migrated once the pipelines are ready. Katonic provides a unified system—leveraging Kubernetes for containerization and scalability, for the portability and repeatability of its pipelines.

To know how you can use the Katonics MLOps platform here is a End to end Katonic Walkthrough

Hope you find this blog insightful!! ✨

What is Docker and Kubernetes ?

Nitesh Thapliyal — Thu, 07 Apr 2022 09:23:51 +0000

In this blog we will discuss about following topics:

Docker
Container
Docker image
Kubernetes
Pod
Service
Ingress
Namespace

What is Docker?

Docker is a software platform that allows you to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime. Using Docker, you can quickly deploy and scale applications into any environment and know your code will run.

What is Container?

Containers are packages of software that contain all of the necessary elements to run in any environment. In this way, containers virtualize the operating system and run anywhere, from a private data center to the public cloud or even on a developer’s personal laptop. From Gmail to YouTube to Search, everything at Google runs in containers.Containerization allows our development teams to move fast, deploy software efficiently, and operate at an unprecedented scale.

Containers are lightweight packages of your application code together with dependencies such as specific versions of programming language runtimes and libraries required to run your software services.

What is Docker image?

In Docker, everything is based on Images. An image is a combination of a file system and parameters. Let’s take an example of the following command in Docker

sudo docker run -it centos

To see the list of Docker images on the system, you can issue the following command.

docker images

What is Dockerfile?

Dockerfile is a simple text file that consists of instructions to build Docker images.

Example:

FROM python:alpine3.7
COPY ./app
WORKDIR /app
RUN pip install requirements.txt
EXPOSE 5001
ENTRYPOINT ["python","app.py"]

What is Kubernetes?

Kubernetes also known as “k8s” or “kube” is a container orchestration platform for scheduling and automating the deployment, management, and scaling of containerized applications.

what is POD?

Pods are the smallest deployable units of computing that you can create and manage in Kubernetes.

A Pod (as in a pod of whales or pea pod) is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers.

To check the pods in kubernetes use command:

kubectl get pods

What is service?

A service can be defined as a logical set of pods. It can be defined as an abstraction on the top of the pod which provides a single IP address and DNS name by which pods can be accessed. With Service, it is very easy to manage load balancing configuration. It helps pods to scale very easily.

A service is a REST object in Kubernetes whose definition can be posted to Kubernetes apiServer on the Kubernetes master to create a new instance.

To check the service use command:

kubectl describe serice

What is Ingress?

Ingress is a load balancer for Kubernetes, the main purpose of ingress is to expose the HTTPS and HTTP routes that are outside the cluster to the services within the cluster itself. By the use of ingress controller, we can also define the rules which can turn control the traffic routing.

What is Namespace?

Kubernetes objects like pods and containers live in namespaces, essentially a namespaces is a way to organize objects in your kubernetes cluster

Namespaces is a very useful tool if you have many different type of applications running or if you have multiple teams that share the cluster.

Hope you find this blog insightful 🌟

Reference:

Kubernetes docs
Tutorial point

What is MLOps and Katonic ?

Nitesh Thapliyal — Sat, 02 Apr 2022 10:21:47 +0000

Content:

What is MLOps?
Why do we need MLops?
How MLOps is different from DevOps?
what is Katonic?

What is MLOps?

MLOps stands for Machine Learning Operations, it is an engineering discipline that aims to unify ML system development(Dev) and ML system deployment(Ops) in order to standardize and streamline the continuous delivery of high-performing models in production.

The complete MLOps process includes three broad phases of “Designing the ML-powered application”, “ML Experimentation and Development”, and “ML Operations”.

The first phase is devoted to:

Business understanding
Data understanding
Designing ML software

Objective of MLops team

The objective of an MLOps team is to automate the deployment of ML models into the core software system or as a service component. This means, to automate the complete ML-workflow steps without any manual intervention.

Automation

To adopt MLOps, we see three levels of automation, starting from the initial level with manual model training and deployment, up to running both ML and CI/CD pipelines automatically.

Manual process
ML pipeline automation
CI/CD pipeline automation

Manual Process:

This is a typical data science process, which is performed at the beginning of implementing ML. This level has an experimental and iterative nature. Every step in each pipeline, such as data preparation and validation, model training and testing, are executed manually. The common way to process is to use Rapid Application Development (RAD) tools, such as Jupyter Notebook

ML pipleline automation:

The next level includes the execution of model training automatically. We introduce here the continuous training of the model. Whenever new data is available, the process of model retraining is triggered. This level of automation also includes data and model validation steps.

CI/CD pipeline automation:

In the final stage, we introduce a CI/CD system to perform fast and reliable ML model deployments in production. The core difference from the previous step is that we now automatically build, test, and deploy the Data, ML Model, and the ML training pipeline components.

Why do we use MLOps?

MLOps focuses on the intersection of data science and data engineering in combination with existing DevOps practices to streamline model delivery across the machine learning development lifecycle. MLOps is the discipline of integrating ML workloads into release management, CI/CD, and operations. MLOps requires the integration of software development, operations, data engineering, and data science.

DevOps vs MLOps

DevOps

Team comprises of DevOps Engineers
Linear Development cycle
Versioning of only code
Projects are not compute intensive
CI comprises of testing and validating the code
Monitor throughout, latency, CPU utilization etc
DevOps is Continuous Integration and Continuous Development

MLOps

Diverse team including data scientists, ML engineers, data engineers and DevOps
Interative development cycle
Versioning of code, data, features, environments etc.
Projects are compute intensive
CI comprises of testing and validating code and data
CD comprises of deploying multiple pipelines
Monitor model accuracy, data drift, features along with the basic health checks
MLOps is Continuous Integration, Continuous Development and Continuous Training

what is Katonic?

Katonic is a MLOps Platform that combines the creative scientific process of data scientists with the professional software engineering process to build and deploy Machine Learning Models into production safely, quickly, and in a sustainable way.

It is a collection of cloud-native tools for all of the stages of MDLC (data exploration, feature preparation, model training/tuning, model serving, model testing, and model versioning). Katonic provides a Unified UI and has tooling that allows these traditionally separate tools to work seamlessly together. An important part of this tooling is the pipeline system, which allows users to build integrated end-to-end pipelines that connect all components of their MDLC.
For detail

There are challenges that are faced by chief data and analytics officers:

I want my Data Science team to deliver value
I need to tie my innovation budget to production solution
Data Scientist struggles to go from small sample data to full size data
Maintaining data science is production is nightmare
Our Engineers and Data Scientist don't speak the same language
Our data scientist only use notebook. Can we deploy them in productions.
We have an extensive existing infrastructure and none of the solution on the market work on it.

These challenges are solved by the Katonic by:

Enabling the deployment of models in production with secure, scalable, highly available endpoint deployment with out of the box auto-scaling, and load balancing.
Allowing data scientists to focus on the core task of building ML models to improve business outcomes rather than managing infrastructure.
And much more

Reference

Hope you find this blog insightful ✨

Happy Learning! 🌟

Diabetes Predictor App

Nitesh Thapliyal — Wed, 18 Aug 2021 11:33:51 +0000

Diabetes is a disease that occurs when your blood glucose, also called blood sugar, is too high. Blood glucose is your main source of energy and comes from the food you eat. Insulin, a hormone made by the pancreas, helps glucose from food get into your cells to be used for energy. Sometimes your body doesn’t make enough—or any—insulin or doesn’t use insulin well. Glucose then stays in your blood and doesn’t reach your cells.

Here is a Flutter app that will help to predict that whether user has diabetes or not.

The prediction is based on the value of:

Number of time pregnant
Diastolic Blood Pressure
Tricep skin fold thickness
2-Hour Serum insulin
Body mass index
Diabetes pedigree function
Age

Tech Stack used:

Frontend

For frontend Flutter is used and the language that is used in flutter is Dart.

Backend

For backend Deep learning model is used which is made by using pima Indians diabetes dataset and for the interface between backend and frontend Python CGI is used.

Backend host

For hosting the backend AWS is used.

Creating App

Created about and main page in flutter code you can see in GitHub
Created Python CGI Script to connect server with frontend

Used http package in flutter for making HTTP requests, documentation

import 'package:http/http.dart' as http;

var url = Uri.parse('https://example.com/whatsit/create');
var response = await http.post(url, body: {'name': 'doodle', 'color': 'blue'});
print('Response status: ${response.statusCode}');
print('Response body: ${response.body}');

print(await http.read('https://example.com/foobar.txt'));

created model and hosted it in AWS

Now we can enter the values so that app can predict whether user has diabetes or not.

Now as soon as we click in search icon, behind the scene values will go to server and process the values and output will be get by flutter and displayed in app.

here you can see the output "you have diabetes" this output was mentioned in CGI script, flutter get the output and displayed in the app.

So that's how we can use DL model and integrate it with app.

Thank you!!

PIP-OS-WebAPP

Nitesh Thapliyal — Fri, 25 Jun 2021 18:38:59 +0000

Hello everyone,

In this blog we are going to see how we can integrate JavaScript with technologies and can build something great.

PIP-OS is a WebAPP where user can work on Docker, Kubernetes and much more technologies in the future.

Technologies that PIP-OS support till now:

Docker
Kubernetes

Prerequisite:

knowledge of HTML/CSS/JavaScript
Linux
Webserver Configured in Linux
Docker installed in Linux
kubernetes configured

So now let's start with Docker

Docker

Docker WebAPP can run the Docker commands in browser and you can see the output on your screen.

Use case that is solves:

User don't have to configure the Docker in their system

Tech Stack used:

HTML
CSS
JavaScript
Python

For the front end part the main component created is input box that will take the docker command as an input.

Below is the HTML and CSS code for creating an input box.

 <div class="center-docker">
        <div class="d-flex justify-content-center">
          <div class="searchbar text-white">
            [root @localhost~]# <input id="inp" class="search_input text-white" type="text" name="" placeholder="Enter the command">
        <a onclick="lw()" class="search_icon">
            <i class="fas fa-arrow-right" aria-hidden="true"></i>
        </a>
      </div>
        </div>
    </div>

Here we can see the input id = "inp" which is associated with the JavaScript code that will get the value with the following id from the Input box.

<script>
    function lw() {

    var i = document.getElementById("inp").value

    var xhr = new XMLHttpRequest();


    xhr.open("GET","http://localhostIP/cgi-bin/PIPOS/docker.py?x="+i,true);
    xhr.send();

    xhr.onload = function() {
        var output = xhr.responseText;
        document.getElementById("d1").innerHTML = output;
        }
    }

</script>

here this part of code takes the input from the user through input box and send the input to the python script which is present in the cgi-bin directory.

what is CGI?

Common Gateway interface(CGI) is a way of running the program from the webserver. For Detail

import cgi
import subprocess
import time 

print("context-type: text/html")
print()
#print("let's build something new Nitesh")#here now if now I change anything it will change in page without refreshing

#time.sleep(10) #this holds program for 10 seconds and then run the prog auto
f=cgi.FieldStorage()
cmd=f.getvalue("x")
#print(cmd)

o = subprocess.getoutput("sudo " +cmd)

print(o)

So this cgi program will get the input from the server and run the command in the browser.

Subprocess is a library that is used to run the new application or program through the python code by creating the process.

so now start the server and open the webpage in the browser and run the docker commands in the browser.

Kubernetes

Kubernetes WebAPP can run kubernetes command in the browser, it takes input as a language like we normally talk not commands and then it process that query and shows output accordingly.

Use case that it solves"

User don't need to know the kubernetes commands
User don't need to configure the kubernetes in their system

Tech stack Used:

HTML
CSS
JavaScript
Python

For front end the main part is input box that will take the input as a query from the user.

To create the input box used HTML and CSS.


    <div class="center-docker">
        <div class="d-flex justify-content-center">
          <div class="searchbar text-white">
            [root @localhost~]# <input class="search_input text-white" id="inp1" type="text" name="" placeholder="Enter the command">
            <a onclick="lw()" class="search_icon">
            <i class="fa fa-arrow-right" aria-hidden="true"></i>
        </a>
          </div>
        </div>
    </div>

here we can see the id associated with input box and function associated with the button. so as soon as user will input query and click in the button than function will get triggered that is associated with JavaScript code that will take input and sends that input to the cgi program that will process the input and shows output in the browser.

import cgi
import subprocess as sp
import time

print("context-type: text/html")
print()
#print("let's build something new Nitesh")#here now if now I change anything it will change in page without refreshing

#time.sleep(10) #this holds program for 10 seconds and then run the prog auto
f=cgi.FieldStorage()
#print(cmd)
#cmd = f.getvalue("x")

#o = subprocess.getoutput(cmd + " --kubeconfig admin.conf")

#print(o)

query = f.getvalue("x")

if "show pods" in query:
    cmd = "kubectl get pods --kubeconfig admin.conf"
    o=sp.getoutput(cmd)
    print(o)

elif "show deployment" in query:
    cmd = "kubectl get deployments --kubeconfig admin.conf"
    o = sp.getoutput(cmd)
    print(o)

here is a program that will process input and show output in the browser as discussed above.

Here admin.conf file is a configuration file of kubernetes that will help to run the kubernetes in the Linux.

To watch the Demo video 👇

Face Recognition Alert System and AWS automation

Nitesh Thapliyal — Wed, 23 Jun 2021 21:30:29 +0000

Hello everyone,

In this blog we are going to create an alert system and AWS automation program.

Alert System : In alert system created a program that will detect the face in the image and as soon as it detects an image it will capture it and send that image to the admins email address and sends whatsapp message to admin.

AWS automation : In AWS automation, created a program that will recognize the users face and in the basis of accuracy of face recognition it will run the Terraform code that will create instance in AWS along with 5GB of volume and attach that volume to the instance.

let's start with the code:

Prerequisites:

Python
OpenCV

Alert System:

Libraries included are :


import cv2
from PIL import Image 
from email.message import EmailMessage
import smtplib, ssl
from email.mime.text import MIMEText
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email import encoders
import imghdr 
import os 
import pywhatkit # for whatsapp

Here pywhatkit library is used for whatsapp, it is used in program for sending the messages.

imghdr library is used to find the types of an image.

PIL stands for Python Image Library which is used for image processing.

email and smtlib libraries are used for using the mail service and helps to send the mail.

load the model

model=cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

Here haarcascade model is used to detect the face in image.

Now let's discuss the code for image processing

while True:
    cap = cv2.VideoCapture(0)
    ret, photo = cap.read()
    faces = model.detectMultiScale(photo)
    if len(faces) == 0:
        pass
    else:
        x1=faces[0][0]
        y1=faces[0][1]
        x2=x1+faces[0][2]
        y2=y1+faces[0][3]

        aphoto = cv2.rectangle(photo, (x1,y1), (x2,y2), [0,255,0], 5)
        cv2.imshow("Image Capturing", aphoto)
        if cv2.waitKey(5)==13: #13 is the code for Enter Key
            break
cv2.destroyAllWindows()
cap.release()

when the following code in run then webcam will open and it will detect the face.

image = Image.fromarray(aphoto)
image.save('Alert.png')
image_show=Image.open(r"Alert.png")
image_crop = image_show.crop((x1,y1,x2,y2))
image_crop.show()
image_crop.save('Alert_face_detected.png')
print("Image Captured")

In the following code image is cropped and saved in the system.

Now let's code for Email Alert


email_id = os.environ['my_email']
email_receiver = os.environ['receiver_email']
password = os.environ['my_password']

#Sender, Reciever, Body of Email
sender = email_id
receivers = email_receiver
body_of_email = 'Alert intrucder has been detected'

#added sender and reciver email addresses
msg = MIMEMultipart()
msg['Subject'] = 'Alert Intruder detected'
msg['From'] = sender
msg['To'] = receivers

part = MIMEBase('application', 'octet-stream')
part.set_payload(open('Alert.png', 'rb').read())#Image attached
encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename ="Alert.png"')
msg.attach(part)

#Connecting to Gmail SMTP Server
s = smtplib.SMTP_SSL(host = 'smtp.gmail.com', port = 465)
s.login(user = sender, password = password)

s.sendmail(sender, receivers, msg.as_string())

In the following code the image that was saved is mailed to other user using smtp protocol, which is used by the gmail.

Now let's code for WhatsApp alert

number = os.environ['phone_number']

import pywhatkit
pywhatkit.sendwhatmsg(number, 'Alert Intruder Detected ',2,29)

In the following code sendwhatmsg is a function from pywhatkit library that is use to send the whatsapp message.

Here Alert Intruder Detected is message and 2,29 is a time when to send the message.

After code is run, whatsaap will open in browser and message will be send.

So that's how as soon as face is detected code will run that will capture image and send it to admin's email address along with the whatsapp alert message.

AWS automation using Terraform with the help of Face Recognition

To create a Face Recognition program, first we need to create Dataset and then using the dataset we need to train the model that will help us in Face recognition.

Create Dataset

To create dataset use the code below, following code will capture the user image 100 times and store it in a directory that will be used by the model.


import cv2
import numpy as np

# Load HAAR face classifier
face_classifier = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

# Load functions
def face_extractor(img):
    # Function detects faces and returns the cropped face
    # If no face detected, it returns the input image

    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    faces = face_classifier.detectMultiScale(gray, 1.3, 5)

    if faces is ():
        return None

    # Crop all faces found
    for (x,y,w,h) in faces:
        cropped_face = img[y:y+h, x:x+w]

    return cropped_face

# Initialize Webcam
cap = cv2.VideoCapture(0)
count = 0
# Collect 100 samples of your face from webcam input
while True:

    ret, frame = cap.read()
    if face_extractor(frame) is not None:
        count += 1
        face = cv2.resize(face_extractor(frame), (200, 200))
        face = cv2.cvtColor(face, cv2.COLOR_BGR2GRAY)

        # Save file in specified directory with unique name
        #path
        file_name_path = 'Path_of_dir/' + str(count) + '.jpg'
        cv2.imwrite(file_name_path, face)

        # Put count on images and display live count
        cv2.putText(face, str(count), (50, 50), cv2.FONT_HERSHEY_COMPLEX, 1, (0,255,0), 2)
        cv2.imshow('Face Cropper', face)

    else:
        print("Face not found")
        pass

    if cv2.waitKey(1) == 13 or count == 100: #13 is the Enter Key
        break

cap.release()


cv2.destroyAllWindows()

Here you can see *Count == 100 * that means image will be capture 100 times and above part image is croped so that only the face part is captured.

like this 100 images will captured and stored in a directory.

Train the Model

To train the model use the code below and provide the directory of image dataset in the code which will help in model training.

import cv2
import numpy as np
from os import listdir
from os.path import isfile, join

# Get the training data we previously made
data_path_1 = 'path_of_image_dataset/'
onlyfiles_1 = [f for f in listdir(data_path_1) if isfile(join(data_path_1, f))]


# Create arrays for training data and labels
Training_Data_1, Labels_1 = [], []
# Create arrays for training data and labels

# Create a numpy array for training dataset 1
for i, files in enumerate(onlyfiles_1):
    image_path = data_path_1 + onlyfiles_1[i]
    images = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    Training_Data_1.append(np.asarray(images, dtype=np.uint8))
    Labels_1.append(i)   


Labels_1 = np.asarray(Labels_1, dtype=np.int32)


Nitesh_model  = cv2.face_LBPHFaceRecognizer.create()
Nitesh_model.train(np.asarray(Training_Data_1), np.asarray(Labels_1))
print("Model trained sucessefully")

Now our model is trained .

Create Face Recognition program

To create the Face Recognition program use the code below, now as it will recognize the face and bases on the accuracy of the face recognition terraform code will be executed which will create an instance in AWS along with 5GB of ebs volume and attach it to the instance.


import cv2
import numpy as np
import os


face_classifier = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

def face_detector(img, size=0.5):

    # Convert image to grayscale
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    faces = face_classifier.detectMultiScale(gray, 1.3, 5)
    if faces is ():
        return img, []


    for (x,y,w,h) in faces:
        cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,255),2)
        roi = img[y:y+h, x:x+w]
        roi = cv2.resize(roi, (200, 200))
    return img, roi


# Open Webcam
cap = cv2.VideoCapture(0)

while True:

    ret, frame = cap.read()

    image, face = face_detector(frame)

    try:
        face = cv2.cvtColor(face, cv2.COLOR_BGR2GRAY)

        # Pass face to prediction model
        # "results" comprises of a tuple containing the label and the confidence value
        results = Nitesh_model.predict(face)

        if results[1] < 500:
            confidence = int( 100 * (1 - (results[1])/400) )
            display_string = str(confidence) + '% Confident it is User'

        cv2.putText(image, display_string, (100, 120), cv2.FONT_HERSHEY_COMPLEX, 1, (255,120,150), 2)

        if confidence > 70:
            cv2.putText(image, "Hello Nitesh", (250, 450), cv2.FONT_HERSHEY_COMPLEX, 1, (0,255,0), 2)
            cv2.imshow('Face Recognitioned', image )
            if cv2.waitKey(1)==13:
                cap.release()
                cv2.destroyAllWindows()
                !terraform init
                !terraform apply --auto-approve


        else:
            cv2.putText(image, "Unrecognised Face", (250, 450), cv2.FONT_HERSHEY_COMPLEX, 1, (0,0,255), 2)
            cv2.imshow('Face Recognition', image )

    except:
        cv2.putText(image, "Face Not Found", (220, 120) , cv2.FONT_HERSHEY_COMPLEX, 1, (0,0,255), 2)
        cv2.putText(image, "Searching for Face....", (250, 450), cv2.FONT_HERSHEY_COMPLEX, 1, (0,0,255), 2)
        cv2.imshow('Face Recognition', image )
        pass

    if cv2.waitKey(1) == 13: #13 is the Enter Key
        break

cap.release()
cv2.destroyAllWindows()

Here you can see 92% is an accuracy which is more than the accuracy mentioned in a program therefore it will execute the terraform code.

To watch the Demo 👇