Forem: GNU/Linux Users' Group, NIT Durgapur

The DeepSeek Revolution: The AI Game Changer You Need to Know About

Ayush Bhartia — Sun, 16 Feb 2025 12:21:08 +0000

Hey everyone! The AI landscape is constantly evolving, with new models pushing the boundaries of what's possible. But one name has been making waves recently, i.e, DeepSeek. Unlike other AI tools, this powerhouse has rewritten the rulebook on efficiency, cost-effectiveness, and performance. If you haven't heard of it yet, here’s why it’s capturing global attention.

What Is DeepSeek and Why Is It Gaining Popularity?

In January 2025, DeepSeek, a Chinese AI company, emerged as a major player in the AI space. Developed by a Chinese AI lab, it rapidly climbed the app store rankings, even surpassing ChatGPT as the most popular free AI assistant in the US.
But what’s driving the hype? DeepSeek-R1 isn’t just another AI model—it’s an example of cutting-edge efficiency at a fraction of the usual cost. Unlike its competitors, which require massive investments, DeepSeek-V3 was trained for just $5.6 million—a game-changing breakthrough in AI development.

Innovation Behind DeepSeek

DeepSeek’s strength lies in its Mixture of Experts (MoE) training method, which splits its AI into specialized sub-models. Instead of a single large neural network handling everything, each part of DeepSeek is optimized for specific tasks, making it both smarter and more resource-efficient.
Another key advantage is its adaptive compute scaling, which adjusts processing power based on task complexity. This means DeepSeek can deliver high performance while using significantly fewer resources compared to other models.

Moreover, instead of relying on Nvidia’s high-end H100 chips, DeepSeek utilizes the more affordable H800 chip, making AI training and deployment 21 times cheaper than models like Claude 3.5 Sonnet.

DeepSeek-R1 helping open-source community

DeepSeek released its flagship model, DeepSeek-R1, under the MIT License. This model, developed with remarkable resource efficiency—using approximately 2,000 Nvidia H800 GPUs over 55 days demonstrated performance on par with leading AI models from established tech giants. DeepSeek has made significant strides in the artificial intelligence landscape by open-sourcing its advanced AI models.
The open-source release of DeepSeek-R1 has lowered barriers to AI development, fostering innovation and competition across the global AI community. The company has also released models like DeepSeek-Coder, designed to enhance code intelligence, and DeepSeek-VL, aimed at advancing vision-language understanding. These models are available for public use and modification, promoting transparency and collaboration in AI research and application.

How DeepSeek Outperforms Other AI Models

Beyond its efficiency, DeepSeek is redefining AI reliability with its R1 Reasoning Model. Unlike traditional models, the probability of the R1 Reasoning Model giving appealing outputs is much higher as compared to other available models since this model is designed to validate its own reasoning supporting its output, improving accuracy in subjects like math, science, and fact-checking.
This makes DeepSeek particularly powerful for research, problem-solving, and advanced computations, offering a level of precision that many AI models like o3-mini from OpenAI, Llama 3 from MetaAI, and Gemini 2.0 Flash from Google, etc struggle to achieve.

Challenges and Limitations

Despite its impressive capabilities, DeepSeek isn’t without challenges. Its business model remains uncertain, as it offers services at a significantly lower cost than competitors, raising questions about long-term sustainability. If integrated into government or corporate infrastructure, DeepSeek AI could be exploited for surveillance by foreign adversaries, affecting nation's cybersecurity. Although being open-source, certain components of the model (e.g., training datasets and fine-tuned versions) might be controlled by external entities, leading to potential data leaks.
Additionally, being developed in China, DeepSeek follows strict content regulations. Topics such as Taiwan’s political status or the Tiananmen Square incident are restricted due to Chinese internet policies, which may affect its global adaptability.

What’s Next for DeepSeek and AI?

DeepSeek’s rise proves that bigger isn’t always better. With its cost-effective approach, rapid deployment, and self-reasoning abilities, it’s challenging industry giants and setting a new standard for AI development.
Whether you’re a developer, researcher, or AI enthusiast, DeepSeek is worth watching. Have you tried it yet? Let us know your thoughts in the comments!
We hope you found this insightful.

Do visit our website to know more about us and also follow us on :
Facebook
Instagram
LinkedIn
YouTube

Also, do not forget to like and comment.

Until then,
Keep exploring and May the Source Be With You!

Introduction to serverless architecture with Firebase and Cloud Functions.

Aiman Aisha — Fri, 01 Sep 2023 06:44:15 +0000

Hey there, let's talk about serverless architecture – the game-changer for how we build and deploy applications, offering a cost-effective, scalable, and efficient solution for various use cases. In this article, we're gonna break down the nitty-gritty of serverless architecture, all the cool perks it brings, and how it's given the whole development scene a crazy makeover. So, let's dive into the world of serverless!

What is Serverless Architecture?

Picture this: developers get to create and run apps without even thinking about setting up servers. The real MVP here? The cloud provider takes care of all the behind-the-scenes server stuff, leaving developers free to go wild with their code.This is how serverless architecture allows rapid development, automatic scaling, and reduced operational overhead.

Some key Components of Serverless Architecture

Function as a Service (FaaS): FaaS is at the heart of serverless architecture. Developers write functions that are triggered by events, such as HTTP requests, database changes, or timers. These functions are executed in a stateless manner, allowing them to scale automatically based on demand.
Backend as a Service (BaaS): BaaS complements FaaS by providing pre-built services, such as databases, authentication, file storage, and messaging queues. BaaS eliminates the need to build these components from scratch, enabling faster development and reducing complexity.

Why Serverless Architecture?

Cost Savings: With serverless, you only pay for the actual usage of your functions and services, rather than paying for idle resources.
Scalability: Serverless platforms automatically scale your functions based on the incoming workload. Whether you have 10 users or 10,000 users, the platform dynamically allocates resources to handle the traffic, ensuring optimal performance and eliminating concerns about handling sudden spikes in usage.
Developer Productivity: Serverless architecture allows developers to focus on writing application logic rather than managing infrastructure. By offloading server management and operations to the cloud provider, developers can deliver features faster, iterate quickly, and improve overall productivity.
High Availability: Serverless platforms provide built-in fault tolerance and high availability. The underlying infrastructure ensures that functions are distributed across multiple data centres, reducing the risk of single points of failure and ensuring robustness.

Serverless Architecture with Firebase

Firebase, developed by Google, is a prominent platform that empowers serverless computing, specifically designed for mobile and web application development. In this article, we will explore the Firebase Emulator Suite, a robust toolkit that equips developers with the ability to locally develop and test their Firebase applications. We will discuss the process of setting up the Firebase Emulator and constructing a Contact Management System utilizing Firebase Cloud Functions.

Contact Management System with Firebase

Before we begin, ensure that you have the following prerequisites in place:

Node.js installed on your machine.
JDK installed on your machine.
Basic knowledge of Firebase concepts and the Command Line Interface (CLI).

1.Setting up Firebase Emulator and initialising Firebase SDK

Firstly, you need to ensure that you have the Firebase CLI (Command-Line Interface) installed. To do so, open your terminal or command prompt and run the following command:

npm install -g firebase-tools

To Check the version of Firebase installed, you can use the following command:

firebase ––version

Next, you need to login to your Firebase account. Typing the following command will redirect you to a login page, where you’ll be prompted to login.

firebase  login

Now, you need to make a new project in your Firebase Console. For this article, we have created a project contact-manage.
Create a new directory for your project and navigate into it. For the next step, initialize Firebase in your project folder using the following command:

firebase  init

You will be asked to select the Firebase features you want to set up for your directory. Press Space to select features, and then press enter. For this project, we have only chosen the ‘function’ feature. You will also be asked to choose a project and a language to write the cloud functions. Select the required options to complete the initialization process.

The Firestore emulator grants you access to a local version of your Firestore database, resembling the view you would see in the Firebase Console. However, the emulator offers faster performance compared to the online console.To install the Emulators for your app, type in the following command:

firebase init emulators

Then, you will be prompted for setup questions in the terminal. First, it will ask you to pick which emulators you want to use. In this article, we’ll cover Authentication, Firestore, and Cloud Functions.
After you pick the emulators you want to use, it will then prompt you to pick the ports they’ll run on. We will just go ahead with the defaults.
It will also ask you if you want to use the emulator’s UI, pick yes, because we’ll be using that to review and interact with our firebase application.

Now, to test that everything is set up, open the terminal again in your project’s root folder, and start the emulator using:

firebase emulators:start

It will start your emulators and show you the emulators' information in the terminal.

After starting the Firebase Emulator, you can access its user interface by navigating to localhost:4000 in your web browser. The UI provides a dashboard displaying all the active emulators you have configured. By clicking on a specific emulator, you will be directed to its corresponding page.

2. Writing Cloud Functions

Our app will allow the user to create a new contact, fetch all the saved contacts, delete a contact, update a contact and finally, search for a contact.

Open the index.js file and start with initialising the Firebase Admin SDK with the following line of code:

 admin.initializeApp();

Reference to the Firestore collection ‘contacts’ for storing all the contacts:

const contactsCollection = admin.firestore().collection('contacts');

We will now define all the required functions.

Create Contacts Function

Our first function is createContacts. This function creates a new contact in the Firestore database. It listens for an HTTP request and expects the contact details in the request body. It uses the contactsCollection reference to add the contact to the Firestore collection. Upon successful creation, it responds with the contact data and a status code of 201.

exports.createContact = functions.https.onRequest(async (req, res) => {
  try {
    const newContact = req.body;
    const contactRef = await contactsCollection.add(newContact);
    const contactId = contactRef.id;
    const contact = await contactRef.get();
    res.status(201).json({id:contactId, ...contact.data()});
  } catch (error) {
    res.status(500).send('Error creating contact');
  }
});

Get Contacts Function

Next, we will create a function called getContacts. This function retrieves all contacts from the Firestore database. It listens for an HTTP request and fetches all documents from the contactsCollection. It then maps the document data and responds with an array of contact objects and a status code of 200.

exports.getContacts = functions.https.onRequest(async (req, res) => {
  try {
    const contactsSnapshot = await contactsCollection.get();
    const contacts = contactsSnapshot.docs.map((doc) => doc.data());
    res.status(200).json(contacts);
  } catch (error) {
    res.status(500).send('Error retrieving contacts');
  }
});

Update Contact Function

Our next function is updateContacts. This function updates an existing contact in the Firestore database. It expects the contact ID in the request URL and the updated contact details in the request body. Using the contactsCollection reference, it locates the specific contact document and updates it with the new information. Upon successful update, it responds with a success message and a status code of 200.

exports.updateContact = functions.https.onRequest(async (req, res) => {
  try {
    const contactId = req.params.contactId;
    const updatedContact = req.body;

    await contactsCollection.doc(contactId).update(updatedContact);
    res.status(200).send('Contact updated successfully');
  } catch (error) {
    res.status(500).send('Error updating contact');
  }
});

Delete Contact Function

To delete a contact from the Firestore database, we will create a function deleteContact. It listens for an HTTP request with the contact ID in the URL. Using the contactsCollection reference, it finds the corresponding document and deletes it. If successful, it responds with a success message and a status code of 200.

exports.deleteContact = functions.https.onRequest(async (req, res) => {
  try {
    const contactId = req.params.contactId;
    await contactsCollection.doc(contactId).delete();
    res.status(200).send('Contact deleted successfully');
  } catch (error) {
    res.status(500).send('Error deleting contact');
  }
});

Search Contacts Function

Finally, we will create a function searchContacts. This function searches for contacts by name in the Firestore database. It expects a search term as a query parameter in the request URL. It queries the contactsCollection using the where method to find contacts with names greater than or equal to the search term. It then maps the query snapshot and responds with an array of matching contacts and a status code of 200.

exports.searchContacts = functions.https.onRequest(async (req, res) => {
  try {
    const searchTerm = req.query.q;
    const querySnapshot = await contactsCollection.where('name', '>=', searchTerm).get();
    const contacts = querySnapshot.docs.map((doc) => doc.data());
    res.status(200).json(contacts);
  } catch (error) {
    res.status(500).send('Error searching contacts');
  }
});

3. Triggering functions on auth events

Create User Function

We will now export a function named addUser. The core functionality of this function revolves around the onCreate event trigger. This event is fired whenever a new user account is created in Firebase Authentication. Upon activation, the function logs a message.
The function takes a single argument named user. This user object holds crucial information about the newly created account, including properties like uid and email.To provide yourself with insight into the function's execution, you can log a message to the Firebase Cloud Functions logs.

exports.addUser = functions.auth.user().onCreate((user)=>{
  console.log(`${user.email} has been created...`)
  return Promise.resolve
})

Delete User Function

We will next define the delUser function, which will serve the purpose of deleting a user from our app. The core functionality revolves around utilising the onDelete event trigger, which is triggered whenever a user account is deleted from Firebase Authentication. Once activated, the function proceeds to execute specific tasks related to user deletion.
The function accepts a single argument named user. This user object encapsulates pertinent information about the deleted account, including attributes such as email and uid.

exports.delUser = functions.auth.user().onDelete((user)=>{
  console.log(`${user.email} has been deleted...`)
  return Promise.resolve
})

4. Testing the Functions

We will now see how we perform operations using the functions we created. To make HTTP requests, you will require the URLs of the functions you have created so far. These URLs will be displayed on the terminal/command prompt when you create the functions.

You can add a new contact in the database using the createContact function. Open your terminal/command prompt and use the curl command to make a POST request to the URL of the createContact function. Pass the contact details in the body.

curl -X POST -H "Content-Type: application/json" -d '{
    "name": "hailey",
    "email": "hailey34@example.com",
    "phone": "5432xxxxx0"
  }' YOUR_FUNCTION_URL

This will add a new document in the contacts collection in our Firestore database. You can check this by navigating to the emulator's UI running on localhost:4000.

The next step is to test the authentication functions. We will take the addUser function as an example. Navigate to the Authentication section of the Emulators' UI and click on the Add User button. Fill in the required fields and add the user.

Adding this user will trigger our addUser function.

Challenges and Considerations

While serverless architecture offers numerous benefits, it's important to consider the following challenges:

Cold Start Latency: The first invocation of a function may experience a delay known as a "cold start" as the serverless platform provisions the necessary resources. Although the platforms are continuously improving cold start times, it's crucial to optimize your code and leverage techniques like connection pooling to mitigate this latency.
Vendor Lock-In: Adopting serverless architecture may tie your application to a specific cloud provider's platform and services.
Performance Monitoring and Debugging: As serverless abstracts away infrastructure management, monitoring and debugging can become more challenging.

In addition to these considerations, a number of other drawbacks have surfaced. These came to light when Amazon Prime recently declared that it would stop using serverless technology.
Scaling limitations, potential cost inefficiencies, issues with data-intensive operations, and constraints related to real-time processing are some scenarios that can pose as hurdles. It is also important to align architecture choices with specific project requirements, and caution must be taken against extreme stances on architecture.
A balanced and context-aware approach is essential for making effective architectural decisions.

Alright, so let's check it out – this whole serverless thing has totally shaken up the game when it comes to building apps. It's like this new era where you can develop your stuff without worrying about servers and all that complications.

Instead of pulling your hair out over server management, you can dive right into crafting your app's core. You'll be dropping features left and right, making users happy because you're lightning-fast at giving them what they want.

Thank you for sticking with us till the end. We hope you had a great time reading this article!

Project Github repository :

aimanaisha / contact-manager

Contact-manager

A contact management system where users can store and manage their contacts.

This project uses Firebase Cloud Functions to handle CRUD operations for contacts, implements search functionality, and allows users to update contact details.

View on GitHub

Project collaborators :

Aiman AishaFollow

sahadat-skFollow

ROUNAK KUMAR JHAFollow

Do visit our website and connect with us

Expressando (Part 1): A real-time sign language detection system

Pritam — Sat, 22 Jan 2022 17:31:10 +0000

Expressando is a real-time sign language detection system made using OpenCV, Keras, Tensorflow and SciPy. It is a beginner-friendly project, suitable for the enthusiasts in the field of Machine-Learning and Deep-Learning. The projects primarily aim to highlight the basic use of OpenCV in image processing manipulation, training of models with Keras using Tensorflow as backend, and finally, the detection of our customised sign-language after constructing a Convolutional Neural Network on it. So, without any further ado, let us begin with the tutorial:
Expressando has been written in Python. So, before we start, you can have a quick recapitulation of the basics from the following resources, which will be beneficial for a better understanding of the concepts:

Python Beginner's Guide
W3Schools
Tutorials Point

Also, make sure that you have Python installed on your system.
In case, you do not have Python pre-installed, download the latest version of Python from here. You will also have to create a virtual environment for this project.

What is a 'Virtual Environment'?

A virtual environment is an isolated Python-stimulated environment where a project's dependencies and packages are installed in a different directory from the other packages installed in the system's default Python path (known as the base environment) and other virtual environments. It is synonymous to a 'container', where you have all your required dependencies installed and ready to be used in your project. To know how to create a virtual environment, click here.

Download the "requirements.txt" from the given link: requirements.txt.

Copy the "requirements.txt" file and store it under the directory "TDoC-2021". You will have the following folder structure:

├── TDoC-2021
|    ├── env       
|    ├── requirements.txt

Now type the following command in your Terminal window:

pip3 install -r requirements.txt

You will now have all the required dependencies and Python packages with their appropriate versions installed in your virtual environment named "env". You can check whether the dependencies are installed according to the "requirements.txt" file by the following command:

pip3 list

This command enlists all the installed dependencies installed in your encironment.

Configuring Input through Webcam using OpenCV

After settinp up your virtual environment, it is time to configure your digital input. The first step of any image manipulation project starts with the configuration of digital image input using OpenCV. So let us first configure the basic webcam input.

Step 1: Create a file named as "check.py" inside the directory of "TDoC-2021". As the name suggests, we are checking for the input through the webcam using the OpenCV library. Open the file in your code-editor/IDE. The folder structure would look like the following:

├── TDoC-2021
|    ├── env    
|    ├── check.py
|    ├── requirements.txt

Step 2: First, import OpenCV into the "check.py" file. It can be accomplished by the following line of code:

import cv2

After importing cv2, we need to create a VideoCapture object, which will initiate the process to retrieve the input through the webcam.

cap = cv2.VideoCapture(0)

Here, "cap" refers to the object that is created using OpenCV to capture the video. It basically returns the video from the first webcam on your computer.
If you are using more than one webcam then the value "0" indicates that the input will be configured through the first webcam of your computer.

For example, if you want to configure the input through your 2nd webcam, then you have to pass "1" instead of "0" as the parameter. In simple words, it means if you want to configure the input through the "n-th" webcam, then you must pass "n-1" as parameter to the VideoCapture method.
Step 3: This step involves rendering a while loop to stimulate asynchronous input through the webcam with the help of a suitable condition. In this step, we will be discussing the most common and important methods that are present in the OpenCV library, which are required for making basic projects and develop sound understanding about the various methods present in OpenCV and their uses. OpenCV is a house to a huge number of methods and functions, so we will be discussing only the important methods, which are necessary for beginners to understand.

Continue in the code-editor as follows:

while (cap.isOpened()):
    ret, img = cap.read()
    img=cv2.flip(img, 1)

The function cap.isOpened() checks whether the VideoCapture object (here 'cap') is functional or not. This is done by usually checking the response from the webcam under consideration. This code initiates an infinite loop (to be broken later by a break statement), where we have "ret" and "frame" being defined as the cap.read(). Basically, ret is a boolean regarding whether or not there was a return at all. On the other hand, frame contains each frame that is being returned in the form of an image array vector.

This is practised in order to avoid unnecessary IO errors. In case, no frame was returned, ret will obtain False as it's return value. Hence, instead of throwing an IO error, it will pass None to the frame.
The next line of code introduces us to the method flip(). This method inverts the frame taken into consideration, laterally. Using flip, the input will be similar to a mirror. It is beneficial as it eases the orientation of the webcam input.

Step 4:

    cv2.rectangle(img, (20, 20), (250, 250), (255, 0, 0), 3)
    cv2.imshow("RGB Output", img)
    img1 = img[20:250,20:250]       
    imCopy = img1.copy()
    gray = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
    blur = cv2.GaussianBlur(gray, (5, 5), 0)

In the next lines of code, we are introduced to few other methods in the OpenCV library. The method rectangle() enables us to draw a rectangle of our desired shape on the frame taken into consideration.

It has the following parameters:

img: It is the frame taken into consideration on which the rectangle is to be drawn.
(20, 20): It is the starting coordinates of rectangle. The coordinates are represented as tuples of two values, the X and Y coordinates respectively.
(250, 250): It is the ending coordinates of rectangle. The coordinates are represented as tuples of two values similarly as the starting point. Both the tuples indicate the right diagonal of the rectangle drawn. If the x and y coordinates of the tuples are same, it will result in a square.
(255, 0, 0): It is the color of border line of rectangle which is to be drawn, passed in the form of BGR index. BGR index comprises of Blue, Green and Red colour values, which are used to define other colours as well. Each of the values ranges from 0 to 255. Here, (255, 0, 0) denotes the blue colour.
3: It denotes the thickness of the rectangle border line in px.

The method imshow() shows the image in the form of an independent window. It has two parameters: The name of the window and the image to be displayed.
Next, we extract the region covered by the rectangle in the form of a list of pixels named "img1". The extraction We also make a copy of the extracted image and name the copy as "imCopy" using the copy() function.

Then we are introduced to the method "cvtColor()". This method is used to convert the image into different color-spaces. There are more then hundreds of color-space filters available in OpenCV, but we will be using COLOR_BGR2GRAY for now. This converts the image taken into consideration, in the form of BGR, and converts the entire image into grayscale. We name the grayscale image as gray.

The left image is the original image, while the right image represents it's grayscale form.

We will also use the GaussianBlur() method here. It is an image-smoothening technique (also known as blurring) to reduce the amount of luminant noise in the image. We will stored the reduced image as blur.

The left image is the original image, while the right image represents it's blurred form.

It has the following parameters:

gray: It is the frame taken into consideration on which the method is to be applied.
(5, 5): It is the gaussian Kernel size defined along the X and Y axes, passed in the form of a tuple.
0: It denotes the thickness of the rectangle border line in px.

Step 5:

    ret, thresh1 = cv2.threshold(blur, 10, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
    hand_resize = cv2.resize(thresh1, (width, height))
    cv2.imshow("Threshold", thresh1)

Thresholding is a technique in OpenCV, which is the assignment of pixel values in relation to the threshold value provided. In thresholding, each pixel value is compared with the threshold value. It is one of the most common (and basic) segmentation techniques in computer vision and it allows us to separate the foreground (i.e., the objects that we are interested in) from the background of the image. A threshold is a value which has two regions on its either side i.e. below the threshold or above the threshold. If the pixel value is smaller than the threshold, it is set to 0, otherwise, it is set to a maximum value.

Here, ret performs the same function as before, while thresh1 contains our thresholded image. Then, we define a width and the height in the form of a tuple, before the initialisation of the cap object.

There are mainly three types of thresholding techniques:

Simple Threshold: In this type of thresholding, we manually supply parameters to segment the image — this works extremely well in controlled lighting conditions where we can ensure high contrast between the foreground and background of the image. The parameters are discussed later.
Adaptive Threshold: In this type of thresholding, instead of trying to threshold an image globally using a single value, it breaks the image down into smaller pieces, and then thresholds each of these pieces separately and individually. It is better in limited lighting conditions.
OTSU Threshold: In Otsu Thresholding, the value of the threshold is not defined but is determined automatically. This works well when we are not sure of the lighting conditions. This is an additive module, i.e, it is applied in addition to Simple or Adaptive threshold and works well with grayscale images.

The function threshold() has the following parameters:

blur: The input image array on which the blur effect is applied.
10: The value of Threshold below and above which pixel values will change accordingly.
255: The maximum value that can be assigned to a pixel, in general the intensity of a colour ranges from 0 to 255.
cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU: The type of thresholding that is applied to the image.

There are other thresholding techniques as well:

cv2.THRESH_BINARY: If pixel intensity is greater than the set threshold, value is set to 255 (white), else it is set to be at 0 (black). Here the brighter pixels are converted to black and darker pixels to white.
cv2.THRESH_BINARY_INV: If pixel intensity is greater than the set threshold, value is set to 0(black), else it is set to be 255 (white). Here the brighter pixels are converted to white and darker pixels to black.
cv2.THRESH_TRUNC: If pixel intensity value is greater than threshold, it is truncated to the mentioned threshold. The pixel values are set to be the same as the threshold. All other values remain the same.
cv2.THRESH_TOZERO: Pixel intensity is set to 0, for all the pixels intensity less than the threshold value.
cv2.THRESH_TOZERO_INV: Pixel intensity is set to 0, for all the pixels intensity greater than the threshold value.

The thresholded image of the region under consideration is displayed using the imshow() function.

The above are the examples of the thresholding modules.

Step 6:

    contours, hierarchy = cv2.findContours(thresh1, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    cv2.drawContours(imCopy, contours, -1, (0, 255, 0))
    cv2.imshow('Draw Contours', imCopy)

Contours are defined as the line joining all the points along the boundary of an image that are having the same intensity. Contours come handy in shape analysis, finding the size of the object of interest, and object detection. It is defined by the minimum number of edges required to define the shape taken into consideration. This is done well with thresholded and grayscale images. It is done by the function findContours().

Normally we use the cv.findContours() function to detect objects in an image. Sometimes, the objects are in different locations and in some cases, some shapes are inside other shapes just like nested figures or concentric figures. In this case, we call outer one as parent and inner one as child. This way, contours in an image has some relationship to each other. And we can specify how one contour is connected to each other, like, is it child of some other contour, or is it a parent etc. Representation of this relationship is called the Hierarchy.

The above picture represents the hierarchy of the contours. Contours that have the same integer have the same hierarchy.

The function has the following parameters:

thresh1: The input image array from which the contours are to be detected.
cv2.RETR_TREE: This is known as the Contour Retrieval Method.
cv2.CHAIN_APPROX_SIMPLE: This is known as the Contour Approximation Method.

Contour Retrieval Method are of the following types:

cv2.CV_RETR_EXTERNAL: It retrieves only the extreme outer contours. It sets hierarchy[i][2]=hierarchy[i][3]=-1 for all the contours. This gives "outer" contours, so if you have (say) one contour enclosing another (like concentric circles), only the outermost is given.
cv2.CV_RETR_LIST: It retrieves all of the contours without establishing any hierarchical relationships. This is applied when the hierarchy and topology of the object cannot be determined from beforehand.
cv2.CV_RETR_CCOMP: It retrieves all of the contours and organizes them into a two-level hierarchy. At the top level, there are external boundaries of the components. At the second level, there are boundaries of the holes. If there is another contour inside a hole of a connected component, it is still put at the top level. (ADVANCED)
cv2.CV_RETR_TREE: It retrieves all of the contours and reconstructs a full hierarchy of nested contours. This full hierarchy is built and displayed. It establishes complete hierarchial relations and imagifies the contours.

Contour Approximation Method are of the following types:

cv2.CHAIN_APPROX_NONE: It stores all the points of the boundary of the shape under consideration. It requires a huge amount of memory to store each unit. It is efficient but highly reduces the speed of execution.
cv2.CHAIN_APPROX_SIMPLE: It removes all redundant points and compresses the contour, thereby saving memory. It stores the key turning points of the shape under consideration and saves a lot of memory by reducing the number of points, hence increasing the speed of execution.

The examples of the approximation methods are shown as above.

The function drawContours() is used to draw the contours that have been traced, superimposed on the top of an image. In case, we do not want to display it over any image, the default is set to black.

The function has the following parameters:

imCopy: The input image array on which the contours are to be displayed.
contours: These refers to the 'contours' array that have been declared and initialised in the findContours() function.
-1: It is the parameter to show all the contours in the 'contours' array. However, if you want to display a specific contour according to the hierarchy, pass the desired number as the parameter. For example, to get the 3rd contour, you have to pass 2 as a parameter.
(0, 255, 0): It is the color of contour which is to be drawn, passed in the form of BGR index. Here, (0, 255, 0) denotes the green colour.

Then we display the contours, superimposed on "imCopy" image using the imshow() function.

Step 7: Now, after checking for the input, it is time to proceed for the termination of the while loop and close all the windows and close our Video Capture object.

To exit the program on a specified keyboard interrupt, type the following code:

    k = cv2.waitKey(10) & 0xFF
    if k == 27:
      break

cap.release()
cv2.destroyAllWindows()

The cv2.waitKey(10) function returns -1 when no input is made whatsoever. As soon the event occurs (a Button is pressed, here 27 is the Unicode value for Escape Key), it returns a 32-bit integer. (ADVANCED)
The 0xFF in this scenario is representing binary 11111111, a 8 bit binary, since we only require 8 bits to represent a character we AND waitKey(10) to 0xFF. As a result, an integer is obtained below 255. ord(char) returns the ASCII value of the character which would be again maximum 255. (we often use 'q' as the keybinding to 'quit'). Hence by comparing the integer to the ord(char) value, we can check for a key pressed event and break the loop. The 0xFF is a hexidecimal input, known as bit mask.(ADVANCED)
32 is also the Unicode value for 'Non-breaking Space', made by the Space Bar.
Now the loop breaks when the key is entered, and exits the control out of the loop.

The function cap.release() closes the webcam input and prevents any resource errors. The function cv2.destroyAllWindows() destroys all the opened windows rendered by the imshow() functions and deallocates the memory used by the image vector arrays and frees them.

Now your check.py should look like the following.

import cv2
(width, height) = (130, 100)
cap=cv2.VideoCapture(0)
while (cap.isOpened()):
    ret, img = cap.read()
    img=cv2.flip(img, 1)
    cv2.rectangle(img, (20, 20), (250, 250), (255, 0, 0), 3)
    cv2.imshow("RGB Output", img)
    img1 = img[20:250,20:250]       
    imCopy = img1.copy()
    gray = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
    blur = cv2.GaussianBlur(gray, (5, 5), 0)
    ret, thresh1 = cv2.threshold(blur, 10, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
    hand_resize = cv2.resize(thresh1, (width, height))
    cv2.imshow("Threshold", thresh1)
    contours, hierarchy = cv2.findContours(thresh1, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    cv2.drawContours(imCopy, contours, -1, (0, 255, 0))
    cv2.imshow('Draw Contours', imCopy)
    k = 0xFF & cv2.waitKey(10)
    if k == 27:
        break
cap.release()
cv2.destroyAllWindows()

Run the code in your Powershell/terminal using

python check.py

To take input for static images, use the follwing code:

import cv2
img = cv2.imread('abc.jpg')
img2 = cv2.resize(img, (200, 400))
cv2.imshow("Image", img2)
interrupt = cv2.waitKey(0) & 0xFF
if interrupt == 27:
    cv2.destroyAllWindows()

Run the code in your Powershell/terminal using

python check.py

ASSIGNMENT 1: Use any 3 functions/modules in OpenCV, and commit the code in the official Expressando TDoC 2021 Repository.

The Github repository where you all shall be pushing your codes on the respective projects assigned is ready.

Link to the repo: Github Repo

You are also asked to follow the tutorial video attached below which will clearly describe how you shall push your code specifically for the TDoC event.

Link to the video: TDoC Instruction Video

Here is also an introductory video on basics of Git & Github so that you are well versed with the git system.

Link to the video: Git and Github

Checking for Convexity Defects in the Camera Input

Since the initial input has been configured through the webcam input, it becomes important to understand the concepts of "defect" as a basic and fundamental method in the domain of detection. In this session, we are going to learn about defects and detect them in our digital video input.

Step 1: Create a file named as 'defects.py' inside the directory of 'TDoC-2021'. As the name suggests, we are checking for the defects in images taken by webcam using the OpenCV library. Open the file your code-editor/IDE. The folder structure would look like the following:

├── TDoC-2021
|     ├── env
|     ├── check.py
|     ├── defects.py
|     ├── requirements.txt

Step 2: First import OpenCV, NumPy (as np), and math into the 'defects.py' file. Here, math is present in the standard Python library, and it need not to be installed separately. It can be accomplished by the following line of code:

import cv2
import numpy as np
import math

After importing the packages, we need to create a VideoCapture object, which will initiate the process to retrieve the input through the webcam.

cap = cv2.VideoCapture(0)

Step 3: The next step involves rendering a while loop to stimulate asynchronous input through the webcam with the help of a suitable condition. In this step, we will be discussing the most common and important methods that are present in the OpenCV library, which are required for making basic projects and develop sound understanding about the various methods present in OpenCV and their uses.
Continue in the code-editor as follows:

while (cap.isOpened()):
    ret, img = cap.read()
    img=cv2.flip(img, 1)
    cv2.rectangle(img,(20,20),(250,250),(255,0,0),3)
    crop_img = img[20:250, 20:250]
    cv2.imshow('Gesture', img)

    grey = cv2.cvtColor(crop_img, cv2.COLOR_BGR2GRAY)
    value = (35, 35)
    blurred = cv2.GaussianBlur(grey, value, 0)
    _, thresh1 = cv2.threshold(blurred, 127, 255, cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)
    cv2.imshow('Binary Image', thresh1)

The above lines of the code is just a recap of what we did in Day 3. (REFER TO THE DOCUMENTATION OF DAY-3). Here, we initialise a while loop, which iterates as long as the webcam input returns a frame or cap.isOpened() returns True value. The cap.read() takes the input of the image in the form of an image array vector. The flip() function basically returns the inverted image of the frame taken into consideration. We define a region by means of the rectangle() function, and then extract the region, naming it as crop_img. It is shown by the name "Gesture", using the function imshow().

Then we apply the cvtColor() function and convert the image into it's equivalent grayscale using cv2.COLOR_BGR2GRAY method and name it as grey. Next we declare the tuple value, which contains the kernel standard deviation for x and y coordinates. This tuple is later used in the GaussianBlur() function, where it is used as a parameter. The blurred is image is named as blurred. Then we apply simple threshold using the modules cv2.THRESH_BINARY_INV and cv2.THRESH_OTSU, and naming the resultant image as thresh1. It is shown by the name "Binary Image", using the function imshow().

Step 4:

    contours, hierarchy = cv2.findContours(thresh1.copy(),cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    cnt = max(contours, key = lambda x: cv2.contourArea(x))
    x,y,w,h = cv2.boundingRect(cnt)
    cv2.rectangle(crop_img,(x,y),(x+w,y+h),(0,0,255),2)

Next, we derive the contours from the threshold using cv2.RETR_TREE as the retrieval method and cv2.CHAIN_APPROX_NONE as the approximation method. We then, store the contours in the array named contours, while the hierarchy order is stored in hierarchy. We define a list called "cnt", which stores the external contour with the maximum area enclosed by it. This refers to the area of the object, as it will have the maximum area under consideration. The function contourArea() returns the area enclosed by the contour, and the max() function returns the enclosed contour with the maximum area.

The key used here is: lambda, which is a constant unit vector, used to determine the direction and order of the contours. (ADVANCED)
Go through the following resources to know more about Thresholding and Contours:

cv2.boundingRect() is a function used to create an approximate rectangle of minimum area which encloses the object/contour that is passed into the function as a parameter. This function’s primary use is to highlight the area of interest after obtaining the image’s outer shape, or the external contour. With proper markings, the users can easily highlight the desired aspect in an image. It ensures clear focus and better understanding of the operations.

cv2.boundingrect() returns 4 numeric values when the contour is passed as an argument. These 4 values correspond to x, y, w, h respectively. These values are more described as –

x - X coordinate of the contour, closest to the origin (Top left of the window)
y - Y coordinate of the contour, closest to the origin (Top left of the window)
w - Width of the rectangle which will enclose the contour.
h - Height of the rectangle which will enclose the contour.

The above shows the use of boundingrect() function to enclose all the 3 shapes in the figure.

Next we draw a rectangle using the rectangle() function, with the coordinates from (x,y) to (x+w,y+h) as the diagonal over the crop_img. This serves as an enclosure to the contours.

Step 5:

    hull = cv2.convexHull(cnt)
    drawing = np.zeros(crop_img.shape,np.uint8)

    cv2.drawContours(drawing,[cnt],0,(0,255,0),0)
    cv2.drawContours(drawing,[hull],0,(0,0,255),0)
    cv2.imshow('Contours', drawing)

What is a 'Convex Hull'?

A Convex object is one with no interior angles greater than 180 degrees. A shape that is not convex is called Non-Convex or Concave. Hull means the exterior or the shape of the object. Therefore, the Convex Hull of a shape or a group of points is a tight fitting convex boundary around the points or the shape. Any deviation of the object from this hull can be considered as convexity defect.

This is an example of a convex hull.

How to display the Convex Hull ?

OpenCV provides a function convexHull() which stores all the points of the hull in the form of list/array of points, on passing cnt as the contour array.
The next line of the program makes use of a NumPy array to store the crop_img, and using the function np.zeroes(), it converts the entire image to black.
Here, we have used black background to clearly visualise the contours. np.uint8 is an 8-bit unsigned integer basically, used to define the source of image.

Then we use the drawContours() function to draw the contour and the hull using green and red colours respectively, over the image "drawing", which is the black coloured background of the same size as "crop_img". Then we show the output under the name "Contours", using the function imshow().

Step 6: Next, we have to detect the defects by making use of the Convex Hull.

What are 'Convexity Defects'?

Any deviation of the contour from its convex hull is known as the convexity defect.
OpenCV provides a function cv2.convexityDefects() for finding the convexity defects of a contour. This takes as input, the contour and its corresponding hull indices and returns an array containing the convexity defects as output.

This figure shows the depiction of the hull, contours and the defect.

hull = cv2.convexHull(cnt,returnPoints = False)
defects = cv2.convexityDefects(cnt,hull)

We redeclare hull with an extra parameter returnpoints=False. This will give us the indices of the contour points that make the hull. The function convexityDefects() is used to find the defects directly by passing the contours array (cnt) and the hull.

Convexity Defects returns an array where each row contains these values :

start point as 's'
end point as 'e'
farthest point as 'f'
approximate distance to farthest point as 'd'

Step 7: Now we use some mathematical expressions to determine the number of convexity defects in the hull, and count them accordingly.

for i in range(defects.shape[0]):
        s,e,f,d = defects[i,0]
        start = tuple(cnt[s][0])
        end = tuple(cnt[e][0])
        far = tuple(cnt[f][0])
        a = math.sqrt((end[0] - start[0])**2 + (end[1] - start[1])**2)
        b = math.sqrt((far[0] - start[0])**2 + (far[1] - start[1])**2)
        c = math.sqrt((end[0] - far[0])**2 + (end[1] - far[1])**2)
        angle = math.acos((b**2 + c**2 - a**2)/(2*b*c)) * 57
        if angle <= 90:
            count_defects += 1
            cv2.circle(crop_img,far,1,[0,0,255],-1)
        cv2.line(crop_img,start,end,[0,255,0],2)

Here, defects returns an array where each row contains these values [start point, end point, farthest point, approximate distance to farthest point] , i.e. (s,e,f,d). We segment and store each of them as a separate independent 2D array, with y-coordinate as 0. These arrays are later converted into a tuple of coordinates, and they are named as start, end and far. Here, start and end points lie on the contour, whereas far points lie on the hull. We, then use the basic distance formula to calculate the lengthd of a, b and c.

Now, this is Math time! Let’s understand the cosine theorem.
In trigonometry, the law of cosines relates the lengths of the sides of a triangle to the cosine of one of its angles. Using notation as in the given figure, the law of cosines states where gamma denotes the angle contained between sides of lengths a and b and opposite the side of length c.

The formula for the same is given below:

By seeing this formula now we understand that if we have the parameters a, b and c, then we can find gamma, the angle between the sides a and b.

For finding gamma, the following formula is used:

The pictorial depiction of the following would look like the following:

Now, gamma is always less than or equal to 90 degrees (maximum), So we can say: If gamma is less than 90 degree or pi/2, we consider it as a finger.

By this point, we can easily derive the three sides: a, b, c (see CODE) and from the cosine theorem we can derive gamma or angle between two fingers. As you read earlier, if gamma is less than 90 degree we treated it as a finger.

We convert gamma into degrees by multiplying with 57, as acos() function returns the angle in radians.
We then check if the angle is less than or equal to 90 degrees, and if it is true, we increase the value of count_defects by 1. The existence of an angle less than 90 denotes the presence of defects.

After knowing gamma we just draw circle with radius 1 in approximate distance to farthest points. The far points are denoted by the line drawn by cv2.line(). The circle drawn would not be uniform as the farthest points are not present in a straight line. Next, we display the number of defects using the function cv2.putText().

The parameters of cv2.circle() are:

img: It is the image on which circle is to be drawn.
far: It is the center coordinates of circle. The coordinates are represented as tuples of two values i.e. (X coordinate value, Y coordinate value).
1: It is the radius of circle.
[0,0,255]: It is the color of border line of circle to be drawn in BGR index.
-1: It is the thickness of the circle border line in px. Thickness of -1 px will fill the circle shape by the specified color.

The parameters of cv2.line() are:

crop_img: It is the image on which line is to be drawn.
start: It is the starting coordinates of line. The coordinates are represented as tuples of two values i.e. (X coordinate value, Y coordinate value).
end: It is the ending coordinates of line. The coordinates are represented as tuples of two values i.e. (X coordinate value, Y coordinate value).
[0,255,0]: It is the color of border line of circle to be drawn in BGR index.
2: It is the thickness of the circle border line in px.

The parameters of cv2.putText() are:

img: It is the image on which text is to be drawn.
"Number : 2": It is the text string to be drawn on the image.
(50,450): It is the coordinates of the bottom-left corner of the text string in the image. The coordinates are represented as tuples of two values i.e. (X coordinate value, Y coordinate value).
cv2.FONT_HERSHEY_SIMPLEX: It denotes the font type, used in OpenCV.
1: It is the fontScale factor that is multiplied by the font-specific base size.
(255,255,255): It is the color of text string to be drawn in BGR. Here, the colour is white.
1: It is the thickness of the line in px.

The fonts available in OpenCV are:

FONT_HERSHEY_SIMPLEX
FONT_HERSHEY_PLAIN
FONT_HERSHEY_DUPLEX
FONT_HERSHEY_COMPLEX
FONT_HERSHEY_TRIPLEX
FONT_HERSHEY_COMPLEX_SMALL
FONT_HERSHEY_SCRIPT_SIMPLEX
FONT_HERSHEY_SCRIPT_COMPLEX

If there are n defects, then there exists n-1 fingers under detection.

    if count_defects == 1:
        cv2.putText(img,"Number : 2", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    elif count_defects == 2:
        cv2.putText(img,"Number : 3", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    elif count_defects == 3:
        cv2.putText(img,"Number : 4", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    elif count_defects == 4:
        cv2.putText(img,"Number : 5", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    elif count_defects == 5:
        cv2.putText(img,"Number : 6", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    else:
        cv2.putText(img,"Number : 1", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)

    cv2.imshow('Defects', crop_img)

The number of defects will be displayed as follows, which will be rendered by the name of "Defects", using imshow():

Step 8: Now, after checking for the defects, it is time to proceed for the termination of the while loop and close all the windows and close our Video Capture object.

To exit the program on a specified keyboard interrupt, type the following code:

    k = cv2.waitKey(10) & 0xFF
    if k == 27:
      break

cap.release()
cv2.destroyAllWindows()

Now the loop breaks when the key is entered, and exits the control out of the loop.

To know more about Convexity Defects, go here: Convexity Defects

Now your defects.py should look like the following.

import cv2
import numpy as np
import math
cap=cv2.VideoCapture(0)
while(cap.isOpened()):
    ret, img = cap.read()
    img=cv2.flip(img, 1)
    cv2.rectangle(img,(20,20),(250,250),(255,0,0),3)
    crop_img = img[20:250, 20:250]
    cv2.imshow('Gesture', img)

    grey = cv2.cvtColor(crop_img, cv2.COLOR_BGR2GRAY)
    value = (35, 35)
    blurred = cv2.GaussianBlur(grey, value, 0)
    _, thresh1 = cv2.threshold(blurred, 127, 255, cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)
    cv2.imshow('Binary Image', thresh1)

    contours, hierarchy = cv2.findContours(thresh1.copy(),cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    cnt = max(contours, key = lambda x: cv2.contourArea(x))
    x,y,w,h = cv2.boundingRect(cnt)
    cv2.rectangle(crop_img,(x,y),(x+w,y+h),(0,0,255),0)
    hull = cv2.convexHull(cnt)
    drawing = np.zeros(crop_img.shape,np.uint8)
    cv2.drawContours(drawing,[cnt],0,(0,255,0),0)
    cv2.drawContours(drawing,[hull],0,(0,0,255),0)
    cv2.imshow('Contours', drawing)

    hull = cv2.convexHull(cnt,returnPoints = False)
    defects = cv2.convexityDefects(cnt,hull)
    count_defects = 0
    cv2.drawContours(thresh1, contours, -1, (0,255,0), 3)
    for i in range(defects.shape[0]):
        s,e,f,d = defects[i,0]
        start = tuple(cnt[s][0])
        end = tuple(cnt[e][0])
        far = tuple(cnt[f][0])
        a = math.sqrt((end[0] - start[0])**2 + (end[1] - start[1])**2)
        b = math.sqrt((far[0] - start[0])**2 + (far[1] - start[1])**2)
        c = math.sqrt((end[0] - far[0])**2 + (end[1] - far[1])**2)
        angle = math.acos((b**2 + c**2 - a**2)/(2*b*c)) * 57
        if angle <= 90:
            count_defects += 1
            cv2.circle(crop_img,far,1,[0,0,255],-1)
        cv2.line(crop_img,start,end,[0,255,0],2)
    if count_defects == 1:
        cv2.putText(img,"Number : 2", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    elif count_defects == 2:
        cv2.putText(img, "Number : 3", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    elif count_defects == 3:
        cv2.putText(img,"Number : 4", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    elif count_defects == 4:
        cv2.putText(img,"Number : 5", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    elif count_defects == 5:
        cv2.putText(img,"Number : 6", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    else:
        cv2.putText(img,"Number : 1", (50,450), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 1)
    cv2.imshow('Defects', crop_img)

    k = cv2.waitKey(10)
    if k == 27:
        break

cap.release()
cv2.destroyAllWindows()

Run the code in your Powershell/terminal using

python check.py

In the part 2 of this post, you will be learning about Collecting data through OpenCV, Demonstration of Data-Collection and TensorFlow, Convolutional Neural Networks (CNN) any many more things. You will also learn about how live prediction works and all the things. Stay tuned for part 2!

Project collaborators:

PritamFollow

Full-stack web developer || Network Security Admin of @editechstudio || Member of @nitdgplug || B.Tech. NIT Durgapur (2020-2024)

Ritabrata DasFollow

Open-Source Fanatic || Full-Stack Developer || Graphic Designer || C.P. Enthusiast || CTF Player

Amool-kkFollow

Full-stack web developer || Member of @nitdgplug || B.Tech. NIT Durgapur (2020-2024)

**May the source be with you! 🐧❤️**

Expressando (Part 2): A real-time sign language detection system

Amool Kuldiya — Fri, 21 Jan 2022 20:49:36 +0000

This is a follow-up article from our previous post.

Expressando (Part 1): A real-time sign language detection system

Pritam for GNU/Linux Users' Group, NIT Durgapur ・ Jan 22 '22

#machinelearning #opencv #tensorflow #python

In the previous post, we discussed the features like What is a 'Virtual Environment'?, Configuring Input through Webcam using OpenCV, Checking for Convexity Defects in the Camera Input, What is a 'Convex Hull'? How to display the Convex Hull ?, What are 'Convexity Defects'?. Today, you will be learning about Collecting data through OpenCV, Demonstration of Data-Collection and TensorFlow, Convolutional Neural Networks (CNN), and many more things. You will also learn about how to live prediction works and all the things.

Collecting data through OpenCV and labeling them

Create a file named collect-data.py inside the directory of TDoC-2021. As the name suggests, collecting data collects the images taken by webcam using the OpenCV library. creating one or more dataset(s) depending on the input, storing them in the directories, and pursuing them according to classes.

First, activate your virtual environment (This is very important as OpenCV is installed inside the virtual environment and not globally). Then import OpenCV and OS (included in python library) as cv2 and os.


import cv2

import os

We will be checking for the presence of the data directory, which will be used to store the train and test data. If the data folder is already created then, it will not create the folders, and proceed to store the data in the form of datasets, each representing a class.


if not os.path.exists("data"): #True

  os.makedirs("data")

  os.makedirs("data/train") 

  os.makedirs("data/test")

  os.makedirs("data/train/0") 

  os.makedirs("data/train/1")

  os.makedirs("data/train/2")

  os.makedirs("data/train/3")

  os.makedirs("data/train/4")

  os.makedirs("data/train/5")

  os.makedirs("data/test/0")

  os.makedirs("data/test/1")

  os.makedirs("data/test/2")

  os.makedirs("data/test/3")

  os.makedirs("data/test/4")

  os.makedirs("data/test/5")

The if not statement checks if the directory data exists or not, whereas the os.path.exists method returns Boolean value & depending on which next lines are executed. Next if os.path.exists returns False then os.makedirs method will create the necessary directories.

Now we will create some variables for the path of the folders where we will save our dataset of images. After that, we need to collect the required images via webcam. For that, we will use the already learned how to use the cv2.VideoCapture() method to initialize.

To make the process of collecting the data more interactive and visually appealing, we will display some statistics in the live view screen of the webcam. So to know how many images we have collected inside the folder, we can simply use these key bindings and record them as count objects and keep the counts of images as a dictionary for every number. we can display it in the live view of the webcam. It can be done very easily, made possible by OpenCV. This can be achieved by the following lines of code:


cv2.putText(frame, "MODE : "+mode, (30, 50), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 1)

cv2.putText(frame, "IMAGE COUNT", (10, 100), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 1)

cv2.putText(frame, "ZERO : "+str(count['zero']), (10, 120), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "ONE : "+str(count['one']), (10, 140), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "TWO : "+str(count['two']), (10, 160), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "THREE : "+str(count['three']), (10, 180), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "FOUR : "+str(count['four']), (10, 200), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "FIVE : "+str(count['five']), (10, 220), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1

Next, we will learning about the Region of Interest (R.O.I).A region of interest (ROI) is an area of an image defined for further analysis or processing. Here, the region of interest will basically contain the region of the hand, used for portraying the gesture. We will be defining the region of interest using the following code:


x1 = int(0.5*frame.shape[1])

y1 = 10

x2 = frame.shape[1]-10

x1 = int(0.5*frame.shape[1])

y1 = 10

x2 = frame.shape[1]-10

y2 = int(0.5*frame.shape[1])

cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,0,0) ,3)

roi = frame[y1:y2, x1:x2] 

roi = cv2.resize(roi, (200, 200)) 

cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)

cv2.imshow("Frame", frame)



roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

_, roi = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

cv2.imshow("ROI", roi)  y2 = int(0.5*frame.shape[1])

cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,0,0) ,3)

roi = frame[y1:y2, x1:x2] 

roi = cv2.resize(roi, (200, 200)) 

cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)

cv2.imshow("Frame", frame)



roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

_, roi = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

cv2.imshow("ROI", roi)

Here, frame.shape[1] returns the shape of the frame or the camera input. Next, we will be defining the variables x1, y1, x2, and y2, which will serve as the diagonals of the rectangle (x1,y1) and (x2,y2). Next, we draw a rectangle surrounding the ROI using the rectangle() function, so that we can record our gesture inside the rectangle. After extraction, we resize and enlarge the region of interest using the resize() function.

we convert the ROI into its grayscale form, using the module cv2.COLOR_BGR2GRAY. After converting the ROI into grayscale, we will apply a simple threshold using the function cv2.threshold(). We will be using the cv2.THRESH_BINARY module to apply the threshold.

Next, we will be recording the images in the dataset according to the key bindings. We first initialize the interrupt variable, which returns 10 frames per second here. Then we define all the key bindings. Whenever, we press 0, data for the 'zero' key will get recorded. Similarly, it applies to the other datasets as well.

Now your collect-data.py should look like the following.


import cv2

import os 



if not os.path.exists("data"): #True

  os.makedirs("data")

  os.makedirs("data/train") 

  os.makedirs("data/test")

  os.makedirs("data/train/0") 

  os.makedirs("data/train/1")

  os.makedirs("data/train/2")

  os.makedirs("data/train/3")

  os.makedirs("data/train/4")

  os.makedirs("data/train/5")

  os.makedirs("data/test/0")

  os.makedirs("data/test/1")

  os.makedirs("data/test/2")

  os.makedirs("data/test/3")

  os.makedirs("data/test/4")

  os.makedirs("data/test/5")





mode = 'train' 

directory = 'data/'+mode+'/' #data/train/



cap=cv2.VideoCapture(0)



while True:

  _, frame = cap.read()

  frame = cv2.flip(frame, 1)



  cv2.putText(frame, "Expressando - TDOC 2021", (175, 450), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 3)



  count = {'zero': len(os.listdir(directory+"/0")), 

       'one': len(os.listdir(directory+"/1")),

       'two': len(os.listdir(directory+"/2")),

       'three': len(os.listdir(directory+"/3")),

       'four': len(os.listdir(directory+"/4")),

       'five': len(os.listdir(directory+"/5"))} 



  cv2.putText(frame, "MODE : "+mode, (30, 50), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 1)

  cv2.putText(frame, "IMAGE COUNT", (10, 100), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 1)

  cv2.putText(frame, "ZERO : "+str(count['zero']), (10, 120), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "ONE : "+str(count['one']), (10, 140), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "TWO : "+str(count['two']), (10, 160), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "THREE : "+str(count['three']), (10, 180), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "FOUR : "+str(count['four']), (10, 200), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "FIVE : "+str(count['five']), (10, 220), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)





  x1 = int(0.5*frame.shape[1])

  y1 = 10

  x2 = frame.shape[1]-10

  y2 = int(0.5*frame.shape[1])

  cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,0,0) ,3)

  roi = frame[y1:y2, x1:x2] 

  roi = cv2.resize(roi, (200, 200)) 

  cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)

  cv2.imshow("Frame", frame)



  roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

  _, roi = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

  cv2.imshow("ROI", roi)



  interrupt = cv2.waitKey(10) 

  if interrupt & 0xFF == 27:

    break

  if interrupt & 0xFF == ord('0'):

    cv2.imwrite(directory+'0/'+str(count['zero'])+'.jpg', roi)

  if interrupt & 0xFF == ord('1'):

    cv2.imwrite(directory+'1/'+str(count['one'])+'.jpg', roi)

  if interrupt & 0xFF == ord('2'):

    cv2.imwrite(directory+'2/'+str(count['two'])+'.jpg', roi)

  if interrupt & 0xFF == ord('3'):

    cv2.imwrite(directory+'3/'+str(count['three'])+'.jpg', roi)

  if interrupt & 0xFF == ord('4'):

    cv2.imwrite(directory+'4/'+str(count['four'])+'.jpg', roi)

  if interrupt & 0xFF == ord('5'):

    cv2.imwrite(directory+'5/'+str(count['five'])+'.jpg', roi)



cap.release()

cv2.destroyAllWindows()

Demonstration of Data-Collection and Introduction to TensorFlow

Now, we are going to have a demonstration on how to collect data with increased precision. This is a vital step, as it governs the future processes which are required in making the model.

What is a 'Dataset'?

A Dataset is a collection of data that is treated as a single unit by a computer. This means that a dataset contains a lot of separate pieces of data/images but can be used to train an algorithm with the goal of finding predictable patterns inside the whole dataset.

Datasets are of three types:

Training Dataset - This dataset which contains the necessary data with which the model is to be trained. This dataset contains the maximum amount of data, and it is primarily used by the machine for comparison.
Validation Dataset - This dataset checks for the validation of the input in accordance with the training dataset. It is a subset of the training dataset. It eliminates unnecessary inputs and increases the speed of operation.
Test Dataset - This dataset is used for testing the models and determining the accuracy and losses which are incurred while training the machine. This is very much important, as it serves as a test set for the training data, and the user can verify the results.

Now let us go through a quick example of how to collect data:

Run the collect-data.py file using the command:


python collect-data.py

The window will open, taking a reference input from the webcam. It will also display a thresholded image of the area under the rectangle (Region of Interest) in another window. This image will be recorded under the respective directories of the dataset.

Let us first begin with the "zero" gesture, which basically involves a closed fist. Bring the gesture close inside the Region of Interest, as much that it is completely enclosed within the rectangle, and covers most of the area of the region.
Make sure, you have a clear background and do not let unexpected disturbances appear inside the rectangle/ROI. Try to collect clear data as much as possible. Make sure all your fingers are visible.
Then, start collecting the data by pressing '0' of the Numpad. This will capture the images, which will be the images stored in the dataset under the train directory in the 0 folder.
Collect approximately 100-120 images to train the model. Make sure to collect data, while changing your orientation over the rectangle, so that the machine can recognize the gesture whenever you show the gesture as input inside the rectangle.

Similarly for other datasets.

Now, as you have collected the data, now let us see what TensorFlow has in store for us.

TensorFlow

Implementing machine learning is a complex stream mainly the concerns related to the creation of real-time models. Hence the end-to-end open-source framework - TensorFlow is used for collecting datasets, training systems through models and providing the results based on it.

TensorFlow is an open-source framework created by Google Creative Labs. It executes machine learning and helps to build neural networks easily (along with Keras) in Python as well as in Javascript. It builds an environment of networks to experiment with the algorithms of machine learning and visualizes it using flow graphs. The graphs represent the progression of all nodes where nodes are the operations in the model.

Why TensorFlow

TensorFlow provides pre-built functions and advanced operations API to ease the task of building different neural network models. It provides the required infrastructure and hardware which makes them one of the leading libraries used extensively by researchers and students in the deep learning domain.

To begin with Tensorflow, you can go through the following tutorials:

Convolutional Neural Networks (CNN)

Artificial Intelligence has been witnessing a monumental growth in bridging the gap between the capabilities of humans and machines. Researchers and enthusiasts alike, work on numerous aspects of the field to make amazing things happen. One of many such areas is the domain of Computer Vision.

The agenda for this field is to enable machines to view the world as humans do, perceive it in a similar manner, and even use the knowledge for a multitude of tasks such as Image & Video recognition, Image Analysis & Classification, Media Recreation, Recommendation Systems, Natural Language Processing, etc. The advancements in Computer Vision with Deep Learning have been constructed and perfected with time, primarily over one particular algorithm — Convolutional Neural Network.

To know more about CNN go through the following link: https://deepai.org/machine-learning-glossary-and-terms/convolutional-neural-network

Now let us go through the code:

Create a file named as 'train_model.py' inside the directory of 'TDoC-2021'. As the name suggests, it will be training the models using TensorFlow and Keras with the Convolutional Neural Network being trained on it.

At first, we will import the model and layers, which we will be using from Keras. Here, Keras uses TensorFlow as it's backend, i.e, the functions in Keras also makes uses of the functions in TensorFlow for all of the results.

Here, we will be making use of the Sequential model. This model is used when you have one input, and one output at a time. This model basically stacks the layers and serves as a container for the layers in connection.

To know more about the Sequential model, go here:

The process of building a Convolutional Neural Network always involves four major steps.

Convolution
Pooling
Flattening
Full Connection

Now, we will create an object of the sequential class below:


from keras.models import Sequential

from keras.layers import Convolution2D, MaxPooling2D, Flatten, Dense



classifier = Sequential()

Next, we will define the first convolutional layer. This will modulate the image input tensor and result in convoluted matrices. The matrices produced will be merged and amalgamated into a single matrix. This basically consists of mathematical operands, each of which serves as a node. The operands are decided on the basis of image size, color, and characteristics. After the convolution, we will be Max-Pooling the matrix, so that the size of the matrix is reduced, and it will help in the efficient determination. It will be achieved by the following code:


classifier.add(Convolution2D(32, (3, 3), input_shape=(64, 64, 1), activation='relu'))

classifier.add(MaxPooling2D(pool_size=(2, 2)))

Here, the add() function helps to add successive layers in the Convolution Neural Network object.

We will be using two types of activation function:

relu: It stands for Rectified Linear Unit. It helps in the independent activation of neurons/nodes in the layers.
softmax: It helps to activate the neurons if there are multiple branching present, and it requires detecting simultaneous figures.

To know more about activation functions, go here: https://www.v7labs.com/blog/neural-networks-activation-functions

Similarly, we will be adding a second Convolutional Layer to it, in order to increase the efficiency of detection. However, we cannot add the infinite number of layers, as it would decrease the speed and also, can result in wrong detections.

Next, we will be flattening the layers, and connecting them to produce our entire Neural Network. Flattening is a very important step to understand. What we are basically doing here is taking the 2-D array, i.e pooled image pixels, and converting them to a one-dimensional single vector. We’ve used flatten function to perform flattening, we no need to add any special parameters, Keras will understand that the “classifier” object is already holding pooled image pixels and they need to be flattened.


classifier.add(Flatten())

Now, we need to create a fully connected layer, and to this layer, we are going to connect the set of nodes we got after the flattening step, these nodes will act as an input layer to these fully-connected layers. As this layer will be present between the input layer and output layer, we can refer to it as a hidden layer.

Dense() is the function to add a fully connected layer. Units is where we define the number of nodes that should be present in this hidden layer, these units value will be always between the number of input nodes and the output nodes but the art of choosing the most optimal number of nodes can be achieved only through experimental tries. Though it’s a common practice to use a power of 2 and the activation function will be a rectifier function.


classifier.add(Dense(units=128, activation='relu'))

classifier.add(Dense(units=6, activation='softmax'))

As, the model has been prepared, you need to compile and compose it.


classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

To know more about Adam Optimisation, go here: https://www.geeksforgeeks.org/intuition-of-adam-optimizer/

Next, we will import our datasets: train and test, and then will pass them through the model for compilation, and ultimately compose the model on the basis of the images fed into them.


from keras.preprocessing.image import ImageDataGenerator



train_datagen = ImageDataGenerator(

    rescale=1./255,

    shear_range=0.2,

    zoom_range=0.2,

    horizontal_flip=True)



test_datagen = ImageDataGenerator(rescale=1./255)



training_set = train_datagen.flow_from_directory('data/train',

                         target_size=(64, 64),

                         batch_size=5,

                         color_mode='grayscale',

                         class_mode='categorical')



test_set = test_datagen.flow_from_directory('data/test',

                      target_size=(64, 64),

                      batch_size=5,

                      color_mode='grayscale',

                      class_mode='categorical')

Here, we will be using the ImageDataGenerator. It is used to manipulate the image input, and convert them into a machine-readable format using the CNN building. We will be first defining the Data Generators for the test and train data individually, and naming them as test_datagen and train_datagen. We will be using the rescale parameter to rescale the images, zoom_range and shear_range to adjust the image identification, so that when the image can be still be recognized under zoomed conditions up to the scale of 0.2. At last, horizontal_flip helps to check for a correct orientation for the image.

After the data generators are created, we need to generate the training_set and test_set for making our model, and testing it. We define the directory, color-space (grayscale), and the class mode(which can be binary or categorical.) We also define the input size of the image as 64 x 64. The batch size refers to the size of the batches, which is used here as 5. Batch is a continuous length of data, that is passed as once to the model for training.

After the datasets are prepared to feed into the model, we fit them into the model layers, for analysis and metrics calculation.


classifier.fit_generator(

    training_set,

    epochs=10,

    validation_data=test_set)

Here, we define the sets, that have been generated in the previous steps. We will be testing the training_set against the test_set, and determine the accuracy and losses. We will pass them in 10 epochs. Epoch is the number of passes of the entire training dataset the machine learning algorithm has completed. It is similar to test cases, which the training set needs to pass. These check for valid input and test the accuracy and losses.

After the model has been prepared and tested, it is necessary to save the model in the form of hierarchial data-model. We will also save the model in the form of JSON format (JavaScript Object Notation). We are saving it in the JSON format so that the user can know about the model composition and characteristics.


model_json = classifier.to_json()

with open("model-bw.json", "w") as json_file:

  json_file.write(model_json)

classifier.save_weights('model-bw.h5')

Now your train_model.py should look like the following.


from keras.models import Sequential

from keras.layers import Convolution2D, MaxPooling2D, Flatten, Dense



classifier = Sequential()



classifier.add(Convolution2D(32, (3, 3), input_shape=(64, 64, 1), activation='relu'))

classifier.add(MaxPooling2D(pool_size=(2, 2)))



classifier.add(Convolution2D(32, (3, 3), activation='relu'))

classifier.add(MaxPooling2D(pool_size=(2, 2)))



classifier.add(Flatten())



classifier.add(Dense(units=128, activation='relu'))

classifier.add(Dense(units=6, activation='softmax'))



classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])





from keras.preprocessing.image import ImageDataGenerator



train_datagen = ImageDataGenerator(

    rescale=1./255,

    shear_range=0.2,

    zoom_range=0.2,

    horizontal_flip=True)



test_datagen = ImageDataGenerator(rescale=1./255)



training_set = train_datagen.flow_from_directory('data/train',

                         target_size=(64, 64),

                         batch_size=5,

                         color_mode='grayscale',

                         class_mode='categorical')



test_set = test_datagen.flow_from_directory('data/test',

                      target_size=(64, 64),

                      batch_size=5,

                      color_mode='grayscale',

                      class_mode='categorical') 



classifier.fit_generator(

    training_set,

    epochs=10,

    validation_data=test_set)



model_json = classifier.to_json()

with open("model-bw.json", "w") as json_file:

  json_file.write(model_json)

classifier.save_weights('model-bw.h5')

Run the code in your Powershell/terminal using:


python train_model.py

Live Prediction of Customised Sign Language

Create a file named as 'prediction.py' inside the directory of 'TDoC-2021'. As the name suggests, it will be predicting the customized sign with which we have trained the model, when we give a similar gesture to what we have trained it. First activate your virtual environment. Also, make sure you have the model-bw.h5 and model-bw.json Prepared before running the code.

Next, we will be importing cv2, operator(an inherent module present in the standard Python library), and model_from_json, which is to be imported to read the model content in the JSON file.


from keras.models import model_from_json

import operator

import cv2

Here, we will be declaring the object json_file, which will parse the model-bw.json present in the same directory as the prediction.py file. Next, we will be parsing the contents from the json_file, which has the model-bw.json already passed in it.

Then, we use the read() and close() function to store the contents in the file in the object model_json.

Next we will load the data model and JSON model simultaneously using the load_weights() and model_from_json() function respectively. When the models are successfully loaded, it will print "Loaded model from disk" in the terminal/Powershell.

Next, we have initialized a Video-Capture object called "cap". Then, we have declared a dictionary called categories, having integer numbers as their key index. This will be helpful in rendering the string value of their respective numbers, which will be shown in the prediction.


json_file = open("model-bw.json", "r")

model_json = json_file.read()

json_file.close()

loaded_model = model_from_json(model_json)

loaded_model.load_weights("model-bw.h5")

print("Loaded model from disk")

cap = cv2.VideoCapture(0) 



categories = {0: 'ZERO', 1: 'ONE', 2: 'TWO', 3: 'THREE', 4: 'FOUR', 5: 'FIVE'}

Then, we will initialize a while-loop which will run infinitely, as long as the video is being captured, or a break statement has been encountered. Then we read the frames of the video using cap.read() function, and invert the reading frame using the flip() function.


while True:

  _, frame = cap.read()

  frame = cv2.flip(frame, 1)



  x1 = int(0.5*frame.shape[1])

  y1 = 10

  x2 = frame.shape[1]-10

  y2 = int(0.5*frame.shape[1])



  cv2.putText(frame, "Expressando - TDOC 2021", (175, 450), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,0), 3)

  cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,255,255) ,3)

  roi = frame[y1:y2, x1:x2]



  roi = cv2.resize(roi, (64, 64)) 

  roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

  cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)



  _, test_image = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

  cv2.imshow("ROI", test_image)

Next, we will be defining the Region of Interest, where we will be showing our hand gestures. We have previously discussed how to define the region of interest in our previous documentation. Then, we will be using the rectangle() function to enclose the region of interest, within the rectangle for the user to know where the sign will be detected.


  result = loaded_model.predict(test_image.reshape(1, 64, 64, 1))

  prediction = {'ZERO': result[0][0], 

         'ONE': result[0][1], 

         'TWO': result[0][2],

         'THREE': result[0][3],

         'FOUR': result[0][4],

         'FIVE': result[0][5]} 

  prediction = sorted(prediction.items(), key=operator.itemgetter(1), reverse=True)

  cv2.putText(frame, "PREDICTION:", (30, 90), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)

  cv2.putText(frame, prediction[0][0], (80, 130), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)   

  cv2.imshow("Frame", frame)

We will use the predict() function, which enables us to predict the labels of the data values on the basis of the trained model. It accepts only a single argument which is usually the data to be tested. It returns the labels of the data passed as an argument based upon the learned or trained data obtained from the model. Thus, the predict() function works on top of the trained model and makes use of the learned label to map and predict the labels for the data to be tested. We will pass the test_image, reshaped to the input size which we will be passing into the Convolutional Neural Network, which is 64 x 64 x 1.

Then, we will declare a dictionary called prediction, which has the integer name in the string as the key, which in turn returns the result[0][key]. Here result[0] refers to the first element in the list, which is the categories dictionary. Then, we parse the results depending on the key value in result[0][key].

Next, we are sorting the prediction.items(), which are the individual probabilities that the input gesture you are showing in the Region of Interest (R.O.I) resembles the images/dataset with which you have trained the models. The operator.itemgetter(1) helps to sort with more than one column. Also, reverse=True refers to sorting in the descending order such that prediction[0][0] contains the string integer with the maximum probability.

We will then show the string using the putText() function on the frame, and then show the frame under the name Frame using the imshow() function.

Next, we initialize the interrupt variable, which returns 10 frames per second here. 0xFF collects and recognizes the keyboard response and matches it to the Escape Key. Then, after the detection is over, we release our webcam and deallocate all the memory rendered to the image array vectors. This has been previously explained in the past documentation.

Now your prediction.py should look like the following.


from keras.models import model_from_json

import operator

import cv2



json_file = open("model-bw.json", "r")

model_json = json_file.read()

json_file.close()

loaded_model = model_from_json(model_json)

loaded_model.load_weights("model-bw.h5")

print("Loaded model from disk")



cap = cv2.VideoCapture(0) 



categories = {0: 'ZERO', 1: 'ONE', 2: 'TWO', 3: 'THREE', 4: 'FOUR', 5: 'FIVE'}



while True:

  _, frame = cap.read()

  frame = cv2.flip(frame, 1)



  x1 = int(0.5*frame.shape[1])

  y1 = 10

  x2 = frame.shape[1]-10

  y2 = int(0.5*frame.shape[1])



  cv2.putText(frame, "Expressando - TDOC 2021", (175, 450), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,0), 3)

  cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,255,255) ,3)

  roi = frame[y1:y2, x1:x2]



  roi = cv2.resize(roi, (64, 64)) 

  roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

  cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)



  _, test_image = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

  cv2.imshow("ROI", test_image)



  result = loaded_model.predict(test_image.reshape(1, 64, 64, 1))

  prediction = {'ZERO': result[0][0], 

         'ONE': result[0][1], 

         'TWO': result[0][2],

         'THREE': result[0][3],

         'FOUR': result[0][4],

         'FIVE': result[0][5]} 

  prediction = sorted(prediction.items(), key=operator.itemgetter(1), reverse=True) 

  cv2.putText(frame, "PREDICTION:", (30, 90), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)

  cv2.putText(frame, prediction[0][0], (80, 130), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)   

  cv2.imshow("Frame", frame)



  interrupt = cv2.waitKey(10)

  if interrupt & 0xFF == 27:

    break





cap.release()

cv2.destroyAllWindows()

Run the code in your Powershell/terminal using:


python prediction.py

Now, see your real-time sign-language detection in action!!!

Thank you for reading this article. Hope you had a great time building this project and have learned a lot more about Machine learning.

May the source be with you! 🐧❤️

ETHEREAL : A Decentralized Virtual Banking system - Part II

Shashank Shekhar — Fri, 07 Jan 2022 12:19:16 +0000

This is a follow-up article from our previous post.

ETHEREAL : A Decentralized Virtual Banking system - Part I

Shashank Shekhar for GNU/Linux Users' Group, NIT Durgapur ・ Jan 7 '22

#blockchain #webdev #beginners #tutorial

In the previous post, we discussed the features like Creating a new account, Displaying Accounts, Account Details and Adding Balance. Today, let us learn how can we add features like : Withdrawing Balance, Transfer Amount to other virtual accounts on the Bank and Displaying Transaction history.

Part 3 : Withdrawing Balance and Transfer Amount to other virtual accounts on the Bank

Just navigate to SimpleStorage.sol file and paste this inside.


    //Function to withdraw balance from your existing accounts, the balance will be transferred to your blockchain account
    function withdrawBalance(
        uint256 _serial,
        uint256 _amount,
        address payable _creator
    ) 
    external returns (bool _success) {
        if (accounts[_serial].balance >= _amount / 1000000000000000000 + 1) {
            _creator.transfer(_amount);
            accounts[_serial].balance -= _amount / 1000000000000000000;
            bankBalance -= _amount / 1000000000000000000;
            transacNum++;
            transactions[transacNum] = Transaction(
                transacNum,
                accounts[_serial].balance,
                _amount / 1000000000000000000,
                block.timestamp,
                "Withdrawal",
                _serial
            );
            emit TransactionCompleted(
                _amount / 1000000000000000000,
                transacNum,
                serialNumber,
                "Withdrawal"
            );
            return true;
        } else {
            revert("Insufficient Funds");
            //   revert InsufficientFunds(_serial,_amount/1000000000000000000,accounts[_serial].balance-1,'Withdrawal');
        }
    }



    //Function to virtually send money from one bank account to other
    function transactAmount(
        uint256 _amount,
        uint256 _serial_2,
        uint256 _serial
    ) public {
        if (accounts[_serial].balance >= _amount / 1000000000000000000 + 1) {
            accounts[_serial].balance -= _amount / 1000000000000000000;
            accounts[_serial_2].balance += _amount / 1000000000000000000;
            transacNum++;
            emit TransactionCompleted(
                _amount / 1000000000000000000,
                transacNum,
                serialNumber,
                "TransferMoneySent"
            );
            transactions[transacNum] = Transaction(
                transacNum,
                accounts[_serial].balance,
                _amount / 1000000000000000000,
                block.timestamp,
                "TransferMoneySent",
                _serial
            );
            transacNum++;
            emit TransactionCompleted(
                _amount / 1000000000000000000,
                transacNum,
                serialNumber,
                "TransferMoneyReceived"
            );
            transactions[transacNum] = Transaction(
                transacNum,
                accounts[_serial_2].balance,
                _amount / 1000000000000000000,
                block.timestamp,
                "TransferMoneyReceived",
                _serial_2
            );
        } else {
            revert("Insufficient Funds");
            //   revert InsufficientFunds(_serial,_amount/1000000000000000000,accounts[_serial].balance-1,'TransferMoney');
        }
    }

Now, let us move on to the Front-end part.

Now, the logic for this feature is slightly counter-intuitive on the frontend but pretty straightforward in the backend, we create a withdrawBalance function similar to the addBalance function taking the same 3 inputs as in the addBalance function but this function will be an external function as it also needs to access the contract balance, and you cannot access the balance of a contract by being a part of the contract you need to access it from outside

Withdraw balance :

Ether from the Contract -> Ganache account

Virtual balance of both the Contract(Bank) and the account are decreased since actual ether is transported out of the Contract.

Now, how to send ether from the Contract to any other ethereum account, well this took a while to figure out, but every address has two methods to access, one is the balance method and the other is the transfer method, now we use the transfer method here to transfer actual ether from the Contract to any other Ganache account(here the account connected to the Metamask account).

Now, like the previous function, let us add a withdrawBalance function now according to intuition you need to subtract the amount that is passed from the frontend, and one more additional line must be added to transfer actual ether from the contract to the creator address which we pass in the withdrawBalance account.

Now, one check should be added though that the virtual account from which we are withdrawing balance should have an amount greater than the amount we want to withdraw.

withdrawBalance function code snippet

if(account[serialNumber].balance){

   account[serialNumber].balance-=amount;

   account[0].balance-=amount;

   _creator.transfer(amount);

}

This is the syntax to transfer amount from the contract to the Ganache account connected to Metamask which is the creator account

For the frontend, let us go to our AccountDetails page let us go to another grid box, whereas done in previous components we create a form input field and a submit button the value of the form input field being the withdrawBalance state variable which will be passed in the backend function.

On the click of the submit button, we pass the withdrawBalance function on the frontend which has a similar syntax to the previous addBalance function but there are some minor differences.

As you can notice that the value is not given, why?
Firstly, why do we need to use the send option when we are withdrawing balance, well that's the catch, to access any function from the backend solidity contract you need to use the send option and it will cost some amount of ether definitely and after you approve the Metamask transaction, in this case, the backend function is executed and actual ether is sent from your contract to the Ganache.

await contract.methods.withdrawBalance(id,withdrawBalance,account).send({from:account})

After this you need to resolve the promise using the .then and .catch methods, you can follow the example given in the above addBalance function where you need to run another async function to get the updated Balance and set the state of the balance variable to that updated balance. You can check with your Ganache client that the amount has been added to it, proving that your withdrawal was successful.

Now, let us implement the next functionality of this module, of transferring the virtual amount from one account to another, the logic for this is the simplest you can find among all the previous functions as there is no actual involvement of ether.

The bank balance won't be increased or decreased as both the accounts are present on the bank, we just need to increase the virtual balance of the account we are transferring to and decrease the virtual balance of the account from which we are sending the amount.

Let us quickly see the backend code for this functionality, we need to create a transferAmount function which is a public function in the backend which by intuition takes 3 inputs, any guesses ???

Well, it is the amount to be transferred, the serial number of the account from which the amount is being transferred, and the serial number of the account to which the amount is to be transferred.

We need to add one check before passing the function that is the amount that is passed should be less than the balance of the account from which it is being transferred.

The code snippet of the transferAmount functionality is given below:

transferAmount code snippet

if(accounts[serial_1].balance>amount){
   accounts[serial_1].balance-=amount;
   accounts[serial_2].balance+=amount;
}

Now let us complete the frontend code for this functionality, the same old process we need to create a form in a grid box, and in this form we need to take two inputs, one is the second account serial number to which you need to send the amount, and the other is the amount to transfer and on clicking of the submit button we need to pass a transferAmount function on the frontend.

This function takes the inputs from the form values and similar to the above functions we need to use the send option to run the function on the backend.

await contract.methods.transferAccount(amount,id,secondserialNumber).send({from:account})

and after this, we can use the similar .then and .catch methods to resolve this promise and run the function to again set the balance to updated balance, similar to the previous examples.

Part 4 : Displaying Transaction history

Navigate to SimpleStorage.sol file and paste this inside.

    //Defining a struct to store transaction record
    struct Transaction {
        uint256 transacNum;
        uint256 currentBalance;
        uint256 amountTransacted;
        uint256 createdAt;
        string transacType;
        uint256 accountSerialNumber;
    }

That concludes our project. You can further customize the styling of the project and also add any more new features like substituting the serial Number with the actual account and using the account for transactions.

Thank you for reading this article. Hope you had a great time building this project and have learned a lot more about Blockchain Technology.

You can find all the Resources required for this Project by clicking here.

Project Github repository :

deucaleon18 / ETHEREAL

A decentralized virtual banking system

About the project:

This project was built with my teammates as a part of our club ,GNU Linux Users' Group 's event Ten Days of Code where we mentored students from all over India to build projects from scratch over a period of ten days.

This DApp is supported using Solidity Smart Contracts and the frontend is built using the framework React JS, the application is deployed to the local blockchain network,Ganache. The whole project runs on an Ethereum supported browser, with the help of the Metamask extension.For, building compiling and testing the written contracts, Truffle was used.Lastly,the frontend was connected to the solidity contract using a wonderful library,web3.

The detailed guide to build this project is present in the resources folder

Display:

Here are few glimpses of the project we built.

HOME PAGE

REGISTER HERE

ACCOUNTS PAGE

ACCOUNT DETAILS

TRANSFER

…

View on GitHub

Project collaborators :

Shubham Shantam RajuFollow

Open Source Enthusiast | Sophomore of NIT Durgapur | Member of GNU Linux Users' Group | Full Stack Devleoper (MERN) | Blockchain Enthusiast

Shashank ShekharFollow

Open Source Enthusiast | Junior Developer at GNU/Linux User's Group NIT Dgp | Full Stack developer | Blockchain Fanatic

Apurva Jyoti PaulFollow

Connect with us :

ETHEREAL : A Decentralized Virtual Banking system - Part I

Shashank Shekhar — Fri, 07 Jan 2022 12:07:08 +0000

In recent times, we have all heard about Blockchain once or twice and it has been a hot and controversial topic ,almost in the news every other day. In the 2 parts of this article-series we have explored the various terms related to Blockchain like Bitcoin, Ethereum, Solidity and we have built an actual working DApp(a decentralized application), where we have guided you in building the project from ground zero. This project was built as part of our event Ten Days of Code held from 17th October 2021 to 26th October 2021, where we mentored students all over India in building projects from scratch across various domains.

In this project, we have built a decentralized banking application, now decentralized here means that the central power doesn't lie in the hands of a single user but is instead divided equally among all the users and hence no one supervises all the account in our bank and everyone is responsible for their own account. We have implemented all the basic features that a bank can perform like deposition, withdrawal, and transfer of money.

Now, you might be wondering why decentralized, well great question indeed for this a simple analogy would suffice, we all hate dictatorships where the whole power lies with a single person and hence prefer democracies, well a debatable point, but let's not get there, now applying the same analogy wouldn't it be great if you and only you had the access to your account opposing the view of a single person having access to all the accounts, being the whole purpose of this project.

This DApp is supported using Solidity Smart Contracts and the frontend is built using the framework React JS, the application is deployed to the local blockchain network, Ganache. The whole project runs on an Ethereum supported browser, with the help of the Metamask extension. For building and compiling Truffle was used. Lastly, the frontend was connected to the solidity contract using a web3.

To understand all these terms and how Blockchain works do watch the video below which was a live session from our event Ten Days of Code.

What is Blockchain?
Blockchain technology is a type of distributed ledger technology (DLT) — It is an accounting system where the ledger (record of transactions) is distributed among a network of computers.
So at its core, blockchain technology is a record-keeping tool.
Beginners guide for Blockchain

What is a Smart Contract ?
All of the code on the blockchain is contained in smart contracts, which are programs that run on the blockchain. All of the code on the blockchain is contained in smart contracts, which are programs that run on the blockchain. They are the building blocks of blockchain applications. We'll write a smart contract in this project to power our Banking system.
For in-depth detail click here

Intro To Solidity :
Solidity is the main programming language for writing smart contracts for the Ethereum blockchain. It is a contract-oriented language, which means that smart contracts are responsible for storing all of the programming logic that transacts with the blockchain. It's a high-level programming language that looks a lot like JavaScript, Python, and C++.
For in-depth detail click here

Prerequisites :

You can watch this video for setting up this project:

Ganache (Local Development Blockchain)

The dependency is a personal blockchain, which is a local development blockchain that can be used to mimic the behavior of a public blockchain. We recommend using Ganache as your personal blockchain for Ethereum development. It will allow you to deploy smart contracts, develop applications, and run tests. It is available on Windows, Mac, and Linux as a desktop application and a command-line tool !

Download it by clicking here (For both Windows and Linux)

For Linux(Ubuntu) Users :

Once the download is complete, go to the folder where you have downloaded the file and open the terminal and type the following to make the file executable:

chmod a+x <filename.AppImage>

Node.JS
Now that you have a private blockchain running, you need to configure your environment for developing smart contracts. The first dependency you'll need is Node Package Manager , or NPM, which comes with Node.js.

You can see if you have node already installed by going to your terminal/cmd and typing:

node -v

If you don't have a node already installed then :

For Windows Users :

Download it by clicking here

It is recommended that you download LTS version.

For Linux(Ubuntu) Users :

Go to your terminal and type :

$ sudo apt install nodejs
$ sudo apt install npm

Truffle Framework
Now let's install the Truffle Framework,
which provides a suite of tools for developing Ethereum smart contracts with the Solidity programming language.

For Windows Users :

Go to your cmd and type :

npm install -g truffle

For Linux(Ubuntu) Users :

Go to your terminal and type :

$ sudo npm install -g truffle

For this project, we have used React JS framework for the frontend and to get the required folder structure for our project we run the following command:

For Windows Users :

Go to your cmd and type :

truffle unbox react

For Linux(Ubuntu) Users :

Go to your terminal and type :

$ sudo npx truffle unbox react

Metamask Wallet

For this project, we would require an Ethereum wallet, and here we will use the popular open-source wallet Metamask.

On Chrome/Firefox/Brave(It is recommended that you use Brave browser) :

Go to the extensions page of your browser and install the Metamask plugin, click here
On installation the user will be redirected to a setup page and the user has to choose a password and a recovery phrase.

For in-depth detail click here

Getting Started with the Project :

After running the command npx truffle unbox react you would have a folder structure of something similar to this in front of you :

Let us understand what each of these means :

node_modules : This folder contains all the required npm packages for your project which are installed when you run npm install after cloning our repository.

src : This folder contains all the components and pages for our React project

public : This is the directory where we save our images and which also contains the main HTML file index.html where all the components from src are finally rendered to give us a single page application.

The .gitignore file contains information-related code which you don't want to commit to Github .

The package.json and package-lock.json are initialized when you run npm init and install npm packages in your project.

For in-depth detail about React, click here

For React tutorial, click here

Must watch useState (3:39:00 to 4:23:00), useEffect (4:47:27 to 5:19:24) and React Router (9:00:41 to 9:23:22)

Part 0 : Creating a simple blockchain project: Ethereum Todo List

Referance to build this simple project:

Part 1 : First Functionality - Creating a new account

Let us begin with the Front-end.

Navigate to your client folder and then to src.

We will be writing all of our frontend code in the form of various components and import them all to the final index.js file which will be rendered onto the HTML file present in the public folder.

Now, that you have explored all the hooks used in the project , we add some basic lines of code to every component to fetch all the details like web3, Metamask connected account and the solidity contract.

This code is common for all the components as we need to fetch data in every component.

The only component which won't contain this snippet is the Navbar because this component is only for Navigation purposes and does not require all these details , for this project purposes we create a Navbar with 2 tabs for now : REGISTER and ACCOUNTS , on clicking the REGISTER tab it should redirect you to the URL "/create" , you can change the name for this route if you want, we have named it "/create" and the ACCOUNTS tab should redirect you to the URL "/accounts". You can import this Navbar in all the components you are creating to navigate to the different tabs of our application which are only 2 currently, but you will soon be creating more tabs.

The only modification is when you fetch more details or the particular function for that component from the contract you need to add this particular code snippet.

 useEffect(() => {
    
    const getBasicDetails = async () => {
        
        try {

             // Get network provider and web3 instance.
            
            const web3 = await getWeb3();

             // Use web3 to get the user's accounts.
            
            const accounts = await web3.eth.getAccounts();

             // Get the contract instance.
            
            const networkId = await web3.eth.net.getId();
            
            const deployedNetwork =
                BankingContract.networks[networkId];
            
            const instance = new web3.eth.Contract(

                BankingContract.abi, deployedNetwork && deployedNetwork.address
                );
            
            setWeb3(web3);
            setAccount(accounts[0]);
            setContract(instance);

            
        } catch (error) {

             // Catch any errors for any of the above operations.
     
            alert(
                `Failed to load web3, accounts, or contract. Check console for details.`

                );
            
            console.error(error);
        }
    }
    getBasicDetails();

}, []);

This is how your first useEffect hook looks. To go through this useEffect hook once would be enough as you will be using this same useEffect hook in every component, even if you create new components.

In this hook we have defined a function getBasicDetails, where we load web3 from the getWeb3.js, and we get the Metamask connected account by using web3, as we have already mentioned in our intro section that web3 acts as a connector between our backend and frontend.

Here we will be using web3 for everything. For creating a new contract, for using the various functions from the backend, for loading accounts. You will soon understand that it makes everything much easier.

Firstly, throughout the application we use web3 and whenever we use web3, it is through asynchronous calls and we use the async await syntax and basic ES7 javascript .then, .catch syntax to resolve promises.

web3.eth ( eth is short for ethereum ) is the object which contains various functions for getting your accounts, getting your network id , creating a new contract from your JSON file .

Read more about the various methods of web3 here

In the getBasicDetails function , we use the getAccounts() , net.getId() , Contract() methods of web3.eth

getAccounts() gets all the accounts connected to the Ethereum browser (here Metamask), to access the present connected account we want the very first element of this accounts array which is at the 0th index, and we change the state of account variable to this

net.getId() gets the network Id of the Solidity contract which is one of the two things to initiate a new contract using web3, the other one is the abi object present in the Banking.json file.

new web3.eth.Contract(BankingContract.abi,deployedNetwork && deployedNetwork.address)

This initiates a contract from our existing JSON file.
We set the empty web3 variable to the loaded web3 object and the empty contract variable to the newly created contract using useState hook of React.

To give an example of the usage of the useParams hook, we have used it in the Transactions component.

In the Transactions component, we have accessed the id parameter from the URL and assigned it to a variable in the Transactions component.

const{id} =useParams(id);

In the input of the useParams hook, we refer to the exact parameter variable we used in the URL, in this case id .

I hope that helps you understand the useParams hook, for more reference visit:

Let us briefly explore the Routing.js file :

In the Routing.js file we import Router and Route from 'react-router-dom' and the basic syntax of the whole Routing component is similar for every Route.

import { BrowserRouter as Router, Route } from "react-router-dom";

import ComponentName1 from "<path-to-the-component>";

import ComponentName2 from "<path-to-the-component>";

<Router>
  <Route exact path="<specify-path1-here>">
    <ComponentName1 />
  </Route>

  <Route exact path="<specify-path2-here>">
    <ComponentName2 />
  </Route>
</Router>;

The first useEffect hook will be present in all functional components like AccountDetails, DisplayAccounts , LoanTransaction, TransferMoney and all the other components except the Navbar .

Let's talk about the second useEffect hook which you will be used in some or almost all of the components to extract the required functionality from the contract because only the creation of the contract is not enough.

We need to access the particular functionality from our contract to perform various operations for us, for example, when we are creating the CreateAccount.js component we actually don't require any additional data, but we just use the second hook to log the contract details in our browser console .

In this hook we run the function only when web3, contract and account are not undefined as we had initialized them as undefined, the dependency array also has web3, account, contract in it so that useEffect runs until all of these variables change from undefined to a definitive value, this is done for easy coding purposes to view the various methods present in the contract (We'll explain what are these various methods).

So, let's continue building the createAccount.js component, for this, we require a basic HTML form

PRO TIP : To build forms in which the data field values are stored in some variable which can be further used at different places we use the useState hook, this has been done in all the components so I am specifying this in the beginning for your ease.

EXAMPLE :

const[name,setName]=useState("")

<input
  value={name}
  onChange={(e) => setName(e.target.value)}
  placeholder="Name"
  type="text"
/>;

In the above example, we initialized the name variable as a blank variable and then when the form input value is changed we pass the onChange function where the state of the name variable is updated to the current target value, this name can be further used any place we want, overall throughout the application we use this same syntax, so when I refer to making a form input with a value equal to the value of the variable "x" what I actually will mean is the above syntax.

So, now if you recall from above in the Solidity contract the function to create a new account takes 3 inputs, and among those 3 - one is the creator address which is the account connected to Metamask which we already stored in the account state variable, the other 2 variables are the name and location inputs which we will get from the form that we are creating.

After creating the form with 2 input fields we need to create a SUBMIT button and we need to pass a function on the click of this button, now the syntax of this function will also be similar in every component and since this is for the first component let me give you the syntax.

EXAMPLE:

We defined the handleSubmit function inside our CreateAccount component but this should be above the rendering of the JSX.

ES7 arrow function syntax is being used here throughout the project, you may use it or may proceed with the normal function syntax.


//Function to handle the submit operation of the form

const handleSubmit = async (e) => {

        //prevents the default action that occurs when someone submits a form 

        e.preventDefault()

        //...rest of the function code ...//

    }

    <
    button type = "submit"
onClick = {
    handleSubmit
} > SUBMIT < button / >

This function is asynchronous because we need to communicate with the contract, send a request to it and wait for its response. Hence we use a similar syntax everywhere using .then and .catch to resolve the promises, more than one .then can also be changed as per the user requirement.

Now in our createAccount.js file, the handleSubmit function should contain the code to access the function to create new accounts from the backend, now how this is done?

Well, firstly we have already created a new contract, every contract is an object which further has an item called methods, now what does this methods item contain?

It contains every public variable and function defined in the solidity contract. That's it, you can refer to the contract object in the browser console, you will find the methods item in it.

For this component, we need to access the createAccount method, now there are 2 ways to access a function using web3, they are call and send, the major differences in these two are you use call only when you want to view data and not modify anything and it does not cost any ether to perform this transaction but if you use the send method it costs some amount of gas fees and it modifies data, why does it cost ether though?

Well, that is an interesting question though but as you already know that you need to send ether for any function to be executed or written on a new block and the miners of this block need to be rewarded somehow since they are securing the whole system by solving complex algorithms which creates new blocks for storing our executed function data, hence this is the a*mount deducted* from our account when we use the send method, which is rewarded to the miners.

In createAccount.js we are creating a new account, that is modifying data so we use the send method of web3 rather than the call method (which is used to just view the data), and the syntax of a send option is such that it also accepts another object which contains details like the from address, to address, and the value being sent.

In, our banking application we deduct 2 ETH from the Metamask account on the creation of every new account as a starting balance to the virtual account and increment the balance of the virtual account from 0 to 2 (the increment of balance is handled on the backend).

await contract.methods
  .createAccount(account, name, location)
  .send({ from: account, value: web3.utils.toWei(2, "ether") })

  .then((res) => {
    console.log(res);

    window.location.href = "/accounts";
  })

  .catch((err) => {
    console.log(err);
  });

When you click on submit the Metamask will prompt you to approve the transaction with the cost of 2 ETH, on approval your new account will be created. On, successful approval according to the logic of the function the page will redirect to "/accounts".

But, always remember if you want to perform a function that updates the existing contract you need to spend some amount of ether and send it from your Metamask connected account, the ether is sent to the contract which will, in turn, be sent to the miner accounts.

Navigate to your contracts folder and open SimpleStorage.sol file.


// SPDX-License-Identifier: MIT
pragma solidity >=0.4.21 <0.7.0;

// import "github.com/Arachnid/solidity-stringutils/strings.sol";
contract Banking {
    //  using strings for *;

    uint256 public serialNumber = 0;
    uint256 public transacNum = 0;
    uint256 public bankBalance = 0;


    //Function to create account , the 2 ETH balance will be taken from your ethereum account and deposited to the banking contract
    function createAccount(
        address payable _creator,
        string memory _name,
        string memory _location
    ) public payable {
        if (_creator.balance >= 3) {
            serialNumber++;
            accounts[serialNumber] = Account(
                serialNumber,
                block.timestamp,
                _name,
                _location,
                _creator,
                2,
                true
            );
            bankBalance += 2;
            transacNum++;
            transactions[transacNum] = Transaction(
                transacNum,
                2,
                2,
                block.timestamp,
                "NewAccount",
                serialNumber
            );
            emit TransactionCompleted(
                2,
                transacNum,
                serialNumber,
                "NewAccount"
            );
        } else {
            revert("Insufficient Funds");
            // revert InsufficientFunds(serialNumber,2,_creator.balance-1,'NewAccount');
        }
    }
}

Now, in the src folder, we further have the folder structure as:

  ->components

  ->contracts

  ->App.css

  ->App.js

  ->getWeb3.js

  ->index.js

  ->Routing.js

  ->serviceWorker.js

Once you understand the purpose of each component you will be able to navigate easily through the code, and trust me when I say organizing your code into folders is much cleaner and accessible in the future.

The components folder contains various sub-folders for each of our components in our application, here in this application we used the components.

->AccountDetials (the component that gets all the account details of the current account serial number)

->DisplayAccounts (displays all the registered accounts connected to the particular Metamask account)

->DisplayTransactions (displays all the transactions that occurred in the particular account)

->LoanTransaction (a temporarily working function that resembles loan transactions)

->MainSection (the main landing page when one visits the application)

->Navbar (used to navigate to various components)

->TransferMoney (the component to transfer virtual amount)

By looking at the names of the components you might get a feel of the job they perform, you can edit the names of these components as you wish and once you edit them they will get edited in the files they were imported into.

The contracts folder contains the JSON file for the Banking contract we had deployed and we will be using and accessing this very JSON file to access our backend, for you full-stack developers out there you can imagine this as an API that can be used in your frontend application using asynchronous fetching.

The App.js contains various imports from all the components of the application and App.css defines the global styles for every component present in the application since all the components will be imported in App.js

The getWeb3.js is a boilerplate template generated for us by Truffle when we run the command truffle unbox react which contains a promise-based function to load Web3.

The index.js is the file where the App.js is finally imported and rendered onto the frontend HTML file.

The Routing.js is an important file that contains the various routes present in the whole application it is imported from react-router-dom which can be installed by running the command:

npm install react-router-dom

Lastly, serviceWorker.js is not of much use for us as it is just a part of the boilerplate code that appears with React.

That was the folder structure! Now let us discuss some of the basic React hooks syntax that we have used in our application for React beginners:

Throughout this application, we have used only three React hooks and by three, I mean exactly three not more, not less all the work can be done using these three hooks. These are useState, useEffect, and useParams.

So, let us just get to know briefly about each of these and a little bit about the project syntax

An example for useParams is where we have imported the Transactions component to our Routing.js file and in the path URL, we have specified the parameter id.

In the Transactions component, we can use this id parameter to perform actions that are specific to the account with that id, we can access this id parameter in another component from the URL using the useParams hook

In the Transactions component, we have accessed the id parameter and assigned it to a variable in the Transactions component.

const{id} =useParams(id);

In the bracket of the useParams hook, we refer to the exact parameter variable we used in the URL.

I hope that helps you understand the useParams hook, for more reference visit :

Let us briefly explore the Routing.js file :

In the Routing.js file we import Router and Route from 'react-router-dom' and the basic syntax of the whole Routing component is similar for every Route.

import { BrowserRouter as Router, Route } from "react-router-dom";

import ComponentName1 from "<path-to-the-component>";

import ComponentName2 from "<path-to-the-component>";

<Router>
  <Route exact path="<specify-path1-here>">
    <ComponentName1 />
  </Route>

  <Route exact path="<specify-path2-here>">
    <ComponentName2 />
  </Route>
</Router>;

Now, that was all about the first useEffect hook which will be present in all functional components like AccountDetails, DisplayAccounts , LoanTransaction, TransferMoney and every other component other than Navbar .

Let's talk about the second useEffect hook which you will be using in some or almost all of the components to extract the functionality from the contract because only the creation of the contract is not enough we need to access the functionality of the contract to perform various operations for us, in the first component we are creating the CreateAccount.js component we actually don't require any additional data, but we just use the second hook to console.log the contract details, and in this hook, we only run the function when web3, contract and account are not undefined as we had initialized them as undefined, the dependency array also has web3, account, contract in it so that useEffect runs until all of these variables change from undefined to a definitive value, this is done for easy coding purposes to view the various methods present in the contract (We'll explain what are these various methods).

So, let's continue building the createAccount.js component, for this, we require a basic HTML form

EXAMPLE :

 const [name, setName] = useState("")

     < input value = {
        name
    }
onChange = {
    (e) => setName(e.target.value)
}
placeholder = "Name"
type = "text" / >

In the above example we initialized name as a blank state variable and then when the form input value is changed we pass the onChange function, where the state of the name variable is updated to the current target value, this name can be further used any place we want, overall throughout the application we use this, so when I refer to making a form input with a value equal to the value of the variable "x" what I actually will mean is the above syntax.

So, now if you recall in the Solidity contract the function to create a new account takes 3 inputs, and among those 3 -one is the creator address which is the account connected to Metamask which we already stored in the account state variable, the other 2 variables are the name and location inputs which we will get from the form which we are creating.

After creating the form with 2 input fields we need to create a Submit button and we need to pass a function on click of this button, now the syntax of this function will also be similar in every component since it is the first component let us give you the format.

EXAMPLE :

We defined the handleSubmit function inside our CreateAccount.js component but this should be above the rendering of the JSX

ES7 arrow function syntax is being used here throughout the project, you may use it or may proceed with the normal function syntax.


//Function to handle the submit operation of the form

const handleSubmit = async (e) => {

        //prevents the default action that occurs when someone submits a form 

        e.preventDefault()

        //...rest of the function code ...//

    }

    <
    button type = "submit"
onClick = {
    handleSubmit
} > SUBMIT < button / >

This function is asynchronous because we need to communicate with the contract and send a request to it and wait for its response, and hence the syntax is similar everywhere using .then and .catch to resolve the promises, more than one .then can also be changed as per the user requirement.

Now in our createAccount.js file, the handleSubmit function should contain the code to access the function to create new accounts from the backend, now how this is done?

Well, firstly we have already created a new contract, every contract is an object which further has an item called methods, now what does this methods item contain?

It contains every public variable and function defined in the solidity contract. That's it, you can refer to the contract object in the console.log, you will find the methods item in it.

For this component we need to access the createAccount method, now there are 2 ways to access a function using web3, they are call and send, the major differences in these two are you use call only when you want to view data and not modify anything and it does not cost any ether to perform this transaction but if you use the send method it costs some amount of gas fees and it modifies data, why does it cost ether though?

Well, that is an interesting question though but as you already know from the Basics module that you need to send ether for the function to be executed or written on a new block and the miners of this block need to be rewarded somehow since they are securing the whole system by solving complex algorithms which creates new blocks for storing our executed function data, hence this is the amount deducted from our account when we use the send method, which is rewarded to the miners.

For, the createAccount.js we are creating a new account, that is modifying data so we use the send option, and the syntax of a send option is such that it also accepts another object which contains details like the from address, to address, and the value being sent.

In, our banking application we deduct 2 ETH from the Metamask connected account on the creation of every new account as a starting balance to the virtual account and increment the balance of the virtual account from 0 to 2 (the increment of balance is handled on the backend).


await contract.methods
  .createAccount(account, name, location)
  .send({ from: account, value: web3.utils.toWei(2, "ether") })

  .then((res) => {
    console.log(res);

    window.location.href = "/accounts";
  })

  .catch((err) => {
    console.log(err);
  });

When you click on SUBMIT, Metamask will prompt you to approve the transaction with the cost of 2 ETH, on approval your new account will be created. On, successful approval according to the logic of the function the page will redirect to "/accounts".

This is an example of how to access any function from the backend passing the values and using the send method, if you just want to display values which we will see in the DisplayAccounts.js in the next module you just need to use the call option and spend no ether. But, always remember if you want to perform a function you need to spend some amount of ether and send it from your Metamask connected account, the ether is sent to the contract which will, in turn, be sent to the miner accounts.

Part 2 : Displaying Accounts, Account Details and Adding Balance

Now let us start coding the solidity component (Just navigate to SimpleStorage.sol file and paste this inside contract Banking {} ).

    //Defining a struct to store the account details
    struct Account {
        uint256 serial;
        uint256 createdAt;
        string name;
        string location;
        address creator;
        //   bytes32 accountName;
        uint256 balance;
        bool doesExist;
    }

    constructor() public {
        accounts[0] = Account(
            0,
            block.timestamp,
            "Ethereal",
            "NIT Durgapur",
            address(this),
            0,
            true
        );



    //Function to get the msg.sender's ETH balance
    function getSenderBalance(address payable _account)
        external
        view
        returns (uint256)
    {
        return _account.balance;
    }

    //Function to get the contract's existing balance
    function getContractBalance() external view returns (uint256) { 
        return address(this).balance;
    }

    //Function to get the owner
    function getOwner() public view returns (address) {
        return msg.sender;
    }


    //Function to add balance to your existing acccount ,the balance will be taken from your ethereum account
    function addBalance(
        uint256 _serial,
        uint256 _amount,
        address payable _creator
    ) public payable {
        if (_creator.balance >= _amount / 1000000000000000000 + 1) {
            accounts[_serial].balance += _amount / 1000000000000000000;
            bankBalance += _amount / 1000000000000000000;
            transacNum++;

            transactions[transacNum] = Transaction(
                transacNum,
                accounts[_serial].balance,
                _amount / 1000000000000000000,
                block.timestamp,
                "AddingBalance",
                _serial
            );

            emit TransactionCompleted(
                _amount / 1000000000000000000,
                transacNum,
                serialNumber,
                "AddingBalance"
            );
        } else {
            revert("Insufficient Funds");
            // revert InsufficientFunds(_serial,_amount/1000000000000000000,_creator.balance-1,'AddingBalance');
        }
    }

Now, let us move on to the Front-end part.

The logic for this is simple, as you might have remembered we have an accounts mapping in the backend and we can access them using the serialNumber key which is the normal index of the account. Here we need to use the call option and call the accounts method (even mappings are part of the methods object) one by one and push it into a state variable of bankingAccounts if and only if the creator of the account is the same as that of the currently connected Metamask account(important check -verification condition to know that the person who created the account is the only person who can access it and perform operations in it)

The basic layout of this component is no different from the other components, but the second useEffect hook here has actual use since you need to call the accounts mapping (from the contract methods), and you need to call this method serialNumber the number of times(serialNumber is also a method from the contract methods) and check for only those accounts whose _creator address is same as that of the Metamask connected account.

In the second useEffect hook we need to do two things, we need to get the value of serialNumber, and then we need to get the various accounts connected to the Metamask address and push it to the bankingAccounts state array.

var serial =await contract.methods.serialNumber().call()       [even public variables from the backend need to be called ]

Now we need to run a for loop from 1 to less than or equal to serialNumber and call the accounts methods with the key as the value of the iterator, the account object is pushed to the bankingAccounts array after it passes the check that the creator address is equal to the Metamask connected address.

for (int i = 1; i <= serial; i++) {
    await contract.methods.accounts(i).call()



        //ES7 promise resolving syntax using .then and .catch 

        .then(() => {

            if (res.creator == account) {

                setBankingAccounts(bankingAccounts.push({
                    name: res.name, ...so on...
                }))

            }

        })

        .catch((err) => {

            console.log(err)
        })

}

The above snippet is an example where you update the state of
bankingAccounts as you are pushing each bankingAccount connected the that particular Metamask address.

Finally, setLoading(false) after the whole for loop is run.

Now you need to use one of the famous array methods mappings in the frontend to print the contents of the bankingAccounts array, to give you an idea of the CSS layout of this page it will only contain various account bars with the name and serial number, which after clicking on them will redirect you to the account details page of that particular account where you can find the complete account details and various operations to perform on the account.

Before printing all the accounts check for if loading is false using the ternary operator and then map the bankingAccounts, if the loading is still true print nothing, now you need to make these bars where you are printing clickable and they should redirect you to the specific page for that account with the parameter as serialNumber ( to "/accounts/:id") and here the usage of useParams begin as we have mentioned in the third module. Also, react throws you an error in the console that every child element must contain a key which is a unique identifier value so in this case, also you can set the key={account. serial}

The location is changed using the window. location method.

!loading ? bankingAccounts.map((account) => {

            <div key = {
                account.serial
            }
            onClick = {
                () => {
                    window.location.href = accounts / $ {
                        serialNumber
                    }
                } >

                {account.name}

                {account.serialNumber}

                </ div >

            }): null

Now that was displaying all the accounts, we need to also keep updating the router and adding all the new routes we are creating to the router for example until now "/" route has been occupied by createAccounts component, and the "/accounts" route is occupied by DisplayAccounts.js, now we add the third route "/accounts/:id" for the AccountDetails.js

Let us begin coding the AccountDetails component!

As usual, the two useEffect hooks are common like in every component.

The second useEffect hook, like the previous component, DisplayAccounts is not empty, in it the specific account pertaining to that page is called, using the id parameter from the URL, and the details are stored in a state variable, specifically, balance item of the bankAccount is stored in another state variable so that it can easily be updated every time the balance changes, for example when we add more balance or withdraw the balance.

  const[bankAccount,setBankAccount]=useState("")

  const[balance,setBalance]=useState("")

  const{id} =useParams(id)

  const [loading,setLoading]=useState(true)

The function in the second useEffect hook :

 await contract.methods
  .accounts(id)
  .call()

  .then((res) => {
    console.log(res);

    setBankingAccount(res);

    setBalance(res.balance);
  })

  .then(() => {
    setLoading(false);
  })

  .catch((err) => {
    console.log(err);
  });

Now we have the data stored in our state variables we just display it somewhere on our application.

To, give a brief idea of the layout of the page let us divide the page into various columns, one in the extreme left to print the account details, there are various other grid boxes for each functionality that contain a simple form, for example, there is an additional balance grid box, a withdraw balance grid box, a transfer amount grid box and a box for transferring amount from one account to other, later we can add more grid boxes which contains buttons that redirect to pages for displaying transaction(last extras section) and the other one for performing loan transactions.

Now, let us move on to the second part of this article, and create a function in the backend to add virtual balance to our bank account.

We create a public payable addBalance function which takes inputs as the serial , amount , and the creator address .

Why these 3 inputs?

Serial number is required to increase the virtual balance of that particular virtual account.
The amount is the amount to be passed that is to be added.
You need the creator address that is connected to Metamask to transfer actual ether from your Ganache account.

We need to increase the virtual balance by accessing the balance item of the particular account from the serialNumber and we also need to increase the bankBalance of the whole application present at the 0th index of the accounts mapping.

We need to also add a check to only update the virtual balance when the Metamask connected Ganache account balance is greater than the amount entered.

Add balance code snippet:

if(creator.balance>amount)

{

account[serialNumber].balance+=amount;

account[0].balance+=amount;

}

Now after displaying the account details, we start coding the add balance functionality grid box on the frontend, it should contain a form input with the state variable balanceAdded as we have specified in previous examples, and also a submit button which onClick should fire an addBalance function which is similar to the syntax of our previous functions and should take the 3 input values.

We use the send method here and the from address is from the Metamask account and the value is the balance to be added which should also be passed in the object. We need to pass this value in the send method since that amount of ether should be transferred from our Ganache account to the contract, if we don't specify the amount only a minimal amount of ether will be costed that will be the gas amount to perform the function from the backend.

Adding Balance: Ether from Ganache -> Contract Address

Virtual Balance of the account and Contract (Bank) are both increased since actual Ether flows from the Contract.

We have to also again call the accounts method with the key serial number, and then update the balance variable to instantly print the new balance in the account details grid section.


const addBalance = async (e) => {
  e.preventDefault();

  `The addBalance is the state variable which we get from the addBalance form and the account is the Metamask connected account`;

  await contract.methods
    .addBalance(id, addBalance, account)
    .send({ from: account, value: addBalance })

    .then((res) => {
      console.log(res);
    })

    .then(async () => {
      await contract.methods
        .accounts(id)
        .call()

        .then((res) => {
          setBalance(res.balance);
        })

        .catch((err) => {
          console.log(err);
        });
    })

    .catch((err) => {
      console.log(err);
    });
};

We have completed the first 2 functionalities of our virtual bank, namely Creating a new account and Updating the Balance for the account in this article, in the 2nd part of this article-series we will be adding features like: Withdrawing Balance, Transfering Amount to other virtual accounts on the Bank and Displaying Transaction history of an account, to our virtual bank.

This is a 2nd Part of this article in which we will be adding features like : Withdrawing Balance, Transfer Amount to other virtual accounts on the Bank and Displaying Transaction history to our Project.

ETHEREAL : A Decentralized Virtual Banking system - Part II

Shashank Shekhar for GNU/Linux Users' Group, NIT Durgapur ・ Jan 7 '22

#blockchain #webdev #beginners #tutorial

You can find all the Resources required for this Project by clicking here.

Project Github repository :

deucaleon18 / ETHEREAL

A decentralized virtual banking system

About the project:

The detailed guide to build this project is present in the resources folder

Display:

Here are few glimpses of the project we built.

HOME PAGE

REGISTER HERE

ACCOUNTS PAGE

ACCOUNT DETAILS

TRANSFER

…

View on GitHub

Project collaborators :

Shubham Shantam RajuFollow

Open Source Enthusiast | Sophomore of NIT Durgapur | Member of GNU Linux Users' Group | Full Stack Devleoper (MERN) | Blockchain Enthusiast

Shashank ShekharFollow

Open Source Enthusiast | Junior Developer at GNU/Linux User's Group NIT Dgp | Full Stack developer | Blockchain Fanatic

Apurva Jyoti PaulFollow

Connect with us :

Control Google Meet With Expressions/Gestures

Ritabrata Das — Thu, 26 Aug 2021 09:41:15 +0000

With the advancements made in the field of modern day machine-learning technologies, it has become quite easy and flexible to develop models and applications based on this domain. So, you might have a perception that the realm of Machine Learning requires complex algorithms and huge expert knowledge for it's projects.

However, in the recent times, it has been quite easy to develop them as most of the standard algorithms and models are made available on the Internet in the form of web-applications. In this project, we will be using Teachable Machine, along with Tensorflow.js to control the basic actions in Google Meet such as muting, switching off your video and a special action, which will be revealed in the later course of the article.

Teachable-Machine

Teachable Machine is an online web-application, created by Google Creative Labs that facilitates easy and fast creation of machine learning models that is accessible to everyone on the net. It is flexible as it makes use of pre-existing files or even capture live examples, which will be recorded as instances, on the basis of which references will be made. You can even make a choice to use your models entirely on-device, without any webcam or microphone data being conveyed outside your system.
The advantage is that you train your computer to recognize your images, sounds, and poses of your personal choice, without writing any complicated and long machine learning code. Then, you can use your model in your own projects, applications, websites and even more. It is powered by Tensorflow.js, which is an open-source library, created by Google for the purpose of Machine-Learning.

Prerequisites

Well, at the most, what you require is the basic knowledge of Vanilla JavaScript, a webcam, an Internet connection and the passion to learn something new. Then, follow the below steps and get your own functional model developed in front of your eyes.

The Project

The project is in the form of a chrome browser-extension, which basically controls all the actions on Google Meet. The article has been divided into the following subtopics to facilitate your understanding:

Step 1: Training your Model

The first step involves creating the basic models on Teachable Machine. The model can be either any image, body pose or sound of your choice. In this project, we have decided to utilize the basic image model for training our computer. However, you can also use body poses and speech to train your model otherwise. Make sure the images, poses or sounds that you wish to implement are quite unnatural and can be performed accurately.

Now, to start with, head over to the Teachable Machine site and click on Get Started. You can further access it from the hamburger menu at the top-left corner of the homepage. You will get the following page:

You will now get the options for creating your model online to train your machine. There are three different projects namely Image Project, Audio Project and Pose Project. For our model, we have used the Pose Project for it's implementation. You can also import your previously saved models from your Google Drive or local file system if you want to use a model which was created beforehand and you wish to reuse and modify it. If you want a new model, just click on Pose Project. Then, the site will redirect you to your project destination:

First, you have to record your images in the form of "classes". The images can be captured live via webcam, or can be uploaded from the local file system as well. The image samples of a particular class should be same and repetitive in nature, while the image samples used in various classes should be different so that the class can be defined more accurately and can be recognized with ease.

It is advisable to record at least 50 to 60 images for increasing the accuracy threshold of the recorded class. In order to record live instances, grant permission to your webcam and press the Hold to Record button to capture your live images.

If you want a free tutorial on how to record your model in the form of images, you can head over to the Teachable Machine Tutorial and click on "Let's Go" for a live demo practice.

You can change the labels of the classes according to your wish. In this project, we have made use of three classes namely: Audio,Video and Escape, which are recorded live via webcam. The images in these classes can be specific hand gestures ,facial expressions or particular objects, which can be identified and recognized by the machine easily.
In order to add more than two classes, click on Add Class. After recording the image samples for the respective classes, it is time to train and export your model.

Step 2: Exporting your Model

After, you have created your respective classes, click on Train Model, which will train your machine to recognize the content of the images, which will be used in our upcoming projects. Teachable Machine mainly uses pattern recognition algorithms and involves the usage of:

Statistical Techniques
Structural Techniques
Template Matching
Neural Network Approach
Fuzzy Model
Hybrid Models

After your model has been trained, you can preview your model before exporting it. This helps you to test the model before implementing it in your projects. Just perform the poses according to the images set by you and check whether the machine can identify them or not.

When your model has been prepared, click on Export Model and export the model in the form of Tensorflow.js. Click on Upload my model, which will publish your model online and it will generate an URL , where your model will be hosted by Teachable Machine for free. You can also download it in the form of Tensorflow or Tensorflow Lite for local uses.

The model used in the project is live at: Project Model

Step 3: Preparing the Manifest.json

The next step involves creating a Chrome browser extension to render the following model to the Google Meet URL. So, we have to create the manifest.json file in the browser directory which will be loaded in the browser. So let's have a look at the json file:

{
    "name": "Gmeet_Controller",
    "description": "An extension to control Google Meet Actions using hand/facial gestures.",
    "permissions": ["activeTab", "storage", "tabs", "notifications"],
    "version": "1.0",
    "manifest_version": 3,
    "content_scripts": [{
        "matches": [
            "https://meet.google.com/*"
        ],
        "js": [
            "src/tf.min.js",
            "src/teachablemachine-pose.min.js",
            "src/background.js"
        ]
    }]
}

You can set the name, description and version according to your choice, which is basically the information which will be displayed when you will load the extension on to the browser. You can also store icons in a separate directory in your extension directory and render them on to the browser. The URL needs to be specified in "matches" under content scripts. You can set permissions as well, if you do not want the extension to have unnecessary access to your system storage, notifications etc.

Step 4: Linking your models to Google Meet

Now create a separate directory to store the javascript files (here src), which will render the functionality of the extension. Download the latest and updated version of tf.min.js and teachablemachine-pose.min.js, and place them in the directory.

Note: This project can also be accomplished using npm and yarn packages such as "@teachablemachine/pose" and "@tensorflow/tfjs" or using their CDNs. However, we have used Vanilla JS to make the project simple and beginner-friendly.

After importing, it is now time to frame the background.js. First, declare a constant variable named URL which will contain the model URL that was generated by Teachable Machine.

 const URL = "https://teachablemachine.withgoogle.com/models/<MODEL_ID>/";
// Use your own personalized model here

Then you have to write the basic Javascript to adjust the frame height and width of the webcam, request access from the webcam and keep on updating the webcam frame.

let model, webcam, ctx, labelContainer, maxPredictions;

async function init() {
    const modelURL = URL + "model.json";
    const metadataURL = URL + "metadata.json";

    model = await tmPose.load(modelURL, metadataURL);
    maxPredictions = model.getTotalClasses();

    const size = 200;
    const flip = true; 
    webcam = new tmPose.Webcam(size, size, flip); 
    await webcam.setup(); 
    await webcam.play();
    window.requestAnimationFrame(loop);

    document.getElementById("webcam-container").appendChild(webcam.canvas);
}

async function loop(timestamp) {
    webcam.update();
    await predict();
    window.requestAnimationFrame(loop);
}

Now we have to write the functions to identify the event on the webcam, compare it with the classes of the models and if the event's probability is more than the threshold probability then the function gets executed.


function Audio(probability) {
    if (probability >= 1) {
        const audioButton = document.querySelectorAll(".VfPpkd-Bz112c-LgbsSe.yHy1rc.eT1oJ.tWDL4c.uaILN")[0];
        if (audioButton.classList.contains("HNeRed")) {
            audioButton.click();
        }
    }
}

function Video(probability) {
    if (probability >= 1) {
        const videoButton = document.querySelectorAll(".VfPpkd-Bz112c-LgbsSe.yHy1rc.eT1oJ.tWDL4c.uaILN")[1];
        if (videoButton.classList.contains("HNeRed")) {
            videoButton.click();
        }
    }
}

function Escape(probability) {
    if (probability >= 1) {
        const Button0 = document.querySelectorAll(".VfPpkd-Bz112c-LgbsSe.yHy1rc.eT1oJ.tWDL4c.uaILN")[0];
        const Button1 = document.querySelectorAll(".VfPpkd-Bz112c-LgbsSe.yHy1rc.eT1oJ.tWDL4c.uaILN")[1];
        if (Button0.classList.contains("HNeRed")) {
            Button0.click();
        }
        if (Button1.classList.contains("HNeRed")) {
            Button1.click();
        }
    }
}

async function predict() {
    const { pose, posenetOutput } = await model.estimatePose(webcam.canvas);
    const prediction = await model.predict(posenetOutput);
    var predictionsArray = prediction.map(function (o, i) {
        return { probability: o.probability.toFixed(2), event: o.className }
    })

    var i;
    var min = predictionsArray[0].probability
    var max = predictionsArray[0].probability
    var event = predictionsArray[0].className;
    var value;
    for (i = 1; i < predictionsArray.length; i++) {
        value = predictionsArray[i].probability
        if (value < min) min = value;
        if (value > max) max = value;
    }
    const index = predictionsArray.findIndex((list) => {
        return list.probability == max;
    })
    event = predictionsArray[index].event;

    if (event === "Audio") {
        Audio(max);
    } else if (event === "Video") {
        Video(max);
    } else if (event === "Escape"){
        Escape(max);
    }
}


const webcamContainer = document.createElement("div");
webcamContainer.id = "webcam-container";
document.body.appendChild(webcamContainer);

init();

Open Google Chrome, and type "chrome://extensions", to navigate to the Extensions window. Now toggle on the Developer mode, and click on Load unpacked. Then open the file directory where your manifest.json is stored. The extension is now ready to work.

Now you can have your own personalized model working on your system!

Demo

Here you can have a short look at the working model of the project:

Resources

You can further refer to the following documentation and tutorials to know more about the libraries and technologies:

Credits: Arju S. Moon

You can find the github repository for the above project here:

RitabrataDas343 / GMeet_Controller

This is a browser extension used for controlling Google Meet using hand/facial expressions.

This article has been written and established by:

Ritabrata DasFollow

Open-Source Fanatic || Full-Stack Developer || Graphic Designer || C.P. Enthusiast || CTF Player

and

Shubham Shantam RajuFollow

Open Source Enthusiast | Sophomore of NIT Durgapur | Member of GNU Linux Users' Group | Full Stack Devleoper (MERN) | Blockchain Enthusiast

Hope you have found this article resourceful.

Have a go through the following links to know more about us and keep yourself updated with the latest stuff:

Do drop a like to the post and comment down below if you have liked the idea and are interested in exploring the domain of Machine-Learning along with us. Any kind of suggestions and propositions are appreciated.

May The Source Be With You! 🐧❤️

How Open-Source changed the world: Cloud Computing

Nirvik Agarwal — Tue, 20 Apr 2021 15:43:05 +0000

The words cloud computing or the cloud have garnered a lot of attention in the context of an average user needing a lot of data and the computing power to process that data in their daily lives, be it saving their photos in google drive, posting pictures on social media or developers deploying applications on serverless platforms.

But what do we actually mean by the cloud?

It’s just a metaphor for the internet. Cloud computing is the on-demand availability of computer resources, especially data storage (cloud storage) and computing power, without direct active management by the user.
By using cloud computing, users and companies don't have to manage physical servers themselves or run software applications on their own machines.

The cloud makes services device-agnostic because the computing and storage take place on servers instead of locally on the user's device.

The cloud hidden in plain sight

A misconception that most people have is that the cloud is some sort of advanced software that only developers can use but in reality, we all are using cloud technology in our daily lives.

One can log in to their Google account from a new device with all their emails, photos, contacts in place.
Seamless sharing and delivery of files and applications is done using cloud-based storage services like Google Drive, Dropbox, etc.
Cloud computing has made real-time collaboration possible on the internet with web applications like Figma and Google Docs running on the cloud.

How Open-Source ties them all up

The heart of Cloud Computing is the kernel that powers it, which to no one's surprise is Linux.
This statement is well supported by the fact that Linux is free and Open Source. Linux systems are in general more stable and reliable than their counterparts and hence become an obvious choice for companies to use to power their servers.

The essential tools above the Operating System including the Web servers, FTP servers, DNS servers, and on and on, are available for Linux first and in a wide variety. Also, most of these technologies are open-source themselves.

Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on-demand, taking care of the servers on behalf of their customers. This is cheaper than setting up your own infrastructure but at the same time, the systems are highly scalable and easy to maintain.

Machine Learning, especially Deep Learning has always been a field that has required a lot of computing power to deliver accurate results and has most been elusive to developers. But with platforms like Google Collab one gets to access powerful GPUs which they can leverage to train state-of-the-art models directly on the cloud.

Impact of Cloud Computing during the Pandemic

The tools and services that are ensuring that our work and our education goes unhindered even in these tough times are mostly powered by the cloud. Be it attending online classes through platforms like Google Classroom and Google meet or attending important meetings all have been made possible with the Cloud.

Open Source has changed the world, it's an undeniable fact. By putting the power of supercomputers in the hands of every user and developers, even smaller companies can leverage the infrastructure previously limited to only the large corporations, and deliver products that push the horizon of technology one step further.

We got more on this topic for you to explore, To know more head over to this YouTube Video -

This article has been co-authored by

Nirvik AgarwalFollow

Computer Science Student | Web Developer | FOSS Enthusiast

Anurag MondalFollow

A student trying to learn as much as humanly possible

We hope you found this insightful.
Do visit our website to know more about us and also follow us on :

Also do not forget to like and comment.

Until then,
stay safe, and May the Source Be With You!

How open source changed the world: The 'WORLD WIDE WEB'

SVS HARSHITH — Mon, 01 Feb 2021 14:09:12 +0000

We all have heard about open source at some point or the other .

So, what is it and why ? how does it work ? and does it really help anything let alone the internet?

Lets answer them one by one.

So for the first question the penguin googled it, and it says :

"denoting software for which the original source code is made freely available and may be redistributed and modified."

So , open source is anything that can be used by anyone freely . It is software with source code that anyone can inspect, modify, and enhance.

So in a world which is predominantly capitalistic, why go for free ?

many reasons -

Freedom
Privacy
Community
its free ,well duh !

Open source code is usually stored in a public repository and shared publicly. Anyone can access the repository to use the code independently or contribute improve the project.

Open source software usually comes with a distribution license.
Now a license is what turns a code into an Open source Component .

Its awesome , isn't it ?
To add more to it, here is a list of how it helps the world :

Open standards
Lower cost for the software
More reliable and better quality software
The Internet

The list goes on and on , and yes The Internet .
a thing which we cant live without and it cant live without open source .

To understand how open source is indispensable to the web, we'll need to understand how it started and and how it works .

Tim Berners-Lee, a British scientist, invented the World Wide Web (WWW) in 1989, while working at CERN. It was started so that scientist in various universities around the world could share information among themselves.

On 30 April 1993, CERN put the World Wide Web software in the public domain. Later, CERN made a release available with an open licence, a more sure way to maximise its dissemination.

This is how making the web open source allowed it to flourish.

But this was just the beginning, open sources still has a large role play in the world wide web . as mentioned earlier we need to understand how the web works first to establish this point .

Working of the web -

The computer using the internet is connected to the Internet Service Provider via cables or wireless connections.
Further the ISP is connected to the DNS with the help overhead cables . the DNS sends the details of the webpage you're looking for the to the ISP.
The DNS is connected to the respective servers hosting the website .
The server now in response to the request made sends back data to the computer in the form of packets.

The fundamental functions of the Internet are built on open source technologies. The Linux operating system regulates the operation of Web servers, and the Apache Web server application negotiates data transfer between the worldwide server infrastructure and personal devices.

Countless Internet applications are also open source, some of which are :

Browsers : most of the browsers that we use today are open source , the biggest example would be Firefox which is the 2nd most used web browser
Internet Protocols : many of communication protocols today are open source .
Tech stacks : these are the technology services used to build and run an application . a majority of which are open source , with the help of the open source community various technologies are built and maintained so that anyone can use it or even contribute towards it .

The presence of open source on the Web is so profound that you can essentially build your own Internet , as every aspect of the internet can be built on Open Source software and protocol.

We got more on this topic for you to explore ,To know more head over to this YouTube Video -

We hope you found this insightful.
Do visit our website to know more about us and also follow us on :

Also do not forget to like and comment.

Until then,
stay safe, and May the Source Be With You!

Learn the basics of Socket.io by making a Multiplayer Game

Souparno Paul — Wed, 13 Jan 2021 12:41:54 +0000

With the increasing demand for Multiplayer Games in today's world, developers must take note of the technologies that go into creating such thoroughly enjoyable and immersive games, while at the same time, keeping in mind the challenges that come with them. Real-time communication of data between players is key when it comes to creating multiplayer games, and there are various libraries capable of handling the intricacies entailed within. One such popular library is Socket.io, which finds major use in the creation of chat applications, real-time collaborative environments, games and whatnot.

So, we decided to dabble into the basics of creating a multiplayer game that would appeal to many and at the same time be simple enough to code. This is when it crossed our minds to recreate Snakes and Ladders, the iconic board game that a lot of us spent countless hours on, growing up.

Prerequisites

There aren't any prerequisites as such, as we will be coding the game from scratch. However, some basic knowledge of setting up an Express server on Node.js and some Vanilla JS would ensure a thorough understanding of the topics covered.

The Project

The entire project has been divided into the following sections for clarity and the separation of concerns:

What we'll be making

Let's formulate what we need to do to achieve the desired result. First of all, we need a bare-minimum server that will send requests to all the connected clients. We'll need to set up socket connections for real-time communication. Finally, we'll need some frontend HTML, CSS and Vanilla JS code for the game logic.

Downloading the Starter Project

We have provided the starter code for the project so that you can directly get down to coding the important stuff without having to go through the trouble of having to organize all the game assets and files into relevant folders. A completely written css file has also been provided to eliminate the need for styling the html components from scratch, as it is not directly related to the purpose of the article. You are always free to include your own custom css if you want to, but it won't be necessary. You can download the starter project here.

Installing the necessary packages

Once you have downloaded the starting files, you need to install the necessary packages. Inside the main folder, you will find the package.json file. Run the following command to install the required packages, namely, express, socket.io and http:



npm install

You must have Node.js installed to run the command. If Node.js is not installed, go to the Node.js official website, as linked above and download the latest version for your desired operating system. Once downloaded and installed, run the command again.

Setting up the server

We begin by setting up our express server and socket.io. Write the following code inside the server.js file:



const express = require("express");
const socket = require("socket.io");
const http = require("http");

const app = express();
const PORT = 3000 || process.env.PORT;
const server = http.createServer(app);

// Set static folder
app.use(express.static("public"));

// Socket setup
const io = socket(server);

server.listen(PORT, () => console.log(`Server running on port ${PORT}`));

The Skeleton

All the front-end code for a Node.js and Express project normally goes into a public folder, which we have already specified inside server.js. Before proceeding to write the game logic, it is important to create an html file with the necessary components for the user to be able to interact with the game. Go ahead and include the following code in the index.html file inside the public folder:



<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Snakes and Ladders</title>
    <link
      href="https://fonts.googleapis.com/css?family=Roboto"
      rel="stylesheet"
    />
    <link rel="stylesheet" href="css/styles.css" />
  </head>
  <body>
    <div class="board"></div>
    <img src="images/red_piece.png" alt="" hidden="true" id="red-piece" />
    <img src="images/blue_piece.png" alt="" hidden="true" id="blue-piece" />
    <img src="images/yellow_piece.png" alt="" hidden="true" id="yellow-piece" />
    <img src="images/green_piece.png" alt="" hidden="true" id="green-piece" />
    <div class="container">
      <canvas id="canvas"></canvas>
    </div>
    <div class="info-box">
      <div class="form-group">
        <input
          type="text"
          class="form-input"
          id="name"
          placeholder="Your name"
          required
        />
        <button class="btn draw-border" id="start-btn">Join</button>
      </div>
    </div>
    <div id="players-box">
      <h3>Players currently online:</h3>
      <br>
      <table id="players-table"></table>
    </div>
    <div id="current-player"></div>
    <button class="btn draw-border" id="roll-button" hidden>Roll</button>
    <div class="dice">
      <img src="./images/dice/dice1.png" alt="" id="dice" />
    </div>
    <button class="btn draw-border" id="restart-btn" hidden>Restart</button>
    <script src="/socket.io/socket.io.js"></script>
    <script src="js/index.js"></script>
  </body>
</html>

The index.html file will contain a very special element, the canvas, where our game would come to life. The canvas tag is used to draw graphics using Javascript. It has built-in functions for drawing simple shapes like arcs, rectangles, lines. It can also display text and images.

For socket.io to be able to communicate with the back-end express server from the front-end, we add the following script tag:



<script src="/socket.io/socket.io.js"></script>

Finally, we use another script tag pointing to the index.js file, which will hold the game logic as well as the code for socket communication across the clients.

Setting up socket connection

The way that Socket.io works, is pretty simple. Essentially, the clients emit certain events, which the server can listen to and in turn pass them on to all or a select portion of the clients, that find a use for that information. To establish the connection, we need to add the connection event listener to the io object in the server.js file as follows:



io.on("connection", (socket) => {
  console.log("Made socket connection", socket.id);
});

This tells the server to establish a socket connection with all the clients and display the id of the sockets as soon as the connection is established. The console.log statement serves as a way to ensure a successful connection in case things go wrong.

At the same time, inside the index.js file under the public folder, add the following code:



const socket = io.connect("http://localhost:3000");

This tells the socket to connect to the front-end of the client, which is available at the mentioned URL.

The Game Logic

Now, we will divert our focus to the logic that dictates the game. We will write all the code in the index.js file. The entire logic can be divided into the following sub-categories:

Initialization - We declare the following global variables:



let canvas = document.getElementById("canvas");
canvas.width = document.documentElement.clientHeight * 0.9;
canvas.height = document.documentElement.clientHeight * 0.9;
let ctx = canvas.getContext("2d");

let players = []; // All players in the game
let currentPlayer; // Player object for individual players

const redPieceImg = "../images/red_piece.png";
const bluePieceImg = "../images/blue_piece.png";
const yellowPieceImg = "../images/yellow_piece.png";
const greenPieceImg = "../images/green_piece.png";

const side = canvas.width / 10;
const offsetX = side / 2;
const offsetY = side / 2 + 20;

const images = [redPieceImg, bluePieceImg, yellowPieceImg, greenPieceImg];

const ladders = [
  [2, 23],
  [4, 68],
  [6, 45],
  [20, 59],
  [30, 96],
  [52, 72],
  [57, 96],
  [71, 92],
];

const snakes = [
  [98, 40],
  [84, 58],
  [87, 49],
  [73, 15],
  [56, 8],
  [50, 5],
  [43, 17],
];

First of all, we set the size of the canvas to match the dimensions of the game board and get the context of the canvas, which will be required to draw player-pins. After this, we declare a collection players, which will be required to keep track of the players currently in the game and a currentPlayer that stores a reference to the player who is playing the game on the particular front-end client. Then we store references to the four player-pins, namely, red, blue, yellow and green. We initialize the variables side, offsetX and offsetY which will be required to adjust the position of the player-pins on the canvas. Finally, the variables ladders and snakes are initialized, which are collections that store the set of points connected by ladders and snakes respectively, as depicted on the game board. This will be required to alter the position of the player-pins when the land on a square with a ladder or a snake.

The Player class - We wanted to use an OOP(Object Oriented Programming) paradigm to represent the players, which makes it easier to assign relevant properties and functions. The Player class is modelled as follows:



class Player {
  constructor(id, name, pos, img) {
    this.id = id;
    this.name = name;
    this.pos = pos;
    this.img = img;
  }

  draw() {
    let xPos =
      Math.floor(this.pos / 10) % 2 == 0
        ? (this.pos % 10) * side - 15 + offsetX
        : canvas.width - ((this.pos % 10) * side + offsetX + 15);
    let yPos = canvas.height - (Math.floor(this.pos / 10) * side + offsetY);

    let image = new Image();
    image.src = this.img;
    ctx.drawImage(image, xPos, yPos, 30, 40);
  }

  updatePos(num) {
    if (this.pos + num <= 99) {
      this.pos += num;
      this.pos = this.isLadderOrSnake(this.pos + 1) - 1;
    }
  }

  isLadderOrSnake(pos) {
    let newPos = pos;

    for (let i = 0; i < ladders.length; i++) {
      if (ladders[i][0] == pos) {
        newPos = ladders[i][1];
        break;
      }
    }

    for (let i = 0; i < snakes.length; i++) {
      if (snakes[i][0] == pos) {
        newPos = snakes[i][1];
        break;
      }
    }

    return newPos;
  }
}

Each Player object requires an id, a name, a position on the board denoted by pos and a pin image as denoted by img. We then write the functions draw, updatePos and isLadderOrSnake respectively, to draw and update the position of the players and to find whether the player's square on the board has a ladder or a snake. The updatePos method just updates pos with the number that the player just rolled on the dice and checks a condition that keeps the player from going beyond the 100th square on the board. One thing to note here is that the player's position, though starts at 1, is denoted by 0, which makes the drawing logic simpler. The isLadderOrSnake function takes an argument as the position of the player and compares it with the squares in the collection ladders and snakes and accordingly returns the new position of the player on the board. The draw function might seem a bit complicated but all it does is draw the player-pins on the correct squares on the board. The function takes care of the alternate right and left movement across rows and the upward movement across columns.

The utility functions - Apart from the functions that we wrote inside the Player class, we need to write two more utility functions as follows:



function rollDice() {
  const number = Math.ceil(Math.random() * 6);
  return number;
}

function drawPins() {
  ctx.clearRect(0, 0, canvas.width, canvas.height);

  players.forEach((player) => {
    player.draw();
  });
}

The rollDice function returns a random number between 1 and 6 while the drawPins function cycles through the players collection and draws the respective player-pins using their draw function.

Firing socket events and handling them - So far, we have written the code for the game entities. However, if we don't fire socket events from the front-end, none of the players will be able to communicate their positions and other data among themselves. First of all, add the following line of code below the io.connect function in the index.js file:



socket.emit("joined");

Then add the following event listeners to the relevant html elements as follows:



document.getElementById("start-btn").addEventListener("click", () => {
  const name = document.getElementById("name").value;
  document.getElementById("name").disabled = true;
  document.getElementById("start-btn").hidden = true;
  document.getElementById("roll-button").hidden = false;
  currentPlayer = new Player(players.length, name, 0, images[players.length]);
  document.getElementById(
    "current-player"
  ).innerHTML = `<p>Anyone can roll</p>`;
  socket.emit("join", currentPlayer);
});

document.getElementById("roll-button").addEventListener("click", () => {
  const num = rollDice();
  currentPlayer.updatePos(num);
  socket.emit("rollDice", {
    num: num,
    id: currentPlayer.id,
    pos: currentPlayer.pos,
  });
});

document.getElementById("restart-btn").addEventListener("click", () => {
  socket.emit("restart");
});

The joined event emitted by the socket informs a new player who has just joined the game about the players who have already joined the game, which means their position and their pin image. That is why it is fired as soon as a new user joins. Following this, we have added three click event listeners to the start button, roll button and restart button. The start button takes the name of the newly joined player and creates a new currentPlayer object. Following this, a few of html tags are manipulated to convey the status of the game, following which a join event is emitted, which notifies the server of the newly joined player. The roll button event listener simply rolls the dice and updates the position of the currentPlayer and sends the number rolled on the dice along with their id and name. The restart button as the name suggests fires a restart event from the front-end.

We also need to be able to receive these events on the server-side. Write the code as provided below inside the connection event listener of the io object3:



socket.on("join", (data) => {
    users.push(data);
    io.sockets.emit("join", data);
  });

  socket.on("joined", () => {
    socket.emit("joined", users);
  });

  socket.on("rollDice", (data) => {
    users[data.id].pos = data.pos;
    const turn = data.num != 6 ? (data.id + 1) % users.length : data.id;
    io.sockets.emit("rollDice", data, turn);
  });

  socket.on("restart", () => {
    users = [];
    io.sockets.emit("restart");
  });
});

The backend has the same event listeners, along with a users collection, which stores and relays information about the players who are playing.

We also need to be able to handle these events on the front-end and the code for that is:



socket.on("join", (data) => {
  players.push(new Player(players.length, data.name, data.pos, data.img));
  drawPins();
  document.getElementById(
    "players-table"
  ).innerHTML += `<tr><td>${data.name}</td><td><img src=${data.img} height=50 width=40></td></tr>`;
});

socket.on("joined", (data) => {
  data.forEach((player, index) => {
    players.push(new Player(index, player.name, player.pos, player.img));
    console.log(player);
    document.getElementById(
      "players-table"
    ).innerHTML += `<tr><td>${player.name}</td><td><img src=${player.img}></td></tr>`;
  });
  drawPins();
});

socket.on("rollDice", (data, turn) => {
  players[data.id].updatePos(data.num);
  document.getElementById("dice").src = `./images/dice/dice${data.num}.png`;
  drawPins();

  if (turn != currentPlayer.id) {
    document.getElementById("roll-button").hidden = true;
    document.getElementById(
      "current-player"
    ).innerHTML = `<p>It's ${players[turn].name}'s turn</p>`;
  } else {
    document.getElementById("roll-button").hidden = false;
    document.getElementById(
      "current-player"
    ).innerHTML = `<p>It's your turn</p>`;
  }

  let winner;
  for (let i = 0; i < players.length; i++) {
    if (players[i].pos == 99) {
      winner = players[i];
      break;
    }
  }

  if (winner) {
    document.getElementById(
      "current-player"
    ).innerHTML = `<p>${winner.name} has won!</p>`;
    document.getElementById("roll-button").hidden = true;
    document.getElementById("dice").hidden = true;
    document.getElementById("restart-btn").hidden = false;
  }
});

socket.on("restart", () => {
  window.location.reload();
});

Most of the socket event listeners are fairly simple, and a good look at the function statements tells you that all we do here is display the current status of the game by disabling and enabling the necessary html elements.

Finishing up

Now that everything is in place, it is time to fire up the terminal and run node server.js, which exposes the Node.js server to the port 3000 of localhost. Following this, you can visit http://localhost:3000 with multiple browser windows and test out the game.

Takeaways

This project is meant to serve an entry point to the endless possibilities of the realm of multiplayer games and socket communication. We have aimed at explaining the absolute basics here and there is room for a lot of improvement. For instance, currently, the game allows only 4 players to play concurrently but in reality, such a game is supposed to have specific rooms which players can join, thus allowing several players to concurrently access the game. You can also add an in-game chatbox, where players can chat with each other while playing. The movement of the player-pins on the canvas is also instantaneous, which isn't that attractive. It is highly recommended that you try out adding such features to the game to get an even stronger grasp of the underlying technicalities.

Resources

Soupaul / snakes-and-ladders-multiplayer

A multiplayer Snakes and Ladders Game made using NodeJS and Socket.IO

The master branch contains the completed project, whereas the starter branch provides the starting code.

This article was co-authored by:

Souparno PaulFollow

Computer Science Student and a FOSS enthusiast

and

Nirvik AgarwalFollow

Computer Science Student | Web Developer | FOSS Enthusiast

We hope you found this insightful.
Do visit our website to know more about us and also follow us on :

Also, don't forget to drop a like and comment below if you are interested in learning more about game development using Javascript. You can freely raise doubts and suggest improvements.

Until then,
Stay Safe and May The Source Be With You!

Stonksmaster: Predict Stock prices using Python and ML - Part II

Rishav Raj Kumar — Thu, 10 Dec 2020 11:43:33 +0000

*This is follow up article from our previous post.*

Stonksmaster - Predict Stock prices using Python & ML 📈

Nirvik Agarwal for GNU/Linux Users' Group, NIT Durgapur ・ Dec 2 '20

#machinelearning #python #beginners #tutorial

In the previous post we discussed the basics of **Machine Learning** and its *regression* models for *stock prices prediction*. Today, let us talk about *ensemble methods* and *boosting models* used in *supervised Machine Learning.*

Ensemble Methods

Ensemble methods is a Machine Learning technique that uses multiple machine learning algorithms together to obtain a better predictive performance that could not be obtained from any of the constituent algorithms alone. The way the base models are combined can range from simple methods like averaging or max voting to more complex like Boosting or Stacking.

To draw a basic comparison we could say, let us imagine a parliament filled with members voting for a particular bill, or expressing their opinion. Let the members be models and their opinions are the results of the models. So to decide if the bill passes or not depends on the members and their votes. Similarly, in Ensemble learning the final outcome is based on the outcome of each of its base models too. This could be the exact situation in the case of two-fold classification problems. But the basic idea is the same for all other ensemble methods too. We hope it would have made the picture more clear.

Over the past few years, Ensemble learning techniques have seen a huge jump in popularity and the reason behind this is quite obvious, they help us to build a really robust model from a few weak base models.

Ensemble Approaches

There are multiple ways to use ensemble models. There are simple ones like max voting or averaging as well as more complex ones like boosting, bagging or stacking.

Majority Voting

There are three versions of majority voting, where the ensemble methods choose the class -

on which all classifiers agree
predicted by at least one more than half the number of classifiers
that receives the highest number of votes, whether or not the sum of those votes exceeds 50%

Averaging

Averaging is a simple method that is generally used for regression problems. Here we simply take the average of the predictions to get a more robust result. Although this method might look pretty simple it almost always gives better results than a single model hence serving the purpose of ensemble learning.

Weighted Average

Averaging the predictions provides great results but it has a major flow and that is in most cases, one model has more predictive power or performs better based on the input problem than the other and hence we want to give it more weight on the final predictions.

Bagging

Bagging is a popular ensemble method that is used in algorithms like Random Forest. It gains accuracy not only by averaging the models but by also creating uncorrelated models which are made possible by giving them different training sets.

Boosting

Boosting is a sequential process, where each subsequent model tries to correct the errors of the previous model. Due to this reason, the succeeding models are dependent on the previous models and so we train the model sequentially instead of training them parallelly.

Gradient Boosting Regressor

In this tutorial, we will be focusing on and using Gradient Boosting Regression (GBR) on the dataset we used in the previous tutorial which was used for stock prediction.

The term gradient in gradient boosting comes from the fact that the algorithm uses gradient descent used to minimize the loss.
Read more on Gradient Descent :

Decision Trees is used as weak learners or base models in the gradient boosting algorithm. Decision Trees transform data into a tree representation. Each internal node of the tree representation denotes an attribute and each leaf node denotes a class label.

How do decision trees look?

Like the Image above? Definitely not ;)

It looks like this :)

Advantages of Gradient Boosting

Better accuracy and precision than most base models like Random Forests
Less pre-processing of data is required
Higher flexibility to produce better on a variety of types of input data

Gradient Boosting parameters

Number of Estimators
It is denoted as n_estimators.
The default value of this parameter is 100.
The number of boosting stages to perform. Gradient boosting is fairly robust to over-fitting so a large number usually results in better performance. In other words, the number of estimators denotes the number of trees in the forest. More number of trees helps in learning the data better. On the other hand, more number of trees can result in higher training time. Hence we need to find the right and balanced value of n_estimators for optimal performance.

Maximum Depth
It is denoted as max_depth.
The default value of max_depth is 3 and it is an optional parameter. The maximum depth is the depth of the decision tree estimator in the gradient boosting regressor. The maximum depth limits the number of nodes in the tree. Tune this parameter for best performance; the best value depends on the interaction of the input variables. We need to find the optimum value of this parameter for best performance.

Learning Rate
It is denoted as learning_rate.
The default value of learning_rate is 0.1 and it is an optional parameter. The learning rate is a hyper-parameter in gradient boosting regressor algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.

Criterion
It is denoted as criterion.
The default value of criterion is friedman_mse and it is an optional parameter. The function to measure the quality of a split. Criterion is used to measure the quality of a split for decision tree. mse stands for mean squared error. The default value of friedman_mse is generally the best as it can provide a better approximation in some cases.

Loss
It is denoted as loss.
The default value of the loss is ls and it is an optional parameter. This parameter indicates the loss function to be optimized. There are various loss functions like ls which stands for least squares regression. The least absolute deviation abbreviated as lad is another loss function. huber is a combination of the two. quantile allows quantile regression (use alpha to specify the quantile).

Subsample
It is denoted as subsample.
The default value of subsample is 1.0 and it is an optional parameter. Subsample is the fraction of samples used for fitting the individual tree learners. If subsample is smaller than 1.0 this leads to a reduction of variance and an increase in bias.

Number of Iteration no change
It is denoted by n_iter_no_change.
The default value of subsample is None and it is an optional parameter. This parameter is used to decide whether early stopping is used to terminate training when validation score is not improving with further iteration. If this parameter is enabled, it will set aside validation_fraction size of the training data as validation and terminate training when validation score is not improving.

For more insight on parameters of this model, refer to this documentation

Improvements to Basic Gradient Boosting

Gradient boosting is a greedy algorithm and can overfit a training dataset very quickly. Therefore we need to tune the model or improve it using various techniques.

Following are a few ways to enhance the performance of a basic gradient boosting algorithm :

Tree Constraints
Shrinkage
Random sampling
Penalized Learning

For reference on GBR tuning read here.

Basic Implementation

Following is the basic implementation of Gradient Boosting Regressor used on the Iexfinance dataset imported in previous part of this article :

# GBR
from sklearn import ensemble
# Fit regression model
params = {'n_estimators': 500, 'max_depth': 4, 'min_samples_split': 2,
          'learning_rate': 0.01, 'loss': 'ls'}
model = ensemble.GradientBoostingRegressor(**params)
model.fit(x_training_set, y_training_set)

from sklearn.metrics import mean_squared_error, r2_score
model_score = model.score(x_training_set,y_training_set)
# Have a look at R sq to give an idea of the fit ,
# Explained variance score: 1 is perfect prediction
print('R2 sq: ',model_score)
y_predicted = model.predict(x_test_set)

# The mean squared error
print("Mean squared error: %.2f"% mean_squared_error(y_test_set, y_predicted))
# Explained variance score: 1 is perfect prediction
print('Test Variance score: %.2f' % r2_score(y_test_set, y_predicted))

We hope you found this insightful.
Do visit our website to know more about us and also follow us on :

Also do not forget to like and comment.

Until then,
stay safe, and May the Source Be With You!

***This article is co-written by***

Rishav Raj KumarFollow

Web Developer | ML Enthusiast | Blockchain | UI Developer | Graphic Designer

Nirvik AgarwalFollow

Computer Science Student | Web Developer | FOSS Enthusiast

Stonksmaster - Predict Stock prices using Python & ML 📈

Nirvik Agarwal — Wed, 02 Dec 2020 11:49:21 +0000

Newbie to Machine Learning?
Need a nice initial project to get going?

You are on the right article!

In this article, we will try to build a very basic stock prediction application using Machine Learning and its concepts. And as the name suggests it is gonna be useful and fun for sure. So let's get started.

We expect you to have a basic exposure to Data Science and Machine Learning.

"The field of study that gives computers the ability to learn without being explicitly programmed"

is what Arthur Samuel described as Machine Learning.

Machine Learning has found its applications in various fields in recent years, some of which include Virtual Personal Assistants, Online Customer Support, Product Recommendations, etc.

We will use libraries like numpy, pandas, matplotlib, scikit-learn, and a few others.

If you are not familiar with these libraries, you can refer to the following resources:

Steps in Machine Learning

While performing any Machine Learning Task, we generally follow the following steps:

Collecting the data
This is the most obvious step. If we want to work on an ML Project we first need data. Be it the raw data from excel, access, text files, or data in the form of images, video, etc., this step forms the foundation of future learning.
Preparing the data
Bad data always leads to bad insights that lead to problems. Our prediction results depend on the quality of the data used. One needs to spend time determining the quality of data and then taking steps for fixing issues such as missing data etc.
Training the model
This step involves choosing the appropriate algorithm and representation of data in the form of the model. In layman terms, model representation is a process to represent our real-life problem statement into a mathematical model for the computer to understand. The cleaned data is split into three parts – Training, Validation, and Testing - proportionately depending on the scenario. The training part is then given to the model to learn the relationship/function.
Evaluating the model
Quite often, we don’t train just one model but many. So, to compare the performance of the different models, we evaluate all these models on the validation data. As it has not been seen by any of the models, validation data helps us evaluate the real-world performance of models.
Improving the Performance
Often, the performance of the model is not satisfactory at first and hence we need to revisit earlier choices we made in deciding data representations and model parameters. We may choose to use different variables (features) or even collect some more data. We might need to change the whole architecture to get better performance in the worst case.
Reporting the Performance
Once we are satisfied by the performance of the model on the validation set, we evaluate our chosen model on the testing data set and this provides us with a fair idea of the performance of our model on real-world data that it has not seen before.

Now coming to our project, as we are dealing with the stock market and trying to predict stock prices the most important thing is being able to Read Stocks

How to Read Stocks?

Reading stock charts, or stock quotes is a crucial skill in being able to understand how a stock is performing, what is happening in the broader market, and how that stock is projected to perform.

Stocks have quote pages or charts, which give both basic and more detailed information about the stock, its performance, and the company on the whole. So, the next question that comes up is what makes up a stock chart?

Stock Charts

A Stock Chart is a set of information on a particular company's stock that generally shows information about price changes, current trading price, historical highs and lows, dividends, trading volume, and other company financial information.

Also we would like to familiarise you some basic terminologies of the stock market

Ticker Symbol

The ticker symbol is the symbol that is used on the stock exchange to delineate a given stock. For example, Apple's ticker is (AAPL) while Snapchat's ticker is (SNAP).

All stock ticker symbols

Open Price

The open price is simply the price at which the stock opened on any given day

Close Price

The close price is perhaps more significant than the open price for most stocks. The close is the price at which the stock stopped trading during normal trading hours (after-hours trading can impact the stock price as well). If a stock closes above the previous close, it is considered an upward movement for the stock. Vice versa, if a stock's close price is below the previous day's close, the stock is showing a downward movement.

Now its time to get your hands dirty and begin setting up the project

Initializing our project

Step 1 : Collecting the data

Use the iexfinance library to download the dataframe. The dataframe which we get contains daily data about the stock. The downloaded dataframe gives us a lot of information including Opening Price, Closing Price, Volume, etc. But we are interested in the opening prices with their corresponding dates.

import pandas as pd
import numpy as np
import iexfinance
from iexfinance.stocks import get_historical_data
from datetime import datetime, date

# start date should be within 5 years of current date according to iex API we have used
# The more data we have, the better results we get!

start = datetime(2016, 1, 1)
end = date.today()
# use your token in place of token which you will get after signing up on IEX cloud
# Head over to https://iexcloud.io/ and sign-up to get your API token
df = get_historical_data("AAPL", start=start, end=end, output_format="pandas", token="your_token")

Step 2 : Preparing the data

Also, it would convenient to convert the dates to their corresponding time-stamps. And finally, we will be having a dataframe which will contain our opening prices and time-stamps.

We need to know that the model we created is good. We are going to hold back some data that the algorithms will not get to see and we will use this data to get a second and independent idea of how accurate the best model might actually be.

We will split the loaded dataset into two, 80% of which we will use to train, evaluate, and select among our models, and 20% that we will hold back as a validation dataset.

from sklearn.model_selection import train_test_split

prices = df[df.columns[0:1]]
prices.reset_index(level=0, inplace=True)
prices["timestamp"] = pd.to_datetime(prices.date).astype(int) // (10**9)
prices = prices.drop(['date'], axis=1)
prices

dataset = prices.values
X = dataset[:,1].reshape(-1,1)
Y = dataset[:,0:1]

validation_size = 0.15
seed = 7

X_train, X_validation, Y_train, Y_validation = train_test_split(X, Y, test_size=validation_size, random_state=seed)

The function train_test_split() comes from the scikit-learn library.

scikit-learn (also known as sklearn) is a free software machine learning library for Python. Scikit-learn provides a range of supervised and unsupervised learning algorithms via a consistent interface in Python.
The library is focused on modeling data. It is not focused on loading, manipulating, and summarizing data.

Step 3 : Training the model

We don’t know which algorithms would be good on this project or what configurations to use.

And So, we are testing with 6 different algorithms:

Linear Regression (LR)
Lasso (LASSO)
Elastic Net (EN)
KNN (K-Nearest Neighbors)
CART (Classification and Regression Trees)
SVR (Support Vector Regression)

from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Lasso
from sklearn.linear_model import ElasticNet
from sklearn.tree import DecisionTreeRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.svm import SVR

# Test options and evaluation metric
num_folds = 10
seed = 7
scoring = "r2"

# Spot-Check Algorithms
models = []
models.append((' LR ', LinearRegression()))
models.append((' LASSO ', Lasso()))
models.append((' EN ', ElasticNet()))
models.append((' KNN ', KNeighborsRegressor()))
models.append((' CART ', DecisionTreeRegressor()))
models.append((' SVR ', SVR()))

Step 4 : Evaluating the model

from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

# evaluate each model in turn
results = []
names = []
for name, model in models:
    kfold = KFold(n_splits=num_folds, random_state=seed, shuffle=True)
    cv_results = cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring)
    # print(cv_results)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)

The output of the above code gives us the accuracy estimations for each of our algorithms. We need to compare the models to each other and select the most accurate.

Once we choose which results in the best accuracy, all we have to do is to

Define the model
Fit data into our model
Make predictions

Plot your predictions along with the actual data and the two plots will nearly overlap.

Step 5 : Reporting the model and making prediction

# Future prediction, add dates here for which you want to predict
dates = ["2020-12-23", "2020-12-24", "2020-12-25", "2020-12-26", "2020-12-27",]
#convert to time stamp
for dt in dates:
  datetime_object = datetime.strptime(dt, "%Y-%m-%d")
  timestamp = datetime.timestamp(datetime_object)
  # to array X
  np.append(X, int(timestamp))

from matplotlib import pyplot as plt
from sklearn.metrics import mean_squared_error

# Define model
model = DecisionTreeRegressor()
# Fit to model
model.fit(X_train, Y_train)
# predict
predictions = model.predict(Xp)
print(mean_squared_error(Y, predictions))

# %matplotlib inline 
fig= plt.figure(figsize=(24,12))
plt.plot(X,Y)
plt.plot(X,predictions)
plt.show()

Hurrah! You finally built a Stock Predictor. We hope this article was of great help to beginners and everyone else alike. For those who are interested in taking this project to the next level, we recommend you to read on LSTMs neural nets and try implementing it.

Though we are predicting the prices, this model is practically not viable because a lot of other factors have to be considered while making predictions!

Model References

Update: We have made a new post following this article in which we have used Ensemble Methods to further enhance our models.

Stonksmaster: Predict Stock prices using Python and ML - Part II

Rishav Raj Kumar for GNU/Linux Users' Group, NIT Durgapur ・ Dec 10 '20

#machinelearning #python #tutorial #programming

We hope you found this insightful.

This was one of the projects in 10 Days of Code organized by GNU/Linux Users' Group, NIT Durgapur

Do visit our website to know more about us and also follow us on :

This article has been co-authored by

Nirvik AgarwalFollow

Computer Science Student | Web Developer | FOSS Enthusiast

Rishav Raj KumarFollow

Web Developer | ML Enthusiast | Blockchain | UI Developer | Graphic Designer