Forem: Arindam Dawn

How to Add Progress Bars and Status Indicators to Your Favicon

Arindam Dawn — Wed, 25 Feb 2026 11:32:12 +0000

magic-favicon 🪄

Ever noticed how GitHub shows build status in your browser tab? That little icon tells you if a build passed or failed without switching tabs. I wanted the same for my projects, so I built magic-favicon - a tiny (~2.5KB) library that transforms your favicon into dynamic indicators.

What it does

favicon.progress(50);   // Shows 50% progress bar
favicon.pie(75);        // Shows pie chart
favicon.badge(3);       // Shows notification count
favicon.status("success"); // Shows green checkmark
favicon.pulse();        // Shows pulsing animation

Features

Progress bars and pie charts – Visualize completion with animated overlays
Notification badges – Auto-formatting counts (e.g., 99+)
Status icons – success, warning, error, check, cross
Animations – Pulse and spin effects for attention-grabbing indicators
Zero dependencies – Works in any modern browser
Non-des tructive – Preserves your original favicon when needed

Demo

🔗 Live Demo

Installation
npm install magic-favicon

Usage

import favicon from 'magic-favicon';

// Show progress
favicon.progress(45, { color: '#22c55e' });

// Show badge
favicon.badge(12);

// Show status
favicon.status('success');

// Reset to original favicon
favicon.reset();

💡 The library automatically handles canvas rendering and favicon updates. All you need is a tag in your HTML.

Would love to hear what you think! 🚀

Stop Wrestling with JSON-LD: Type-Safe Structured Data for Next.js

Arindam Dawn — Mon, 16 Feb 2026 08:31:48 +0000

Structured data is one of those things every developer knows they should implement, but few actually do. Why? Because writing JSON-LD by hand is error-prone, tedious, and easy to get wrong.
What if you could generate type-safe structured data with autocomplete, validation, and zero runtime overhead?

The Problem with JSON-LD

Here's what typical JSON-LD looks like:

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "My Blog Post",
  "author": {
    "@type": "Person",
    "name": "Jane Doe"
  },
  "datePublished": "2026-02-10"
}

Looks simple, right? But try maintaining this across dozens of pages:

Did you forget the @context?
Is datePublished in the right format?
Did you typo Article as Aticle?
Is this even valid according to schema.org? These mistakes cost you search rankings and AI visibility.

Enter Schema Sentry

Schema Sentry is a type-safe library for generating JSON-LD structured data. It gives you:
✅ Full TypeScript autocomplete - No more guessing field names

✅ Compile-time validation - Catch errors before they reach production

✅ Zero runtime overhead - Everything happens at build time

✅ Works with both Pages Router and App Router - Your choice of Next.js patterns

✅ CI/CD validation - Automated checks in your pipeline

Quick Example

Instead of hand-writing JSON:

// ❌ Error-prone manual JSON
<script
  type="application/ld+json"
  dangerouslySetInnerHTML={{
    __html: JSON.stringify({
      "@context": "https://schema.org",
      "@type": "Article",
      "headline": "My Post",
      // Did I get all required fields? Who knows!
    })
  }}
/>

Use type-safe builders:

import { Schema, Article } from "@schemasentry/next";
// ✅ Type-safe with autocomplete
const article = Article({
  headline: "My Post",
  authorName: "Jane Doe",
  datePublished: "2026-02-10",
  url: "https://example.com/blog/post",
});
export default function Page() {
  return <Schema data={article} />;
}

Same output, zero guesswork.

Pages Router vs App Router

Good news: Schema Sentry works identically with both Next.js routing patterns.

App Router (Next.js 13+)

// app/blog/[slug]/page.tsx
import { Schema, Article, BreadcrumbList } from "@schemasentry/next";
export default function BlogPost({ params }: { params: { slug: string } }) {
  const article = Article({
    headline: "Getting Started",
    authorName: "Jane Doe",
    datePublished: "2026-02-10",
    url: `https://example.com/blog/${params.slug}`,
  });
  const breadcrumbs = BreadcrumbList({
    items: [
      { name: "Home", url: "https://example.com" },
      { name: "Blog", url: "https://example.com/blog" },
    ],
  });
  return (
    <>
      <Schema data={[article, breadcrumbs]} />
      <article>{/* content */}</article>
    </>
  );
}

Pages Router (Classic Next.js)

// pages/blog/[slug].tsx
import Head from "next/head";
import { Schema, Article, BreadcrumbList } from "@schemasentry/next";
export default function BlogPost() {
  const article = Article({
    headline: "Getting Started",
    authorName: "Jane Doe",
    datePublished: "2026-02-10",
    url: "https://example.com/blog/post",
  });
  const breadcrumbs = BreadcrumbList({
    items: [
      { name: "Home", url: "https://example.com" },
      { name: "Blog", url: "https://example.com/blog" },
    ],
  });
  return (
    <>
      <Head>
        <title>Getting Started - My Blog</title>
      </Head>
      <Schema data={[article, breadcrumbs]} />
      <article>{/* content */}</article>
    </>
  );
}

The only difference? App Router doesn't need next/head because it supports native metadata.

Supported Schema Types

Schema Sentry includes builders for the most common types:

Organization - Company/publisher info
WebSite - Site metadata
Article / BlogPosting - Blog posts and articles
Product - E-commerce products with offers
FAQPage - FAQ sections with expandable questions
HowTo - Step-by-step guides
BreadcrumbList - Navigation breadcrumbs
Person - Author/contributor info ## CI/CD Validation Here's where it gets powerful. Schema Sentry includes a CLI for automated validation:

// schema-sentry.manifest.json
{
  "routes": {
    "/": ["Organization", "WebSite"],
    "/blog": ["WebSite"],
    "/blog/getting-started": ["Article"],
    "/products/widget": ["Organization", "Product"],
    "/faq": ["FAQPage", "Organization"]
  }
}

Add to your GitHub Actions:

# .github/workflows/schema-check.yml
name: Schema Validation
on: [push, pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v2
      - run: pnpm install
      - run: pnpm build
      - run: pnpm schemasentry validate

Now your CI will fail if:

A page is missing expected schema
Schema types don't match the manifest
Invalid JSON-LD is generated Never ship broken structured data again. ## Why Structured Data Matters ### For Traditional SEO Google uses JSON-LD to create rich snippets in search results:
Article headlines and publication dates
Product prices and ratings
FAQ dropdowns
Breadcrumb navigation These rich results get higher click-through rates than plain blue links. ### For AI Discovery This is the big one that most developers are missing. AI systems like:
ChatGPT
Claude
Perplexity
Google Bard ...all use structured data to understand and cite content. Without proper schema, your content is invisible to the AI discovery layer. ## Getting Started

# Install packages
npm install @schemasentry/next
npm install -D @schemasentry/cli
# Or with pnpm
pnpm add @schemasentry/next
pnpm add -D @schemasentry/cli

Create your first schema:

import { Schema, Organization } from "@schemasentry/next";
const org = Organization({
  name: "My Company",
  url: "https://example.com",
  logo: "https://example.com/logo.png",
  description: "We build amazing things",
});
export default function Home() {
  return (
    <>
      <Schema data={org} />
      <h1>Welcome</h1>
    </>
  );
}

That's it! The <Schema /> component injects the JSON-LD into your page's <head> automatically.

Live Examples

Check out the complete working examples:

App Router Example - Modern Next.js 13+ patterns
Pages Router Example - Classic Next.js patterns Both demonstrate the same UI with all 11+ schema types. ## Summary Adding structured data doesn't have to be painful:
Type safety catches errors at compile time
Autocomplete guides you to the right fields
CI validation prevents broken schema from shipping
Zero runtime cost - everything is static
Works with any Next.js routing pattern Your future self (and your search rankings) will thank you. --- Have you implemented structured data on your Next.js site? What challenges did you face? Schema Sentry is open source under MIT license. Star us on GitHub if you found this helpful!

Frontend Shots - Web Workers

Arindam Dawn — Sat, 05 Sep 2020 18:04:14 +0000

Introduction

Frontend Shots is a series of posts where I shall try to explain various important concepts in frontend web development landscape. This post covers a topic which I understood very recently although it has been around for more than a decade!

Web Workers. What are they? Why do we need them? Let's get the answers.

What are Web Workers?

Imagine you have been given a task to fill water in a container of capacity 2litres using a mug. You can do that easily within few seconds.

Now suppose you are asked to next fill up a container of capacity 100litres using the same mug. The idea of it will make you sweat! Instead of doing the work all alone, you request your siblings to help you and you delegate it to them so that you can do some other important work in the meanwhile.

In the JavaScript universe, Web Workers can be thought of something like that. Web Workers can run scripts doing computation-heavy tasks in the background and can communicate with the main thread without freezing the user interface. Browsers first introduced Web Workers in around 2009 to allow programmers to run heavy CPU intensive tasks concurrently utilizing the multi cores of the processor. Web Workers are not a part of JavaScript. It is an API provided by the web browsers just like the browser's Web APIs. Web Workers can do lot of cool stuffs like doing network requests, accessing some window objects like Navigator, XMLHttpRequest, Array, Date, Math, String, window timers. However they cannot directly access the DOM. Web Workers can be thought of as an independent JavaScript environment running in a parallel universe and can communicate with the main JavaScript universe.

Web Workers are also known as "Dedicated Workers".

Why Web Workers?

Whenever any task is performed that requires heavy computation, it makes the UI irresponsive, making it non-interactive. This is not a great experience for end users as they no longer can interact with the UI until the task is executed. A lot of users still use low end feature phones that do not up top level hardware specs. Running heavy CPU intensive operations on such devices often result in an undesirable experience for users. Even a for loop operation can make the UI unresponsive until it has completed execution.

Web Workers come in extremely handy when it comes to optimizing performance and keeping the UI very responsive. The actual time taken for performing the task remains almost the same, however, the user experience is enhanced to a great extent since the user is not blocked from doing any task.

How to use Web Workers?

Let's see how a simple code such as Calculating the Fibonacci number at a specific position can make the UI unresponsive and how we can enhance the user experience to a great extent using Web Workers.

index.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <title>Web Workers Demo</title>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <link rel="stylesheet" type="text/css" media="screen" href="main.css" />
  </head>
  <body>
    <main>
      <h1>Fibonacci Calculator</h1>
      <label for="position">Position</label>
      <input type="text" placeholder="Enter a position" id="position" />
      <button type="button" onclick="showResult(position)">
        Calculate Fibonacci
      </button>
      <p id="answer"></p>
    </main>

    <script>
      function calculateFibonacci(num) {
        if (num < 2) {
          return num;
        } else {
          return calculateFibonacci(num - 1) + calculateFibonacci(num - 2);
        }
      }
      function showResult(num) {
        const answerEl = document.getElementById('answer');
        const position = Number(document.getElementById('position').value);
        const button = document.querySelector('button');
        button.disabled = true;
        button.innerText = 'Loading...';
        const answer = calculateFibonacci(position);
        answerEl.innerText = `The Fibonacci number at position ${position} is ${answer}`;
        button.disabled = false;
        button.innerText = 'Calculate Fibonacci';
       }
    </script>
  </body>
</html>

main.css

#position {
    border: 1px solid #c1c1c1;
    padding: 4px 16px;
    font-size: 18px;
    display: block;
}

button {
    border: none;
    display: block;
    margin-top: 8px;
    padding: 8px 16px;
    background-color: blueviolet;
    color: white;
    font-weight: bold;
    cursor: pointer;
}
button:disabled {
    opacity: 0.5;
    cursor: not-allowed;
}

The above code creates an input text box that accepts any number as position to calculate the Fibonacci number at that position. I have deliberately kept the calculateFibonacci code unoptimized as of now.

Try running this code and enter a high value such as 45 and try to calculate the Fibonacci. The UI instantly becomes unresponsive and cannot be interacted unless the function execution is complete.

Let's try improving the user experience by simply using Web Workers and not doing any kind of optimization to the Fibonacci function.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <title>Web Workers Demo</title>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <link rel="stylesheet" type="text/css" media="screen" href="main.css" />
  </head>
  <body>
    <main>
      <h1>Fibonacci Calculator</h1>
      <label for="position">Position</label>
      <input type="text" placeholder="Enter a position" id="position" />
      <button type="button" onclick="showResult(position)">
        Calculate Fibonacci
      </button>
      <p id="answer"></p>
    </main>

    <script>
      async function showResult(num) {
        const answerEl = document.getElementById('answer');
        const position = Number(document.getElementById('position').value);
        const button = document.querySelector('button');
        button.disabled = true;
        button.innerText = 'Loading...';
         if (window.Worker) {
          const worker = new Worker('fibonacci.js');
          worker.postMessage(position);
          worker.addEventListener('message', ({ data }) => {
            answerEl.innerText = `The Fibonacci number at position ${position} is ${data}`;
            button.disabled = false;
            button.innerText = 'Calculate Fibonacci';
            worker.terminate();
          });
        } 
      }
    </script>
  </body>
</html>

Let's see what is happening.

First we check if Web Workers are available or not (Although Web Workers are supported by almost all browsers, its better to check).

window.Worker // checks if Worker is available

A worker is created using the Worker constructor function.

const worker = new Worker('fibonacci.js')

A worker is a script file that needs to be created and the path to the file needs to be provided while creating the worker instance.

The Web Worker API provides some events and methods to transfer data between the worker and the main script.

postMessage is a method that can be called on the worker to transfer data from the main script to the worker.
message is an event that can be attached to the worker. This provides the data from the worker. The data is stored in a 'data' object which I have destructured to get the data from the worker.

Now let's see the code in the worker

function calculateFibonacci(num) {
  if (num < 2) {
    return num;
  } else {
     return calculateFibonacci(num - 1) + calculateFibonacci(num - 2);
  }
}

self.addEventListener('message', (e) => {
  const answer = calculateFibonacci(e.data)
  self.postMessage(answer);
});

The code is pretty self-explanatory. In the worker file, we use the message event to listen for data passed from the main thread and perform calculations here.

The result of the operation is transferred back to the main thread via the postMessage method.

We can also listen for any errors happening during this communication using the error event listener.

The above code for calculating Fibonacci can be optimized by using some dynamic programming concepts (in simple terms by storing results in a dictionary or a cache)

const cache = {};
function calculateFibonacci(num) {
  if (num < 2) {
    return num;
  } else {
    if (!cache[num]) {
      cache[num] = calculateFibonacci(num - 1) + calculateFibonacci(num - 2);
      return cache[num];
    } else {
      return cache[num];
    }
  }
}

self.addEventListener('message', (e) => {
  const answer = calculateFibonacci(e.data);
  setTimeout(() => {
    self.postMessage(answer);
  }, 3000);
});

This will drastically reduce the recursion steps and almost instantly produce the results even for higher numbers.

Conclusion

This is the basic concept of Web Workers and how they can be used to built performant apps and offer better user experience. The example I provided is very basic. However Web Workers can be used on various ocassions such as :

Doing image processing inside canvas.
Prefetching data from network to provide a faster and seamless user experience.
Doing end-to-end data encryption etc,

My idea was to cover the basics of Web Workers in this post which should be a part of a frontend developer's roadmap. There are few concepts such as Shared Workers which are almost the same as that of dedicated workers, but can be shared among script belonging to the same origin (for eg different tabs of the same origin).

The MDN Docs is a great place to know more about Web Workers and understand the ways in which they can be used.

Here is another great post on the basics of Web Workers.

Have a great one!

30 Days of Python 👨‍💻 - Day 30 - Free Python Resources

Arindam Dawn — Tue, 21 Jul 2020 06:39:54 +0000

It has been a great experience learning the Python language for the past month. Although there are areas which I have been not been able to cover, I now am familiar with the basics and can start building projects and dive into more advanced concepts.

Over the last month, I have bookmarked some free and useful Python resources which I found very interesting. I would like to conclude this challenge by compiling all the resources in this post. It will come in handy for future reference and might help anyone looking to explore the world of Python.

Free Python Resources

That's all for today! Hope you find these resources handy.

Have a great one!

30 Days of Python 👨‍💻 - Day 29 - Automation Testing

Arindam Dawn — Sun, 19 Jul 2020 17:48:14 +0000

Today I explored end-to-end Browser Automation Tests using Python by creating a basic automation testing project.

When it comes to implementing browser automation, Selenium is one of the most popular and widely used libraries. The selenium python library makes it very easy to use it with Python and run scripts to perform automated tests on websites.

Automation tests are essential for testing web applications in a real-world scenario just as how a user would normally use the application. It is also used to perform cross-browser tests to ensure the application works on all target browsers.

I created a basic project to understand and implement the basics of browser automation testing with Selenium.

Creating a project

I created a new directory python-selenium and added a virtual environment to the project.

python -m venv venv

Installing Selenium

Next step is to install the Selenium package which can be installed in the project environment using the command

pip install selenium

Installing the drivers

Selenium uses drivers to interface with the browser. So browser-specific drivers need to be installed so that Selenium is able to create the browser instance for running the automation tests.

I downloaded the Firefox driver for this particular project. The driver needs to be the executable path.

(In windows 10 place the driver file in the path where the python application is installed or any other location whose path is there in the environment variables)

Creating a sample automation script

I created a python script file automation.py in the root project directory.

Here is a simple code for opening a webpage using Selenium

automation.py

from selenium import web-driver

browser = webdriver.Firefox()
browser.get('https://tabandspace.com')

Selenium Cheat-sheet

Here is a cheat-sheet for the commonly used selenium methods

# import selenium
from selenium import web-driver 

#create a chrome driver
chromedriver = "C:/tests/chromedriver.exe"
driver = webdriver.Chrome(executable_path = chromedriver)

#create a firefox driver
geckodriver = "C:/tests/geckodriver.exe"
driver = webdriver.Firefox(executable_path = geckodriver)

#create an IE driver
iedriver = "C:/tests/IEDriverServer.exe"
driver = webdriver.Firefox(executable_path = iedriver)

#open a website
the_url = "https://dev.to"
driver.get(the_url)

#finding an element by id
the_id = 'register'
element = driver.find_element_by_id(the_id)

#finding an element by name
the_name = 'register'
element = driver.find_element_by_id(the_name)

#findding an element by class name
the_class_name = 'nav-link'
element = driver.find_element_by_class_name(the_class_name)

#finding an element by tag name
the_tag_name = 'a'
element = driver.find_element_by_tag_name(the_tag_name)

#finding an element by link text ( anchor elements)
the_link_text = 'Sign Up'
element = driver.find_element_by_link_text(the_link_text)

#finding an element by partial link text (anchor elements)
the_partial_link_text = 'Sign'
element = driver.find_element_by_partial_link_text(the_partial_link_text)

#finding element using css selectors
the_css_selector = 'a[href="/sign-up"]'
element = driver.find_element_by_css_selector(the_css_selector)

#finding element using x-path
the_xpath = '//a[@href = "/sign-up"]'
element = driver.find_element_by_xpath(the_xpath)

#clicking an element
the_id = 'register'
element = driver.find_element_by_id(the_id)
element.click()

#type inside an element
the_id = 'email'
the_email = 'klaus@werner.de'
element = driver.find_element_by_id(the_id)
element.send_keys(the_email)

#select an option from select elements
the_id = 'country'
element = driver.find_element_by_id(the_id)
select_element = Select(element)
select_element.select_by_visible_text('Canada')

#taking screenshots
the_path = 'C:/tests/screenshots/1.png'
driver.save_screenshot(the_path)

#uploading a file
the_file_path = 'C:/tests/files/example.pdf'
the_id = 'upload_button'
element = driver.find_element_by_id(the_id)
element.send_keys(the_file_path)

#execute javascript
js_code = 'document.getElementById("pop-up").remove()'
driver = execute_script(js_code)

#switch to iframe
the_iframe_id = 'payment_section'
the_element_id = 'card_number'
the_iframe = driver.find_element_by_id(the_iframe_id)
driver.switch_to.frame(the_iframe)
element = driver.find_element_by_id(the_element_id)
element.send_keys('41111111111111')
driver.switch_to.default_content()

#switch to next tab
global nextTab
global currentTab
nextTab = currentTab + 1
driver.switch_to_window(driver.window_handles[nextTab])
currentTab = currentTab + 1

#swtich to previous tab
global previousTab
global currentTab
previousTab = currentTab - 1
driver.switch_to_window(driver.window_handles[previousTab])
currentTab = currentTab - 1

#close tab
driver.close()

#close alert
driver.switch_to.alert.accept()

#refresh
driver.refresh()

#hover
the_id = "register"
the_element = driver.find_element_by_id(the_id)
hover = ActionChains(driver).move_to_element(the_element)
hover.perform()

#right click
the_id = "register"
the_element = driver.find_element_by_id(the_id)
right_click = ActionChains(driver).context_click(the_element)
right_click.perform()

#set window size
driver.set_window_size(1600, 1200)

#press key
the_id = 'register'
element = driver.find_element_by_id(the_id)
element.send_keys(Keys.RETURN)

#configure element load timeout
from selenium.webdriver.support.ui import WebDriverWait

the_id = 'register'
WebDriverWait(driver,10).until(EC.presence_of_element_located((By.ID, the_id)))

#emulate mobile device
google_pixel_3_xl_user_agent = 'Mozilla/5.0 (Linux; Android 9.0; Pixel 3 XL Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.98 Mobile Safari/537.36'
pixel_3_xl_emulation = {
   "deviceMetrics": {
      "width": 411, 
      "height": 731, 
      "pixelRatio": 3
   },
   "userAgent": google_pixel_3_xl_user_agent
}
options = webdriver.ChromeOptions()
options.add_experimental_option("mobileEmulation", pixel_3_xl_emulation)
driver = webdriver.Chrome(
   executable_path = chromedriver, 
   chrome_options = options)

Configure automation testing in a build pipeline

I found a great article which describes the steps to implement web automation scripts in a build pipeline such as Github Workflow. Selenium Base https://github.com/seleniumbase/SeleniumBase is a great wrapper on top of selenium that makes it easy to install drivers and add the automation script to a build pipeline.

https://dev.to/seleniumbase/running-browser-tests-from-github-workflows-with-seleniumbase-1oic

That's all for today. Tomorrow is the last day of this challenge where I shall be sharing the various Python resources I have collected during this month while learning Python.

Have a great one!

30 Days of Python 👨‍💻 - Day 28- ML & Data Science II

Arindam Dawn — Sat, 18 Jul 2020 15:16:14 +0000

Today I explored the Scikit-Learn library and created a notebook project to go over some of the basics and try creating a machine learning model. Scikit-Learn is a vast library and it takes a lot of practice and exploration to get a grasp of it. I followed some tutorials and articles to try building a simple classifier model just to figure out how it works. It looked a bit intimidating to me but I decided to create a basic workflow in a Jupyter Notebook so that I can use it as a reference when I decide to dive deep into the ML and Data Science Domain.

Scikit-Learn is a popular Python library for Machine Learning. Scikit-Learn can process data provided to it and create machine learning models to learn patterns within the data and makes predictions using its tools.

Why Scikit-learn?

Built on top of numpy and matplotlib libraries
Has tons of built-in machine learning models
Lot of methods to evaluate machine learning models
Easy to understand and well-designed API

Usually, Machine Learning can be a bit overwhelming as it involves complex algorithms and statistics to analyze data. Scikit-learn abstracts this complexity and makes it easy to build models and train them without having to know much about mathematics and statistics.

Here is the Notebook I created today. The link to the Github repository is https://github.com/arindamdawn/jupyter_notebooks

Basics of the scikit-learn library

This notebook covers some of the basics of the amazing scikit-learn python library. Some of the important use cases of the library have been listed in this notebook which can be used as a cheat-sheet for reference.

Some of the topics covered are:

Getting the data ready
Selecting the appropriate algorithm/estimator for the specific problem
Fit the model/algorithm to use it to make predictions on the data
Evaluting a model
Improve a model
Saving a loading a trained model

Getting the data ready

The data used for this project will be the heart disease data set available from https://www.kaggle.com/ronitf/heart-disease-uci

import pandas as pd
import numpy as np
heart_disease = pd.read_csv('data/heart.csv')
heart_disease.head()

	age	sex	cp	trestbps	chol	fbs	restecg	thalach	exang	oldpeak	slope	thal	target
0	63	1	3	145	233	1	0	150	0	2.3	0	1	1
1	37	1	2	130	250	0	1	187	0	3.5	0	2	1
2	41	0	1	130	204	0	0	172	0	1.4	2	2	1
3	56	1	1	120	236	0	1	178	0	0.8	2	2	1
4	57	0	0	120	354	0	1	163	1	0.6	2	2	1

The aim is to predict based on the above data whether a patient has a heart disease or not. The target column determines the result and the other columns are called the features

# Create Features Matrix (X)
X = heart_disease.drop('target', axis=1)

# Create Labels (Y)
y = heart_disease['target']

Choose the appropriate model/estimator for the problem

For this problem, we will be using the RandomForestClassifier model form sklearn which is a classification machine learning model.

from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier()
clf.get_params() # lists the hyperparameters

{'bootstrap': True,
 'ccp_alpha': 0.0,
 'class_weight': None,
 'criterion': 'gini',
 'max_depth': None,
 'max_features': 'auto',
 'max_leaf_nodes': None,
 'max_samples': None,
 'min_impurity_decrease': 0.0,
 'min_impurity_split': None,
 'min_samples_leaf': 1,
 'min_samples_split': 2,
 'min_weight_fraction_leaf': 0.0,
 'n_estimators': 100,
 'n_jobs': None,
 'oob_score': False,
 'random_state': None,
 'verbose': 0,
 'warm_start': False}

Fit the model to the training data

In this step the model is split into training and testing data

# fit the model to data
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.2) 
# Means 20% of the data will be used as testing data

clf.fit(X_train, y_train);

# make prediction
y_label = clf.predict(np.array([0,2,3,4]))

y_preds = clf.predict(X_test)
y_preds

array([1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0,
       1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0,
       1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1], dtype=int64)

y_test.head()

72     1
116    1
107    1
262    0
162    1
Name: target, dtype: int64

Evaluate the model

In this step the model in evaluated on the training data and test data

clf.score(X_train, y_train)

1.0

clf.score(X_test, y_test)

0.7704918032786885

from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

print(classification_report(y_test, y_preds))

              precision    recall  f1-score   support

           0       0.77      0.71      0.74        28
           1       0.77      0.82      0.79        33

    accuracy                           0.77        61
   macro avg       0.77      0.77      0.77        61
weighted avg       0.77      0.77      0.77        61

print(confusion_matrix(y_test, y_preds))

[[20  8]
 [ 6 27]]

print(accuracy_score(y_test, y_preds))

0.7704918032786885

Improve the model

This step involves improving the model to get more accurate results

# Try different amount of n_estimators
np.random.seed(42)
for i in range(1, 100, 10):
    print(f'Trying model with {i} estimators')
    clf = RandomForestClassifier(n_estimators=i).fit(X_train, y_train)
    print(f'Model accuracy on test set: {clf.score(X_test, y_test) * 100:.2f}%')
    print('')

Trying model with 1 estimators
Model accuracy on test set: 72.13%

Trying model with 11 estimators
Model accuracy on test set: 83.61%

Trying model with 21 estimators
Model accuracy on test set: 78.69%

Trying model with 31 estimators
Model accuracy on test set: 78.69%

Trying model with 41 estimators
Model accuracy on test set: 75.41%

Trying model with 51 estimators
Model accuracy on test set: 75.41%

Trying model with 61 estimators
Model accuracy on test set: 75.41%

Trying model with 71 estimators
Model accuracy on test set: 73.77%

Trying model with 81 estimators
Model accuracy on test set: 73.77%

Trying model with 91 estimators
Model accuracy on test set: 75.41%

Save the model and load it

Will be using the pickle library from Python to save the model

import pickle

pickle.dump(clf, open('random_forest_model_1.pkl', 'wb'))

#load the model
loaded_model = pickle.load(open('random_forest_model_1.pkl','rb'))
loaded_model.score(X_test, y_test)

0.7540983606557377

That's all for today. Since Machine Learning and Data Science is an ocean in itself, I decided to look into it in more details as after being more adept with its tools and concepts, share my experience as blog posts and projects. For the remaining two parts of this challenge, I would like to explore domains such as Automation Testing with Python using Selenium and create another post on a compilation of Python resources.

Have a great one!

30 Days of Python 👨‍💻 - Day 27 - ML & Data Science I

Arindam Dawn — Sat, 18 Jul 2020 12:11:16 +0000

It is time to dig into some real Machine Learning and Data Science coding stuffs. Today I mainly focused on getting started with the Jupyter Notebook workflow and creating a basic project to understand how it works. Finally search for some data set and then follow the basic principles of Machine Learning on it to generate useful information from it. I will also share the notebook I created. The great thing about Jupyter Notebooks it can be literally organized like a blog post or article along with the interactive code, data and other information.

Working with Jupyter Notebooks

I would like to provide a reference to some cool resources to understand the Jupyter Notebook interface, installation guide and its workflow overview.

Jupyter Notebook Tutorial Video -
Installation guideline - (It is recommended to install it using Anaconda toolkit as it comes with a lot of useful tools.)
Documentation -

Since I am a windows user, I would like to provide a quick to tip:

In windows, open Anaconda Prompt from the start menu, navigate to the directory where you want to create jupyter projects, then run the command jupyter notebook. It will open up the notebook in the browser.

As per the basic steps of Machine Learning and Data Science, we shall be creating the project and create a readable notebook that documents the entire process which can then be shared with anyone.

Basics of Data Science and ML using Netflix Shows project

The basic steps of ML and Data Science are:

Importing data from some source
Cleaning up the data to remove any irrelevant data if needed
Splitting up data into Training Set and Test Set.
Creating a model or an algorithm or a function
Checking the output
Improve and repeat the above steps

We shall explore the first two steps in this basic project

1. Importing Data and manipulation

The first and the most important thing for Machine Learning and Data Science is the data itself. To obtain good meaningful conclusions, we must have good data sets. This input data can be collected in a number of ways - from databases, by scraping websites, public APIs or public shared data sets.

Kaggle is a popular website among Machine Learning and Data Science enthusiasts where tons of publicly shared data sets can be found.

I decided to search for a Netflix Shows data set and found this one from Kaggle - https://www.kaggle.com/shivamb/netflix-shows. It contains the data in a CSV format which will be used for this project. After downloading the file, it can be placed in the root directory of the project. I have named it netflix_titles.csv

Since this data is in a kind of tabular format meaning it is arranged in rows and columns, pandas is a great open-source library to process this kind of data and analyze it. It comes along with the Anaconda toolkit, so it can be used directly in the notebook.

import pandas as pd
data_frame = pd.read_csv('netflix_titles.csv')
data_frame.head(10) # show first 10 results
# prints the data frame in as a table

	show_id	type	title	director	cast	country	date_added	release_year	rating	duration	listed_in	description
0	81145628	Movie	Norm of the North: King Sized Adventure	Richard Finn, Tim Maltby	Alan Marriott, Andrew Toth, Brian Dobson, Cole...	United States, India, South Korea, China	September 9, 2019	2019	TV-PG	90 min	Children & Family Movies, Comedies	Before planning an awesome wedding for his gra...
1	80117401	Movie	Jandino: Whatever it Takes	NaN	Jandino Asporaat	United Kingdom	September 9, 2016	2016	TV-MA	94 min	Stand-Up Comedy	Jandino Asporaat riffs on the challenges of ra...
2	70234439	TV Show	Transformers Prime	NaN	Peter Cullen, Sumalee Montano, Frank Welker, J...	United States	September 8, 2018	2013	TV-Y7-FV	1 Season	Kids' TV	With the help of three human allies, the Autob...
3	80058654	TV Show	Transformers: Robots in Disguise	NaN	Will Friedle, Darren Criss, Constance Zimmer, ...	United States	September 8, 2018	2016	TV-Y7	1 Season	Kids' TV	When a prison ship crash unleashes hundreds of...
4	80125979	Movie	#realityhigh	Fernando Lebrija	Nesta Cooper, Kate Walsh, John Michael Higgins...	United States	September 8, 2017	2017	TV-14	99 min	Comedies	When nerdy high schooler Dani finally attracts...
5	80163890	TV Show	Apaches	NaN	Alberto Ammann, Eloy Azorín, Verónica Echegui,...	Spain	September 8, 2017	2016	TV-MA	1 Season	Crime TV Shows, International TV Shows, Spanis...	A young journalist is forced into a life of cr...
6	70304989	Movie	Automata	Gabe Ibáñez	Antonio Banderas, Dylan McDermott, Melanie Gri...	Bulgaria, United States, Spain, Canada	September 8, 2017	2014	R	110 min	International Movies, Sci-Fi & Fantasy, Thrillers	In a dystopian future, an insurance adjuster f...
7	80164077	Movie	Fabrizio Copano: Solo pienso en mi	Rodrigo Toro, Francisco Schultz	Fabrizio Copano	Chile	September 8, 2017	2017	TV-MA	60 min	Stand-Up Comedy	Fabrizio Copano takes audience participation t...
8	80117902	TV Show	Fire Chasers	NaN	NaN	United States	September 8, 2017	2017	TV-MA	1 Season	Docuseries, Science & Nature TV	As California's 2016 fire season rages, brave ...
9	70304990	Movie	Good People	Henrik Ruben Genz	James Franco, Kate Hudson, Tom Wilkinson, Omar...	United States, United Kingdom, Denmark, Sweden	September 8, 2017	2014	R	90 min	Action & Adventure, Thrillers	A struggling couple can't believe their luck w...

data_frame.info()
# shows information about column data types

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6234 entries, 0 to 6233
Data columns (total 12 columns):
 #   Column        Non-Null Count  Dtype 
---  ------        --------------  ----- 
 0   show_id       6234 non-null   int64 
 1   type          6234 non-null   object
 2   title         6234 non-null   object
 3   director      4265 non-null   object
 4   cast          5664 non-null   object
 5   country       5758 non-null   object
 6   date_added    6223 non-null   object
 7   release_year  6234 non-null   int64 
 8   rating        6224 non-null   object
 9   duration      6234 non-null   object
 10  listed_in     6234 non-null   object
 11  description   6234 non-null   object
dtypes: int64(2), object(10)
memory usage: 584.6+ KB

data_frame.shape
# provides information of rows and columns as a tuple

(6234, 12)

data_frame.describe()
# shows some basic description

	show_id	release_year
count	6.234000e+03	6234.00000
mean	7.670368e+07	2013.35932
std	1.094296e+07	8.81162
min	2.477470e+05	1925.00000
25%	8.003580e+07	2013.00000
50%	8.016337e+07	2016.00000
75%	8.024489e+07	2018.00000
max	8.123573e+07	2020.00000

data_frame['title'].head() # lists a specific column data with first 5 entries (head)

0    Norm of the North: King Sized Adventure
1                 Jandino: Whatever it Takes
2                         Transformers Prime
3           Transformers: Robots in Disguise
4                               #realityhigh
Name: title, dtype: object

# Filtering Data
data_frame[data_frame['country'] == 'India'].head()

	show_id	type	title	director	cast	country	date_added	release_year	rating	duration	listed_in	description
35	81154455	Movie	Article 15	Anubhav Sinha	Ayushmann Khurrana, Nassar, Manoj Pahwa, Kumud...	India	September 6, 2019	2019	TV-MA	125 min	Dramas, International Movies, Thrillers	The grim realities of caste discrimination com...
37	81052275	Movie	Ee Nagaraniki Emaindi	Tharun Bhascker	Vishwaksen Naidu, Sushanth Reddy, Abhinav Goma...	India	September 6, 2019	2018	TV-14	133 min	Comedies, International Movies	In Goa and in desperate need of cash, four chi...
41	70303496	Movie	PK	Rajkumar Hirani	Aamir Khan, Anuskha Sharma, Sanjay Dutt, Saura...	India	September 6, 2018	2014	TV-14	146 min	Comedies, Dramas, International Movies	Aamir Khan teams with director Rajkumar Hirani...
58	81155784	Movie	Watchman	A. L. Vijay	G.V. Prakash Kumar, Samyuktha Hegde, Suman, Ra...	India	September 4, 2019	2019	TV-14	93 min	Comedies, Dramas, International Movies	Rushing to pay off a loan shark, a young man b...
99	80225885	TV Show	Bard of Blood	NaN	Emraan Hashmi, Viineet Kumar, Sobhita Dhulipal...	India	September 27, 2019	2019	TV-MA	1 Season	International TV Shows, TV Action & Adventure,...	Years after a disastrous job in Balochistan, a...

# Sorting Data
data_frame.sort_values('release_year', ascending=False).head()

	show_id	type	title	director	cast	country	date_added	release_year	rating	duration	listed_in	description
3467	81011449	TV Show	Medical Police	NaN	Erinn Hayes, Rob Huebel, Malin Akerman, Rob Co...	United States	January 10, 2020	2020	TV-MA	1 Season	Crime TV Shows, TV Action & Adventure, TV Come...	Doctors Owen Maestro and Lola Spratt leave Chi...
3249	81006825	Movie	All the Freckles in the World	Yibrán Asuad	Hánssel Casillas, Loreto Peralta, Andrea Sutto...	Mexico	January 3, 2020	2020	TV-14	90 min	Comedies, International Movies, Romantic Movies	Thirteen-year-old José Miguel is immune to 199...
3220	80997687	TV Show	Dracula	NaN	Claes Bang, Dolly Wells, John Heffernan	United Kingdom	January 4, 2020	2020	TV-14	1 Season	British TV Shows, International TV Shows, TV D...	The Count Dracula legend transforms with new t...
3427	81060049	Movie	Leslie Jones: Time Machine	David Benioff, D.B. Weiss	Leslie Jones	United States	January 14, 2020	2020	TV-MA	66 min	Stand-Up Comedy	From trying to seduce Prince to battling sleep...
3436	80239306	TV Show	The Healing Powers of Dude	NaN	Jace Chapman, Larisa Oleynik, Tom Everett Scot...	NaN	January 13, 2020	2020	TV-G	1 Season	Kids' TV, TV Comedies, TV Dramas	When an 11-year-old boy with social anxiety di...

This is a great cheat-sheet for Data Science with Python which lists all the commonly used Pandas methods and properties along with other libraries for data science as well.

2. Cleaning Data

The next step is cleaning up data and removing any kinds of information that is not required for the analysis.
Let's consider an example use case where we want to find which Netflix comedy movies and shows that are suitable for all ages(TV-G rating)

# Let's select the relevant columns for analysis
df_shows = pd.DataFrame(data_frame, columns=['title','rating', 'listed_in'])
# filter comedy shows
df_comedy_shows = df_shows[df_shows['listed_in'].str.contains('Comed')]
df_comedy_shows.head()

	title	rating	listed_in
0	Norm of the North: King Sized Adventure	TV-PG	Children & Family Movies, Comedies
1	Jandino: Whatever it Takes	TV-MA	Stand-Up Comedy
4	#realityhigh	TV-14	Comedies
7	Fabrizio Copano: Solo pienso en mi	TV-MA	Stand-Up Comedy
10	Joaquín Reyes: Una y no más	TV-MA	Stand-Up Comedy

# filter shows for all ages
df_all_ages = df_comedy_shows[df_comedy_shows['rating']=='TV-G']
df_all_ages.head()

	title	rating	listed_in
1034	Luccas Neto in: Summer Camp	TV-G	Children & Family Movies, Comedies
1043	A Holiday Engagement	TV-G	Children & Family Movies, Comedies, Romantic M...
1205	A Fairly Odd Summer	TV-G	Children & Family Movies, Comedies
1206	Bella and the Bulldogs	TV-G	Kids' TV, TV Comedies
1211	Jinxed	TV-G	Children & Family Movies, Comedies

The Github repository for this notebook can be found here

Resources:

That's all for today's post. Tomorrow will continue exploring more on the other steps of machine learning and data science and perform a visual analysis of data by building charts and diagrams along with creating machine learning models.

Have a great one!

30 Days of Python 👨‍💻 - Day 26 - Machine Learning Basics

Arindam Dawn — Thu, 16 Jul 2020 18:44:42 +0000

My Python 30 days of learning challenge series is in the last phase and thought out the challenge, I have explored the fundamental concepts of the Python language along with experimenting with some intermediate and advanced concepts as well. I wanted to use this challenge as an opportunity to explore another domain which I have heard a lot about but never tried anything out myself - Machine Learning and Data Science. So for the final days of this challenge, I would like to understand the basic concepts of Machine Learning and Data Science using Python and try building some projects and share whatever I learn. This would be a good starting point to explore deep into this domain in future.

What is Machine Learning?

Machine Learning is the field of computer science by which computers are able to generate algorithms and eventually improve their ability to solve problems by analyzing data provided to them. It is a sub-set or a segment of Artificial Intelligence that allows computers the ability to perform tasks without human intervention.

Machine learning (ML) is the study of computer algorithms that improve automatically through experience. It is seen as a subset of artificial intelligence. Machine learning algorithms build a mathematical model based on sample data, known as "training data", in order to make predictions or decisions without being explicitly programmed to do so. - Wikipedia

Why Machine Learning?

There is a famous quote by Phil Knight - The founder of Nike

“Don't tell people how to do things, tell them what to do and let them surprise you with their results.”

This quote came to my mind while studying the reasons for using machine learning.

If we replace 'people' with 'computers' in the above quote it becomes

“Don't tell computers how to do things, tell them what to do and let them surprise you with their results.”

I think that pretty much summarizes why machine learning is useful and so relevant.

Here are some articles in case you are interested in knowing more

Types of Machine Learning

Machine Learning is all about predicting results based on incoming data. Machine Learning can be broadly classified into these following types:

Supervised - Here the input data set is sort of categorized.
- Classified - The provided input data has some kind of labels to help the machine segment them based on their classifications. Such a provided data set of apples and oranges would segment apples and oranges.
- Regression - Performing tasks repeatedly to obtain results such as determining stock prices
Unsupervised - When the input data is not classified by any labels, unsupervised algorithms are used
- Clustering - Clustering or grouping data
- Association Rule Learning
Reinforcement - It is often also referred to as Skill acquisition or real-time learning. In this, the machine uses trial and error to determine the best outcome possibilities. For example, a computer can teach itself to play a game million times to find out to obtain the highest score.

Basic Steps of Machine Learning

Machine Learning in general, involves these main steps:

Importing data from some source
Cleaning up the data to remove any irrelevant data if needed
Splitting up data into Training Set and Test Set.
Creating a model or an algorithm or a function
Checking the output
Improve and repeat the above steps

Python for Machine Learning

Machine Learning or in-fact A.I is language agnostic. Python however, is one of the popular languages that is being used by Machine Learning experts, data scientists and large companies due its vast community and plethora of community-built tools made available for anyone to start exploring the possibilities of this domain.

Python basically has

A rich library ecosystem
A low entry barrier - Easy to pick up for someone with no prior experience in programming
It's very flexible
Is Platform agnostic
Easy to read
Huge Community

Machine Learning Tools

The tools that I would be exploring to go through the basics of Machine Learning and Data Science are:

Jupyter Notebooks - It comes as a part of the Anaconda toolkit
Numpy A package for scientific computing with Python
Pandas- A library for data analysis with Python
Mathplotlib - Library for creating visualizations with Python

There are several other tools but I would mostly go over the basics of these over the last few days of the challenge. I will be exploring other tools such as Tensorflow in future while exploring more on Machine Learning.

Resources

I decided to bookmark a few handpicked resources related to Machine Learning and Data Science which would serve as a good reference in future.

Here is an inspiring documentary which I found deeply engrossing

I am quite excited about learning this new domain using Python. I will be sharing the projects that I develop in the upcoming days and hope to learn new concepts in the journey.

Have a great one!

30 Days of Python 👨‍💻 - Day 25 - Web Development Extras

Arindam Dawn — Wed, 15 Jul 2020 18:33:12 +0000

Starting from the basic skeleton of the Flask app, today I implemented some basic features to complete our simple minimalist blog application. First using flask's template inheritance pattern, I worked on re-using common template code and then added two dummy posts to render them dynamically. Finally, generating a requirements.txt file to store all the package dependencies in a single file which can then be used to download all the packages using a single command.

Template Inheritance

Currently, the index.html, about.html templates contain duplicate HTML code. Since Flask uses Jinja as the templating engine, we can use the powerful template inheritance concept to create a base template file which would contain all the common HTML code such as the HTML skeleton, navbar, footer etc. Other templates can then extend this base template hence making our template code more reusable. I created a base template file layout.html to contain the common template structure as follows.

layout.html

{% extends "layout.html" %}
{% block title %}Index{% endblock %}
{% block head %}
  {{ super() }}
  <style type="text/css">
    .important { color: #336699; }
  </style>
{% endblock %}
{% block content %}
  <h1>About</h1>
  <p class="important">
    Hi. I am Arindam. I love building User Interfaces. I am currently learning
    Python and created this simple blog using the Flask web development
    framework.
{% endblock %}

A template re-usable block can be created and the code for that block needs to be placed within {% block block_name %} {% endblock %} The {% block %} informs the template engine that a child template may override these portions of the template.

The other files can then inherit this base template like this:

about.html

{% extends "layout.html" %}
{% block title %}Index{% endblock %}
{% block head %}
  {{ super() }}
  <style type="text/css">
    .important { color: #336699; }
  </style>
{% endblock %}
{% block content %}
  <h1>About</h1>
  <p class="important">
    Hi. I am Arindam. I love building User Interfaces. I am currently learning
    Python and created this simple blog using the Flask web development
    framework.
{% endblock %}

Here the super() is used to render the contents of a block defined in the parent template.

More on templating using Flask can be found here https://flask.palletsprojects.com/en/1.1.x/patterns/templateinheritance/

Creating a page not found route

In Flask we can handle various exceptions that can happen while loading the contents from the server such showing a custom 404 page when a route is not found, showing a custom page when there is a 500 internal server error etc. I created a simple not_found.html file which would be rendered when a route is not found.

not_found.html

{% extends 'layout.html' %} 
{% block title %}Page Not Found{% endblock %}
{% block content %}
  <h1>Page Not Found</h1>
  <p class="important">
    Sorry, this page does not exist!
{% endblock %}

server.py

@app.errorhandler(404)
def page_not_found(error):
    return render_template('not_found.html'), 404

More on Flask error handling can be found here https://flask.palletsprojects.com/en/1.1.x/patterns/errorpages/

Adding Sample Posts

For simplicity, I created a posts folder inside the templates directory and created two dummy post files first-post.html and second-post.html. Then I created a [posts.py](http://posts.py) file to read all the post files and store the names of the files in a list which can be rendered in the home page.

posts.py

import os

def get_all_post_names():
    try:
        post_files = os.listdir('templates/posts')
        post_names = list(map(lambda x: x.split('.')[0], post_files))
        return post_names
    except:
        print('An error occurred while fetching posts')
        return []

{% extends "layout.html" %} 
{% block title %}Index{% endblock %} 
{% block head%}
{{ super() }}
 {% endblock %} 
{% block content %}

<h1 class="title">
  Welcome to the Python Blog.
</h1>
<h2>Recent Posts</h2>
<ul class="post-list">
  {% for post in post_names %}
  <li><a href="posts/{{post}}.html">{{post}}</a></li>
  {% endfor %}
</ul>

{% endblock %}

All Python statements such are written inside {% %} blocks as per the Jinja template syntax. Dynamic values are placed with {{ }}. We should now be able to see the two posts on the home page. Now on clicking on a post, it should open the respective post page. For that, we have to create a route which handles the post pages dynamically.

@app.route('/posts/<string:post_name>')
def show_post(post_name):
    return render_template(f'posts/{post_name}.html')

This route will handle page names and render the respective post pages dynamically.

Generating a requirement.txt file

In our Python project, we can use several external packages. If we want to share this project with someone, we need to share all the project files and also mention all the packages that are required to run the project along with the correct version number. This is not quite a practical approach.

All the installed dependencies used in the project can be listed in a single requirements.txt file (as per convention) along with their version using this command.

$ pip freeze > requirements.txt

This generates the requirements.txt file for the project. This file can be uploaded to any GIT repository. When someone downloads the project, they just need to run pip install -r requirement.txt and all the project related dependencies would be installed with the exact versions automatically.

The complete project code can be found here

References

The above project is a very minute use case of creating web apps using Python. I would like to list some useful references for further exploration:

That's all for today. For the remaining part of this series, I will be briefly exploring advanced topics such as the basics of Machine Learning and Data Science with Python.

Have a great one!

30 Days of Python 👨‍💻 - Day 24 - Web Development Basics

Arindam Dawn — Tue, 14 Jul 2020 19:09:04 +0000

Today I explored how to develop websites using Python. There are broadly two aspects of web development - The client-side (Frontend) and the server-side (Backend). Whenever we enter the URL of any website in the browser, the browser does a request to the server which acknowledges the request and provides the contents of the website (HTML, CSS, JS, images etc) which is then rendered in the browser to display the website. Using Python, we can create a server that can acknowledge the requests made to it and server appropriate content to the requester. The server can be written in any language such as Ruby, JavaScript, Java etc. However using our existing knowledge of Python, we can easily build full-fledged websites.

Python has an in-built module http.server https://docs.python.org/3/library/http.server.html which enables us to create a simple server quite easily. However, as per the documentation of the module, it is recommended not to use it in production as it does not have a robust security check in-place.

There are many web frameworks available in Python that provides a comprehensive set of features required to develop web applications. Some of the highly popular ones are :

Django - Probably the most famous Python web development framework and provides the widest array of features necessary for building web apps. More can be found in their documentation https://docs.djangoproject.com/en/3.0/
Flask - It is listed as a micro-framework. It is comparatively lean and small but is highly extensible and provides a lot of functionalities necessary for building web apps.
Pyramid is another light-weight web framework for Python.

Here is an interesting article to know more about web frameworks in Python.

Creating a Web Application using Flask

Flask is a simple, easy to use framework so I opted to use it to build my first web application project using Python. The documentation is quite easy to understand and provides all the information required to build our own web application project.

I decided to build a simple blog application. In today's post, we will just do the basic setup and build the rest of it tomorrow.

I named the project python-blog. The next step is to create a virtual environment inside the project. Python 3 comes with a built-in venv command to create a virtual environment.

$ python -m venv venv

This creates a venv folder in the project with all the virtual environment files. Next step is to activate the environment.

$ . venv/Scripts/activate

Based on the terminal being used, the command will differ. https://docs.python.org/3/tutorial/venv.html This document a detailed guide to virtual environments and activating it.

After activating the virtual environment, the flask library can be installed. This will now install the package only within the project's virtual environment.

$ pip install flask

Now we can test flask by building a simple server. I created a Python server.py to contain the server code.

server.py

# import Flask class from flask  package
from flask import Flask
# create an instance of the Flask class by providing the application module as a parameter
app = Flask(__name__) 

@app.route('/')
def my_first_app():
    return 'This is my first Python Web Application!'

Flask uses decorator functions to handle different routes of the application. In this, we have handled the default home route of the application.

Flask provides some environment variables. To run the app, the FLASK_APP environment variable needs to be assigned the server file which is server.py. Then the app is run using flask run

$ export FLASK_APP=hello.py
$ flask run
 * Running on http://127.0.0.1:5000/

On opening the 'localhost' URL http://127.0.0.1:5000 we should see the content of the app.

To ensure that the server watches for changes made in the code, the Flask debug mode needs to be enabled like this

$ export FLASK_ENV=development
$ flask run

Now the server does not need to be restarted after any changes made to the server.py file.

Serving files using Flask

To build the actual startup website, we need to serve website files such as HTML, CSS and maybe some JS. Flask has some conventions for serving files.

All template files (HTML) need to kept inside a templates folder and all static files (CSS, JS, assets) need to be placed inside a static folder. Flask internally uses the Jinja https://jinja.palletsprojects.com/en/2.11.x/ templating engine to render the template files. I will be providing the link to the project's Github repo to check the file structure.

I created two template files, index.html, about.html and a CSS file styles.css to manage the global styles. To serve template files, Flask provides a function render_template.

server.py

# import Flask class from flask  package
from flask import Flask, render_template
# create an instance of Flask class by providing the application module as a parameter
app = Flask(__name__) 

@app.route('/')
def home():
    return render_template('index.html')

@app.route('/about')
def about():
    return render_template('about.html')

We can pass dynamic variables in the URL as <variable> and then render them in the template dynamically using Variable Rules. Since we will be building a simple blog, let's create a post route which can accept post id as parameters and dynamically render different content. Optionally a converter type can be passed to the variable as <converter:variable>. These are the following types for converter types available

server.py

@app.route('/post/<int:post_id>')
def show_post(post_id):
    # show the post with the given id, the id is an integer
    return 'Post %d' % post_id

Now on entering any numeric value after the URL /post, it should dynamically show the entered post id. Since we have added an int converter, passing any value apart from a number, will result in a page not found.

References

I just mentioned very briefly about the basics of Web Development with Python. Here are some great reference articles to know more and explore in deep.

The project's code can be found in this repo https://github.com/arindamdawn/python-blog/tree/master

That's all for today. Tomorrow, I will continue adding more features to this blog application while exploring more of Flask's concepts. I have used Flask to create a web app. Feel free to explore Django by consulting the documentation and try building simple apps with it as well.

Have a great one!

30 Days of Python 👨‍💻 - Day 23 - Web Scraping

Arindam Dawn — Mon, 13 Jul 2020 19:28:54 +0000

Web Scraping is the technique or concept of extracting data from a website by crawling it. It is mainly used to collect meaningful data from websites specially when there is no available APIs to extract information. Today I explored the basics of web scraping with Python and would like to share my experience.

Scraping is a form of scripting that allows us to automate the process of extracting large unstructured data from websites and organize it in a structured way to use it for several purposes such as gathering emails, product price, stock prices, flight data or any other relevant information. Doing such things manually takes a lot of time and effort. Python has some amazing libraries to make web scraping quite an easier and fun task to implement. I mainly explored the most basic and popular library Beautiful Soup to familiarize myself with the concept.

Good Practices

Web Scraping is extremely powerful and there is a lot of debate over its uses. Most websites have a robots.txt file which mentions which specific URLs should be crawled (scraped) and which should not be. This file is mainly an instruction for various search engine bots like google bot, yahoo bot, bing bot etc which specific pages they should crawl for search engine optimization. So all search engine crawlers are mainly web scrapers that extract data from the website to rank them as per relevant keywords. However, a website cannot literally restrict a web scraping program to not crawl its data even if it is disallowed in the robots.txt file. It is a good and ethical practice to go over a website's robots.txt file if present and extract data from only mentioned URLs to prevent any kind of data breach issues.

Scraping using Beautiful Soup

For today's session, I decided to try extracting data from the Hacker News website - An extremely popular website among the dev community. These are the rules defined in its robots.txt file

User-Agent: * 
Disallow: /x?
Disallow: /vote?
Disallow: /reply?
Disallow: /submitted?
Disallow: /submitlink?
Disallow: /threads?
Crawl-delay: 30

So we are allowed to crawl and fetch data from the news page https://news.ycombinator.com/newest which lists the latest articles from the development world. The goal is to crawl the first 5 pages and extract the articles with at-least 100 points along with their links. This can be pretty useful to automatically fetch all highly voted items and read them from the terminal itself without having to visit hacker news website and manually search for popular posts.

First two libraries need to be installed, requests for doing HTTP requests and beautifulsoup4 for scraping the website.

pip install requests pip install beautifulsoup4

hacker_news_scraper.py

import requests
from bs4 import BeautifulSoup

BASE_URL = 'https://news.ycombinator.com'
response = requests.get(BASE_URL)
# extract the text content of the web page
response_text = response.text
# parse HTML
soup = BeautifulSoup(response_text, 'html.parser')
print(soup.prettify()) # prints the html content in a readable format

The documentation for Beautiful Soup https://www.crummy.com/software/BeautifulSoup/bs4/doc/ showcases the various use cases. Using the browser's inspect element tools, the selector for the elements can be viewed and then used for extracting data. In this case, all the articles have a storylink class and their associated points have the class score. These selectors can now be used to fetch the respective data and combine them.

# extract all the links using the class selector
links_list = soup.select('.storylink')

# extract all the points using the class selector
points_list = soup.select('.score')

After looping through the links, the associated, title, link and their points can be combined as a dictionary object and then appended to a popular posts list.

It is to be noted that the enumerate function is used to get the index of each element to fetch the respective points as the points are not contained within the links container.

Only posts with a minimum of 100 points are appended to popular lists.

# loop though all links
for idx, link in enumerate(links_list):
    # fetch the title of the post
    post_title = link.get_text()
    # fetch the link of the post
    post_href = link.get('href')
    # fetch the point text using the index of the link
    # convert the point to integer
    post_points = int(points_list[idx].get_text().replace(' points', ''))
    # append to popular posts as a dictionary object if points is atleast 100
    if post_points >= 100:
        popular_posts.append(
        {'title': post_title, 'link': post_href, 'points': post_points})

There is a useful built-in Python library pprint that prints data in the console in a more readable format.

import pprint

It can then be used to view the popular lists

# loop though all links
for idx, link in enumerate(links_list):
    # fetch the title of the post
    post_title = link.get_text()
    # fetch the link of the post
    post_href = link.get('href')
    # fetch the point text using the index of the link
    # convert the point to integer
    post_points = int(points_list[idx].get_text().replace(' points', ''))
    # append to popular posts as a dictionary object if points is atleast 100
    if post_points >= 100:
        popular_posts.append(
        {'title': post_title, 'link': post_href, 'points': post_points})

pprint.pprint(popular_posts) # prints in a readable format

The above script only fetches the popular posts from the first page of Hacker News. However, as per the desired goal, we need to fetch the lists from the top five pages or probably any entered number of pages. So the script can be modified accordingly.

Here is the final script to scrape the popular lists. The code can also be found in the Github repository https://github.com/arindamdawn/python-scripting

import requests
from bs4 import BeautifulSoup
import pprint
import time

BASE_URL = 'https://news.ycombinator.com'
# response = requests.get(BASE_URL)

def get_lists_and_points(soup):
    # extract all the links using the class selector
    links_list = soup.select('.storylink')

    # extract all the points using the class selector
    points_list = soup.select('.score')

    return (links_list, points_list)

def parse_response(response):
    # extract the text content of the web page
    response_text = response.text
    # parse HTML
    soup = BeautifulSoup(response_text, 'html.parser')
    return soup

def get_paginated_data(pages):
    total_links_list = []
    total_points_list = []
    for page in range(pages):
        URL = BASE_URL + f'?p={page+1}'
        response = requests.get(URL)
        soup = parse_response(response)
        links_list, points_list = get_lists_and_points(soup)
        for link in links_list:
            total_links_list.append(link)
        for point in points_list:
            total_points_list.append(point)
        # add 30 seconds delay as per hacker news robots.txt rules
        time.sleep(30)
    return (total_links_list, total_points_list)

def generate_popular_posts(links_list, points_list):
    # create an empty popular posts list
    popular_posts = []

    # loop though all links
    for idx, link in enumerate(links_list):
        # fetch the title of the post
        post_title = link.get_text()
        # fetch the link of the post
        post_href = link.get('href')
        # fetch the point text using the index of the link
        # convert the point to integer
        # if points data is not available, assign it a default of 0
        try:
            post_points = int(
                points_list[idx].get_text().replace(' points', ''))
        except:
            points_list = 0
        # append to popular posts as a dictionary object if points is atleast 100
        if post_points >= 100:
            popular_posts.append(
                {'title': post_title, 'link': post_href, 'points': post_points})
    return popular_posts

def sort_posts_by_points(posts):
    return sorted(posts, key=lambda x: x['points'], reverse=True)

def main():
    total_links_list, total_points_list = get_paginated_data(5)
    popular_posts = generate_popular_posts(total_links_list, total_points_list)
    sorted_posts = sort_posts_by_points(popular_posts)
    # print posts sorted by highest to lowest
    pprint.pprint(sorted_posts)

if(__name__ == '__main__'):
    main()

Now using this script, we don't even need to visit Hacker News and search for popular news. We can run this script from our console and get the latest news delivered. Feel free to tweak the script as per your needs and experiment with it or try scraping data from your own favourite website.

We can possibly do a lot of things with the above data such as

Create an API to use it for an app of website
Use it for analysing trends using keywords
Create a news aggregator website and more

Popular Scraping Libraries

Beautiful Soup has its restrictions when related to scraping data from websites. It is quite simple to use but for scraping data from complex websites that are rendered at the client-side (Angular, React-based websites), the HTML markup won't be available when the website loads. To fetch data from such websites, more advanced libraries can be used. Here are some popular libraries and frameworks for Python.

lxml
Selenium
Scrapy - It is a complete framework for web scraping

References

Web Scraping is a vast field. Using Beautiful Soup, we probably just scratched the surface. There are a whole lot of possibilities in this domain which I would explore while exploring more on data analysis with Python. Hopefully, I have been able to cover the basic concepts needed for further exploration.

Tomorrow I shall be going over the concepts of Web Development with Python.

Have a great one!

30 Days of Python 👨‍💻 - Day 22 - Scripting Extras

Arindam Dawn — Sun, 12 Jul 2020 19:00:15 +0000

Today I continued exploring more on the possibilities of scripting with Python. I read some interesting articles which I will be sharing at the end of this post.

Doing HTTP Requests

One of the most common things that we often need to implement is communicating with some API to fetch data and process it. So I decided to find out ways in which we can talk to an API using Python and access the data. There are some libraries to do HTTP requests such as httplib, httplib2, urllib but the most commonly used library is requested https://requests.readthedocs.io/en/master/

It helps to do HTTP requests with quite an ease and has excellent documentation.

It can be installed using the command pip install requests

I decided to try doing something interesting while exploring how to use the requests library. I found an API to track the COVID-19 statistical data for India. This API provides the official data as per ICMR stats. I decided to use the API to create a script that can show us the daily stats for COVID-19 along with some other useful information.

Script to display daily statistics.

I created a new script file covid_tracker_india.py in the existing scripts project. The link to the GitHub repo is here https://github.com/arindamdawn/python-scripting.

covid_tracker_india.py

import requests

API_URL_ALL_DATA = 'https://api.covid19india.org/data.json'
API_URL_DISTRICT_WISE = 'https://api.covid19india.org/state_district_wise.json'

response = requests.get(API_URL_ALL_DATA)

def get_daily_stats(response):
    try:
        all_cases_data_list = response.json()['cases_time_series']
        latest_cases_data = all_cases_data_list[len(all_cases_data_list) - 1]
        formatted_data = f'''
            COVID INDIA DAILY STATS:
            AS of {latest_cases_data['date']}
            ******************************
            Total Confirmeed Cases : {latest_cases_data['totalconfirmed']}
            Total Recovered Cases : {latest_cases_data['totalrecovered']}
            Total Deaths Reported: {latest_cases_data['totaldeceased']}
            Confirmed Cases Yesterday: {latest_cases_data['dailyconfirmed']}
            Confirmed Recovered Cases Yesterday: {latest_cases_data['dailyrecovered']}
            Deaths Reported Yesterday: {latest_cases_data['dailydeceased']}
            ******************************
        '''
        print(formatted_data)
    except:
        print('An error occurred while processing data')

if __name__ == '__main__':
    get_daily_stats(response)

On running the script, it should print the data in this format

COVID INDIA DAILY STATS:
AS of 11 July
******************************
Total Confirmeed Cases : 850364
Total Recovered Cases : 536232
Total Deaths Reported: 22689
Confirmed Cases Yesterday: 27755
Confirmed Recovered Cases Yesterday: 19981
Deaths Reported Yesterday: 543
******************************

I experimented further with the API data and created another function that displays the top 5 states with the most active cases. Here is the updated script.

covid_tracker_india.py

import requests

API_URL_ALL_DATA = 'https://api.covid19india.org/data.json'
API_URL_DISTRICT_WISE = 'https://api.covid19india.org/state_district_wise.json'

response = requests.get(API_URL_ALL_DATA)

def get_daily_stats(response):
    try:
        all_cases_data_list = response.json()['cases_time_series']
        latest_cases_data = all_cases_data_list[len(all_cases_data_list) - 1]
        formatted_data = f'''
            COVID INDIA DAILY STATS:
            AS of {latest_cases_data['date']}
            ******************************
            Total Confirmeed Cases : {latest_cases_data['totalconfirmed']}
            Total Recovered Cases : {latest_cases_data['totalrecovered']}
            Total Deaths Reported: {latest_cases_data['totaldeceased']}
            Confirmed Cases Yesterday: {latest_cases_data['dailyconfirmed']}
            Confirmed Recovered Cases Yesterday: {latest_cases_data['dailyrecovered']}
            Deaths Reported Yesterday: {latest_cases_data['dailydeceased']}
            ******************************
        '''
        print(formatted_data)
    except:
        print('An error occurred while processing data')

def get_top5_states_with_active_cases(response):
    try:
        all_states_data_list = response.json()['statewise']
        all_states_data_list.sort(
            key=lambda x: int(x['active']), reverse=True)
        top5_active_states = all_states_data_list[1:6]
        print('Top 5 states with most active cases in India:')
        for index, state in enumerate(top5_active_states):
            formatted_data = f'''
            ********{index + 1}*************
            State: {state['state']}
            Active: {state['active']}
            Total Confirmed : {state['confirmed']} 
            ***************************

            '''
            print(formatted_data)
    except Exception as error:
        print(f'An error occured while processing data, {error}')

if __name__ == '__main__':
    get_daily_stats(response)
    get_top5_states_with_active_cases(response)

From my JavaScript based mental model, I was often trying to access the JSON object data using the . method which does not work in Python. Here we have to access object values using the bracket notation []. This is what I had to remember and update my mental model.

The script can be hosted in a server and can be run each day if we want to fetch the daily stats automatically.

Apart from building the simple COVID-tracker, I also explored building a basic twitter bot using the very convenient https://www.tweepy.org/ library that provides a wrapper over the twitter API to automate twitter tasks. There are tons of useful articles available on it already so I will be sharing the links to the resources instead.

Here is a list of resources for that leverages Python scripting to create useful tasks.

https://github.com/geekcomputers/Python - Awsome Python scripts (My favorite)
https://github.com/realpython/python-scripts - Github repo of some useful Python scripts
https://github.com/Logan1x/Python-Scripts/tree/master/bin - Cool Python Scripts
https://github.com/hastagAB/Awesome-Python-Scripts - Curated Python Scripting Projects

Here are some reference articles for scripting with Python.

I shall be exploring more on scripting with Python while working on projects in the upcoming days and learn about the new possibilities that can be unlocked. Scripting assists un in automating a lot of the monotonous redundant tasks by delegating them to computers so that we can focus on other important aspects of programming 🙂

Tomorrow I shall be diving into another interesting area - Web Scraping with Python and explore the basics while searching for new possibilities.

Have a great one!

	age	sex	cp	trestbps	chol	fbs	restecg	thalach	exang	oldpeak	slope	thal	target
0	63	1	3	145	233	1	0	150	0	2.3	0	1	1
1	37	1	2	130	250	0	1	187	0	3.5	0	2	1
2	41	0	1	130	204	0	0	172	0	1.4	2	2	1
3	56	1	1	120	236	0	1	178	0	0.8	2	2	1
4	57	0	0	120	354	0	1	163	1	0.6	2	2	1

	age	sex	cp	trestbps	chol	fbs	restecg	thalach	exang	oldpeak	slope	thal	target
0	63	1	3	145	233	1	0	150	0	2.3	0	1	1
1	37	1	2	130	250	0	1	187	0	3.5	0	2	1
2	41	0	1	130	204	0	0	172	0	1.4	2	2	1
3	56	1	1	120	236	0	1	178	0	0.8	2	2	1
4	57	0	0	120	354	0	1	163	1	0.6	2	2	1

	age	sex	cp	trestbps	chol	fbs	restecg	thalach	exang	oldpeak	slope	thal	target
0	63	1	3	145	233	1	0	150	0	2.3	0	1	1
1	37	1	2	130	250	0	1	187	0	3.5	0	2	1
2	41	0	1	130	204	0	0	172	0	1.4	2	2	1
3	56	1	1	120	236	0	1	178	0	0.8	2	2	1
4	57	0	0	120	354	0	1	163	1	0.6	2	2	1