Forem: Sachin

How to Authenticate AI Agents in B2B SaaS: Delegated Auth, Scoped Tokens, and Audit Trails

Sachin — Mon, 23 Mar 2026 14:36:54 +0000

Let's start with a scenario that should sound familiar.

You've shipped an AI agent inside your B2B SaaS product. It summarizes meetings, drafts content, creates notes in Notion, and manages knowledge workflows — all on behalf of your users. It's fast. It's delightful. Your customers love it.

Now ask yourself: when your agent creates a Notion page on behalf of John from XCorp — does Notion's API actually know it's John? Does it know it's XCorp? Does it know the agent is only supposed to write to specific workspaces and not read everything John has ever written?

If your answer involves a shared API key, a service account with broad permissions, or a vague "we trust the agent to behave" — this article is for you.

How Most Teams Handle Agent Auth Today (And Why It's a Liability)

Most teams building customer-facing AI agents have stitched together authentication in one of three ways. They all work in development. They all carry risk that doesn't surface until something goes wrong in production.

Pattern 1: The Hardcoded API Key

This is the fastest path from prototype to production, which is exactly why it's so common. Your agent needs to create Notion pages, read databases, manage workspaces — you grab a Notion integration token, stuff it in an environment variable, and ship it.

The token is tied to a single Notion integration. It has whatever permissions that integration has. It has no concept of which user triggered the action, and no concept of which org the request is for. It will keep working indefinitely, or until someone revokes the integration and everything breaks at once.

A hardcoded API key is like giving your delivery driver a master key to every apartment in the building because you don't want to bother with individual unit codes. There's no revocation per user, no scoping per tenant. And if someone asks "which agent action ran on behalf of John from Sales at XCorp last Tuesday?" — the answer is a spreadsheet of server logs and a best guess.

Pattern 2: The Shared Service Account

A single Notion integration with broad workspace access handles every agent action, for every user, for every org. The downstream system sees requests from one identity. You've lost all user context the moment the agent touches an external tool.

If you're in a multi-tenant environment — and if you're building B2B SaaS, you are — this means your agent is running as a single undifferentiated identity across every customer organization you have. That's not multi-tenancy. That's a liability waiting to manifest.

Pattern 3: The "Trust the Agent" Architecture

You trust the application layer to enforce access controls. The agent determines from context which user it's acting for, and you trust it won't do anything it shouldn't.

This works precisely until it doesn't. An edge case in the LLM's context handling. A subtly malformed request. A prompt injection attack that convinces the agent it's acting for a different user. There's no cryptographic guarantee of identity at the infrastructure level — authorization is enforced by application logic you wrote, and are relying on the model to respect.

The Multi-Tenancy Failure Mode

Here's what the failure looks like in practice — and it's rarely dramatic. It's a quiet data bleed.

Your AI agent is helping a content manager at TechCorp. The agent creates a Notion page and writes to a database. Somewhere in the permission model there's a misconfiguration — the agent's shared integration token has access to a Notion workspace that doesn't properly filter by organization. Because the agent doesn't carry a user-scoped token, Notion's API has no way to enforce tenant boundaries at the credential level. The agent writes data. Some of it lands in the wrong workspace.

This isn't a hypothetical — it's a class of bugs that appears in any system where authorization decisions are made at the application layer rather than the infrastructure layer. Agents make it worse because they operate autonomously, at speed, across many tenants, often without a human in the loop to catch when something looks off.

Tenant isolation isn't a nice-to-have in B2B SaaS. It's the whole point. Every enterprise customer assumes it's airtight. If your agent auth model relies on application-level trust rather than cryptographically verified, tenant-scoped credentials, there's a gap — and it will be found eventually, either by a security audit or by an incident.

What Correct Agent Auth Actually Looks Like

The Identity Problem, Stated Clearly

When a human logs into your app, authentication is straightforward. They prove who they are, you issue a session token tied to their identity, and every downstream request carries that identity as a cryptographic claim. The system knows who is asking, what they're allowed to see, and can log it against their account.

An AI agent acting on a user's behalf introduces a new actor into this chain. The agent isn't the user. It isn't your service. It's something operating in between, and classical auth models weren't designed for it.

The mental model you need: delegated authorization. The user explicitly grants the agent permission to act on their behalf. The agent receives a token that carries the user's identity and org context. That token is scoped to exactly the actions the user authorized. The downstream system doesn't trust the agent — it trusts the token, which was issued through a consent flow the user explicitly approved.

The question isn't "does the agent have permission to do this?" It's "did this specific user, in this specific org, authorize this agent to do exactly this?"

Delegated Authorization in Practice

The mechanism is OAuth — specifically, OAuth flows adapted for the agent context. The flow works like this:

A user interacts with your product and authorizes the agent to access external tools on their behalf. Your system initiates an OAuth flow that ties the resulting token to that specific user, in their specific org, with a specific set of scopes. The agent receives this token and uses it for downstream API calls — the token carries the user's identity, and the downstream system can verify it cryptographically. When the user revokes consent, the token is immediately invalidated. The agent loses access at the exact moment authorization is withdrawn.

This is fundamentally different from a service account. The agent doesn't accumulate permissions over time. It has exactly what this user authorized for this session, and nothing more.

Scoping: How Least Privilege Becomes Real

Token scoping is the mechanism that makes least privilege concrete at the agent layer. Rather than issuing a token that says "this agent can do anything this user can do," you issue a token that says "this agent can create Notion pages in approved workspaces, on behalf of this user, in this org, until revoked."

This matters because agents make mistakes. LLMs hallucinate. Prompt injection is a real attack vector. A well-scoped token acts as a hard boundary at the infrastructure layer — even if the agent logic goes sideways, the token cannot be used to take actions it was never authorized to take.

In a multi-tenant context, scoping also means every token is inherently org-scoped. There's no mechanism by which a token issued on behalf of John at XCorp can write into TechCorp's Notion workspace. The tenant boundary is encoded in the credential, not enforced by application logic you might forget to write.

What a Token Should Actually Carry

A properly constructed agent token is typically a short-lived OAuth access token backed by a JWT. Here's what the payload should look like:

{
  "sub": "user_01HXYZ123",          // stable user identifier — not email, not name
  "org_id": "org_xcorp_456",        // tenant context, baked in at issuance
  "scope": "notion:pages:write notion:workspace:read-specific",
  "agent_id": "agent_content_mgr",  // which agent received this delegation
  "auth_event_id": "authz_789",     // reference to the consent event that produced this token
  "iat": 1714000000,
  "exp": 1714003600                 // 1 hour; vault handles refresh, not your app
}

Each field is doing specific work. sub makes every downstream action attributable to an individual — not a service account, not a role. org_id encodes the tenant boundary in the credential itself rather than relying on application logic to filter correctly. scope constrains what the agent can do at the infrastructure layer, so even a misbehaving agent can't exceed what the user approved. auth_event_id is the key to clean revocation: revoke that authorization event, and every token derived from it is immediately invalid, regardless of whether it's technically unexpired. The vault tracks this mapping; your application doesn't have to.

The short exp limits blast radius if a token leaks. The vault handles silent refresh — your application always gets a live token via get_connected_account, never needs to detect expiry or trigger a refresh cycle itself.

Revocation

This one gets underweighted in most agent auth discussions. With a hardcoded API key or a service account, revocation is a blunt instrument — you revoke the key, everything breaks, you scramble to reissue and reconfigure. With delegated tokens tied to per-user authorization events, revocation is precise: the user withdraws consent, that token is invalidated, the agent loses access for exactly that user and no one else. Nothing else in your system is disrupted.

This is the property that makes enterprise customers comfortable. They want to be able to say "if I change my mind, I can revoke access in one click." Delegated OAuth gives you that. Service accounts do not.

The Audit Trail You'll Actually Need

At some point a customer is going to ask what your agent did. Maybe it's a compliance audit. Maybe something went wrong and they're reconstructing what happened. Maybe their InfoSec team wants to review the logs before they sign the renewal.

With properly scoped, delegated tokens, every agent action is traceable to a specific authorization event. Each log entry should carry: the user who authorized, the org they belong to, the token ID used, the scope exercised, the timestamp, and whether the authorization was still valid at the time of the action.

"What did the agent do on behalf of John from Sales at XCorp last Tuesday?" becomes a query. Not an investigation.

Implementing This Pattern

Rolling delegated agent auth from scratch is doable — you're essentially building an OAuth authorization server with a token vault, per-tenant scoping logic, rotation policies, and audit hooks. Most teams building on top of this pattern reach for an existing implementation rather than owning that infrastructure, especially when it needs to be multi-region, SOC 2 compliant, and available at 99.99%.

The example below uses Scalekit's Agent Auth, which handles the OAuth flows, token vault, rotation, and audit logging described above. The pattern is identical whether you use Scalekit or build it yourself — the goal is to make the mechanics concrete.

A Python Example: Creating a Notion Page on Behalf of a User

This example walks through a minimal but complete implementation of the full flow: user consent, token issuance, scoped API call, no hardcoded keys.

Set Up Your Environment

pip install scalekit-sdk-python python-dotenv requests

import scalekit.client
import os
import requests
from dotenv import load_dotenv

load_dotenv()

scalekit_client = scalekit.client.ScalekitClient(
    client_id=os.getenv("SCALEKIT_CLIENT_ID"),
    client_secret=os.getenv("SCALEKIT_CLIENT_SECRET"),
    env_url=os.getenv("SCALEKIT_ENV_URL"),
)
actions = scalekit_client.actions

Step 1: Get the User's Authorization

Before the agent touches anything, the user needs to explicitly grant access. get_authorization_link generates a Notion OAuth consent URL. Scalekit handles the entire OAuth handshake — exchanging the auth code, storing the resulting token securely in the vault, and refreshing it automatically when it expires.

link_response = actions.get_authorization_link(
    connection_name="notion",
    identifier="Sachin"  # stable identifier for this user in your system
)

print(f"Authorization link: {link_response.link}")
input("Press Enter after completing authorization...")

The user clicks the link, sees Notion's standard OAuth consent screen, approves access, and that's it. They've defined what the agent is allowed to do. Everything from here is bounded by that decision.

Note: Make sure the notion connection is set up in your Scalekit dashboard under Agent Actions → Connections → Create Connection. Use Scalekit's built-in credentials for faster development — no need to create your own Notion OAuth app.

Step 2: Retrieve a Live, Scoped Token

response = actions.get_connected_account(
    connection_name="notion",
    identifier="Sachin"
)
connected_account = response.connected_account
tokens = connected_account.authorization_details["oauth_token"]
access_token = tokens["access_token"]

This token represents the user's delegated identity. It's scoped to what they approved on the Notion consent screen — their workspaces, their pages, nothing else. Scalekit keeps it fresh in the background; you always get a valid token without managing rotation yourself.

Step 3: Act on the User's Behalf

headers = {
    "Authorization": f"Bearer {access_token}",
    "Content-Type": "application/json",
    "Notion-Version": "2022-06-28",  # Always pin the Notion API version
}

# Find an accessible parent page to create under
search_response = requests.post(
    "https://api.notion.com/v1/search",
    headers=headers,
    json={
        "filter": {"value": "page", "property": "object"},
        "page_size": 1
    }
)
results = search_response.json().get("results", [])

if not results:
    print("❌ No accessible pages found. Make sure the user shared a page with the integration.")
else:
    parent_page_id = results[0]["id"]

    # Create a new page under the found parent
    create_response = requests.post(
        "https://api.notion.com/v1/pages",
        headers=headers,
        json={
            "parent": {"page_id": parent_page_id},
            "properties": {
                "title": {
                    "title": [{"text": {"content": "Page Created by AI Agent 🤖"}}]
                }
            },
            "children": [
                {
                    "object": "block",
                    "type": "paragraph",
                    "paragraph": {
                        "rich_text": [
                            {
                                "type": "text",
                                "text": {
                                    "content": "This page was created by an AI agent on behalf of Sachin using Scalekit Agent Auth. The agent had exactly the permissions this user approved — nothing more."
                                }
                            }
                        ]
                    }
                }
            ]
        }
    )

    new_page = create_response.json()
    print(f"\n✅ Notion page created successfully!")
    print(f"Page ID  : {new_page.get('id')}")
    print(f"Page URL : {new_page.get('url')}")
    print(f"Created  : {new_page.get('created_time')}")

Notion's backend sees this request as the authorized user — their workspaces, their permissions, scoped to exactly what they approved. The agent cannot access private pages, other users' workspaces, or anything outside the approved scope. That constraint isn't enforced by application logic — it's encoded in the credential itself.

Skipping the Raw API Calls

If you'd rather not manage Notion's API directly, Scalekit ships pre-built connectors for supported services. The same flow using Scalekit's built-in tools:

import scalekit.client
import os
from dotenv import load_dotenv

load_dotenv()

# Initialize Scalekit client using environment variables
scalekit_client = scalekit.client.ScalekitClient(
    client_id=os.getenv("SCALEKIT_CLIENT_ID"),
    client_secret=os.getenv("SCALEKIT_CLIENT_SECRET"),
    env_url=os.getenv("SCALEKIT_ENV_URL"),
)
actions = scalekit_client.actions

# Generate Notion OAuth consent link for this user
link_response = actions.get_authorization_link(
    connection_name="notion",
    identifier="Sachin"
)
print(f"Click this link to authorize Notion: {link_response.link}")
input("Press Enter after completing authorization...")

# Step 1: Use Scalekit's built-in tool to search for existing Notion pages
# No raw API call needed — Scalekit handles auth injection internally
search_response = actions.execute_tool(   # <-- Scalekit tool call
    tool_name="notion_page_search",       # <-- built-in Notion tool
    identifier="Sachin",                  # <-- user identity passed here
    tool_input={
        "page_size": 3,
        "sort_direction": "descending",
        "sort_timestamp": "last_edited_time"
    }
)

# Response data lives in search_response.data — no json.loads() needed
results = search_response.data.get("results", [])
print(f"Pages found: {len(results)}")

if not results:
    print("❌ No pages found. Please share a page on the Notion consent screen.")
else:
    parent_page_id = results[0]["id"]
    print(f"✅ Parent page found: {parent_page_id}")

    # Step 2: Use Scalekit's built-in tool to create a new Notion page
    new_page_response = actions.execute_tool(   # <-- Scalekit tool call
        tool_name="notion_page_create",         # <-- built-in Notion tool
        identifier="Sachin",
        tool_input={
            "parent_page_id": parent_page_id,   # <-- from Step 1 result
            "properties": {
                "title": {
                    "title": [{"text": {"content": "Page Created by AI Agent 🤖"}}]
                }
            },
            "child_blocks": [
                {
                    "object": "block",
                    "type": "paragraph",
                    "paragraph": {
                        "rich_text": [{"type": "text", "text": {
                            "content": "This page was created by an AI agent on behalf of Sachin using Scalekit Agent Auth. The agent had exactly the permissions this user approved — nothing more."
                        }}]
                    }
                }
            ],
            "notion_version": "2022-06-28"
        }
    )

    new_page = new_page_response.data   # <-- response data directly accessible
    print(f"\n✅ Notion page created successfully!")
    print(f"Page ID  : {new_page.get('id')}")
    print(f"Page URL : {new_page.get('url')}")
    print(f"Created  : {new_page.get('created_time')}")

execute_tool handles auth injection internally — Scalekit looks up the connected account for that identifier, retrieves the live token from the vault, and injects it into the request. The response comes back as a Python dict. No token handling, no JSON parsing, no API version management.

The raw API approach is more transparent and useful if you want to understand what's happening or work with APIs Scalekit doesn't yet cover. The built-in tools are more practical if you're moving fast and the connector exists. Use whichever fits the context.

What Actually Happens When You Run This

No magic, no hand-waving. Here's exactly what the flow looks like end to end once you run the program:

1. You run the program

Your terminal prints a Scalekit-generated authorization link and waits. The program is paused — nothing happens until you say so.

2. You open the link — Scalekit shows an authorization interface

You follow the link in your browser. Scalekit presents a clean authorization screen that tells you exactly what the agent is asking for — which app (Notion), which permissions, on whose behalf. No fine print. No vague "access to everything."

You click Allow access.

3. Notion's OAuth flow completes in the background

Scalekit handles the entire OAuth handshake with Notion — exchanges the auth code for an access token, stores it securely in the token vault, and marks the connected account for identifier="Sachin" as ACTIVE. You don't see any of this. It just works.

4. You come back to the terminal and press Enter

The program resumes, fetches the live token from the vault, and moves on to the Notion API call.

5. The agent creates a page in Notion on your behalf

You access the page URL. A new page is sitting right there — created by the agent, using your delegated identity, with exactly the permissions you approved and nothing more.

The agent acted as you. Notion knew it was you. And you stayed in control the whole time.

Conclusion

Agents are no longer just internal tools — they're acting on behalf of real users, inside real organizations, touching real data. A hardcoded API key got you to the demo. It won't get you to production.

The pattern is simple: delegated auth, scoped tokens, full audit trail. Scalekit gives you the infrastructure to do it right — without rebuilding your entire auth stack.

Your users are trusting your agent with their data. Make sure it's worthy of that trust.

BrightData's MCP: Fetching Real-time Web Data For LLMs

Sachin — Thu, 05 Jun 2025 15:57:34 +0000

In the world of artificial intelligence, especially large language models (LLMs), there's a quiet revolution happening. It's called Model Context Protocol, or simply MCP. You may have encountered the term in AI communities or tool integrations, but what exactly is MCP, and why should you care?

Let’s break it down, step by step, and show you how this simple yet powerful concept is solving one of the biggest headaches in AI, which is accessing real-time web data.

What Is MCP (Model Context Protocol)?

Imagine you’ve trained a brilliant AI that knows history, science, programming, and can even write poetry. But ask it what’s trending on Twitter or what the latest YouTube comments are on a viral video, and you’ll likely get outdated, static responses. Why? Because LLMs are not built for real-time web interaction out of the box.

That’s where MCP steps in.

MCP is an open standard that acts as a bridge between AI agents (like Claude, GPT, etc.) and real-time data sources like websites, APIs, or databases. Think of it as a universal translator that allows AI models to interact with the constantly-changing web securely and efficiently.

Instead of hardcoding access to different websites or APIs, MCP standardizes the way AI tools request data. It makes live data scraping and interaction much more streamlined and predictable — no more breaking scripts or bot detection errors.

Why Do LLMs Struggle with Real-Time Data?

Let’s address the elephant in the room.

LLMs like Claude, ChatGPT, and others are trained on massive datasets, but that training is static. This means the AI only knows what it was trained on. If you ask it for stock prices, breaking news, or fresh YouTube comments, it simply doesn’t know. Even worse, if it tries to guess, it might give you hallucinated or flat-out wrong answers.

Sure, you could plug in some APIs or attempt to write your own scraping scripts. But anyone who's tried this knows how quickly you hit roadblocks:

JavaScript rendering
CAPTCHA and anti-bot detection
Proxy rotation and management
Data formatting inconsistencies
Rate limits
And much more...

To obtain a list of recent comments or product prices, you find yourself immediately immersed in backend development.

A Real-World Example: When LLMs Fail

Let me give you a quick example.

I asked Claude to fetch the latest comments from a YouTube video. It understood the request just fine… but couldn’t get the data. That’s because it doesn’t have native access to YouTube’s dynamic content. Claude can only interpret requests, but it can’t execute scraping logic on its own.

This is a classic scenario where even the smartest LLMs fall short without real-time access tools.

Building Your Own MCP? It’s Harder Than It Looks

Technically, you can build your own MCP setup. You’d need to:

Set up a scraping infrastructure.
Manage rotating proxies.
Handle headless browser environments.
Deal with dynamic web rendering.
Write adapters to plug data into your AI agent.

Sounds doable at first, right? But trust me, you’ll soon find yourself debugging edge cases and fighting CAPTCHAs, all for a task that should feel simple. What starts as a weekend project quickly turns into a full-time scraping job.

That’s why most developers and AI researchers look for managed solutions.

Bright Data’s MCP: Your AI’s Web Gateway

This is where Bright Data’s MCP changes the game.

Bright Data, a leading data collection platform, has created a plug-and-play MCP service that gives your AI agent real-time access to the web without dealing with any of the complexity.

Here’s what it offers:

Unblockable access to websites like Amazon, YouTube, LinkedIn, Instagram, X (Twitter), Facebook, TikTok, Zillow, Reddit, and even Google.
Built-in proxy handling – no need to rotate or manage IPs.
Headless browser support – for rendering JavaScript-heavy pages.
Remote browser APIs – so your agent can interact with pages like a real user.
Plenty of ready-made scraping tools – for different websites and use cases.

Everything is handled under the hood, and all you need to do is configure the access; no scraping code required.

Let’s See It in Action

I ran the same prompt with more tasks using Bright Data’s MCP: “Fetch the latest comments from a YouTube video and provide that data in a json file”.

You can see that it worked and that too without any hassle.

Behind the scenes, Bright Data’s tools were doing all the heavy lifting: launching a browser, unlocking the page, pulling the data, formatting it, and sending it back to the AI agent.

No errors. No retries. Just results.

How to Integrate Bright Data’s MCP with Claude Desktop

Okay, now let’s go through how to actually connect Bright Data’s MCP with Claude Desktop. It’s simpler than you might think.

Step 1: Install Node.js

Make sure Node.js is installed on your system. You’ll need the npx command to run the MCP server.

Step 2: Create a Bright Data Account

Head over to Bright Data and sign up. Once your account is ready, log in to the user dashboard.

Step 3: Prepare the Claude Desktop Configuration

Open the Claude desktop app.

Click on File → Settings → Developer → Edit File.

This opens a file named claude-desktop-config.json.

Open it in your favorite text editor.

Step 4: Add MCP Configuration

Now, go to the GitHub repository provided by Bright Data and copy the MCP config code.

Paste that into the claude-desktop-config.json file.

Step 5: Insert Your Real MCP Credentials

Once you’ve pasted the configuration block into the claude-desktop-config.json file, it’s time to replace the placeholders with your actual values. This is what will connect your Claude Desktop app to Bright Data’s MCP system.

Here’s how to do it:

Head to Your Bright Data Dashboard

Log in to your Bright Data account and navigate to the Proxies & Scraping section. This is where you’ll manage the tools needed to enable real-time scraping.

Create a Scraping Zone

Click the “Add” button and choose “Web Unlocker API” as your tool. Once created, you’ll see a dashboard showing your new zone.

Copy the API Key

You’ll find your API key in the “Account Settings” section. Alternatively, Bright Data may have emailed it to you when you signed up. Either way, copy this key and paste it into the relevant "API_TOKEN" field in your config file.

Add the Web Unlocker Zone Name

Still in your dashboard, find the name of the Web Unlocker zone you just created. Copy it and paste it into the "WEB_UNLOCKER_ZONE" field in your configuration.

Enable Remote Browsing with Browser API

Want to supercharge your scraping setup with headless browser access? You can!

Just add a Browser API by clicking “Browser API” in the “Add” section.

Give it a name — something like my-browser-api.
After it's created, you’ll see a string.
Copy just the key part (not the full URL) and paste it into the "BROWSER_AUTH" field in your config.

This optional step enables your AI agent to interact with websites that require full browser emulation, which is perfect for scraping JavaScript-heavy pages or logging into websites.

Step 6: Restart Claude

Save the configuration file, quit Claude Desktop, and restart the app.

Once you open it again, you’ll see that all Bright Data MCP tools are now available to use directly within Claude. That’s it, now you’re ready to start scraping real-time web data with zero friction.

The Big Picture

The MCP isn’t just another API — it’s a protocol that aims to standardize how AI tools interact with dynamic data sources. And with platforms like Bright Data offering powerful plug-and-play integrations, you can start building real-time, context-aware agents without worrying about the plumbing.

Instead of writing your own scrapers, debugging browser sessions, or juggling proxies, you can focus on what matters: creating intelligent workflows that adapt to live information.

Whether you're working on:

Competitive intelligence
Social media monitoring
Real-time news summaries
E-commerce pricing trackers
AI research assistants

...MCP opens up a whole new world of possibilities.

Conclusion

Large Language Models are impressive — but without live data, they’re limited. MCP is the missing puzzle piece, and Bright Data’s implementation makes it incredibly easy to adopt.

If you're tired of watching your AI fumble on basic real-time tasks, give MCP a shot. It’s fast, powerful, and surprisingly easy to set up. And if you're using Claude Desktop, the whole thing takes under 10 minutes to integrate.

Your AI just got a whole lot smarter.

That’s all for now.

Keep Coding✌✌

Install Redis Locally in Windows

Sachin — Mon, 06 Jan 2025 14:55:41 +0000

Redis is an in-memory database that actually makes it the fastest among all the databases.

What does "in-memory" mean? It means that it stores the data in temporary memory or RAM, in the form of key-value pairs. This makes it perfect for serving results instantly.

In this article, you’ll learn how to install Redis locally on Windows without any hassle.

Installation Process Begins

You probably don’t know that Redis isn’t officially available for Windows yet but there is a way that you can install Redis locally on Windows and get started with it.

Redis is available for MacOS, Linux but for Windows, they say:

Redis is not officially supported on Windows. However, you can install Redis on Windows for development.

So, how to install it then? Simple, actually not simple in my opinion but I will guide you step by step installing Redis on your Windows machine.

Enable WSL on Windows

To install Redis on Windows, you need to enable WSL (Windows Subsystem for Linux). It simply means that you need to install Linux on your Windows system.

To enable WSL, you must be running Windows 10 version 2004 and higher (Build 19041 and higher) or Windows 11.

You can enable the WSL2 (WSL version 2) by default with a single command.

Open Powershell in administrator mode and run the following command.

wsl --install

This command will enable the features necessary to run WSL and install the Ubuntu distribution of Linux.

You can change or install another distribution of Linux if you want to, just use the following command.

wsl --install -d <Distribution Name>

You can replace <Distribution Name> with the actual Linux distribution you want to install on the Windows machine. But, I would recommend keeping the default settings.

After all this installation process, restart your Windows machine once to see the changes.

If your Windows machine is a bit older and WSL2 is not supported then check out this guide from Microsoft itself to manually enable WSL on Windows.

Troubleshooting Error

There will be less chance that you will get errors while setting up WSL but in case you encounter any error such as “failed to register wsl distribution” or something like that then consider changing the WSL version.

Run the following command in Powershell in administrator mode to change the version of your WSL.

wsl --set-default-version 1

This command will change the default version which is WSL2 to WSL1. This will solve most of the problems.

Now, restart your Windows machine again to see the effect.

Set Up Linux

After enabling WSL and installing Linux distribution, you need to set up user info for Linux.

Open Ubuntu or whichever distribution you installed and give it a few minutes to install. After that, it will ask you to enter a username for Linux and set a password.

After setting up the Linux distribution, the next step is to install Redis.

Install Redis on Windows

Redis can be installed on Windows using the following commands

curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list

sudo apt-get update
sudo apt-get install redis

Copy the above commands all together and paste them into the Ubuntu terminal, this will start the process of installing Redis on Windows locally.

Start Redis

Now run the following command to start the Redis server.

sudo service redis-server start

Now to check if you’re connected to Redis, run the following command.

redis-cli

Redis server started on localhost and listening on port 6379 which is its default port.

That’s all for now.

Keep Coding✌✌

💰Save Money on API Testing: Try EchoAPI Instead of Thunder Client!

Sachin — Sun, 01 Dec 2024 03:30:00 +0000

You might have heard of Thunder Client, a REST API client extension for Visual Studio Code that allows developers to test APIs. Thanks to its lightweight nature, it quickly became the first choice for developers worldwide.

Thunder Client was once completely free, but that’s no longer the case. It has now transitioned to a paid model, pushing many key features behind a paywall. With the free plan offering limited functionality, users have few options left.

Thunder Client used to be a one-stop destination for developers to perform basic API testing and debugging. However, with the shift to a paid model, developers are increasingly turning to alternatives like Postman. Among these, EchoAPI for Visual Studio Code stands out as a strong and worthy alternative.

Why EchoAPI is a Better Alternative?

While EchoAPI and Thunder Client share some common features, EchoAPI excels in the following areas:

Completely Free: EchoAPI is entirely free to use, with a commitment to remain so.
Ultra-Lightweight: Its minimalistic design ensures that it doesn’t burden your system.
No Login Required: Start testing immediately without the need for sign-ups or logins.
Scratchpad Feature: Dive straight into testing APIs without any setup.
Postman Script Compatibility: Migrating from Postman? No problem! EchoAPI supports Postman script syntax, so you don’t need to learn anything new.
Collaboration Ready: Enjoy team workflows for shared testing without extra costs.
Unlimited Requests: There’s no limit to the number of requests you can send.
Unlimited Collection Runs: Run your collections as many times as needed, without restrictions.
Local Storage: Store your data securely and access it from anywhere.

Let’s explore these features in detail.

Install EchoAPI

Getting started with EchoAPI is easy. Simply install it from the Visual Studio Code extensions marketplace:

Search for “EchoAPI”. Click on the first result and hit Install.

Once installed, the EchoAPI logo will appear in the sidebar, giving you access to the extension’s user interface (UI).

Now we can start testing our APIs. Let’s take a tour of the UI and explore EchoAPI functionalities.

Get Started with EchoAPI

EchoAPI provides demo APIs to help you explore its features. Let’s use these demo APIs for testing and debugging before moving on to testing our own.

For example, the first API in the “Samples” collection creates a new user with fields like id, username, firstName, lastName, email, password, phone, and userStatus.

Let’s tweak this data and send a request to the server.

After sending the request, we can see a successful response (HTTP status code 200).

Here, we can also test Server-sent Events (SSE) requests, which are used to send real-time updates to clients. These are often used in web applications for real-time notifications, live data feeds, or status updates.

Running Collections Without Limit

One of EchoAPI’s standout features is its unlimited collection runs. Here’s how you can run a collection:

Navigate to the Tests tab and select the collection or folder containing your requests.

In this case, we selected the “Samples” collection and then clicked on the Run All button. It brings up this interface where we can set configurations to the test.

You can configure the following options:

Iterations: Specify how many times the test should execute.
Intervals: Set time intervals between each request.
Data: Upload a .csv file containing test data.
Environment: Choose the testing environment with pre-configured environment variables.

In this case, the iterations are set to 3, and the interval between the execution of each request is set to 1000 ms (1 second).

Once configured, click Run to execute the collection. This will start executing requests with the specified configurations.

A report is generated showing the stats of the performed test and if we click on any request, we’ll see an individual report of the request.

Setting up Environment Variable

We can set up environment variables like API keys, URLs, authentication tokens, etc. Depending on the sensitivity of the data, these variables can be used in the header or body of the request.

Let’s see how we can set up an environment in EchoAPI.

Access the environment setup interface via the three dots menu in the top left corner or directly from the request interface.

Click on the “New Environment” button to set up a new environment.

We can set a name for the environment, the base URL of the server, and environment variables.

Testing APIs

Here’s a practical example using a Python FastAPI application with three routes. We’ll test these routes using EchoAPI.

from fastapi import FastAPI, HTTPException, Depends
from pydantic import BaseModel
from typing import List
from fastapi.security import OAuth2PasswordBearer
import jwt
import datetime

app = FastAPI()
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")

key = 'secretkey'

books = []
current_id = 1

class Book(BaseModel):
    title: str
    author: str
    price: float

class BookOut(Book):
    id: int

def authenticate_user(token: str = Depends(oauth2_scheme)):
    try:
        payload = jwt.decode(token, key, algorithms=["HS256"])
        return payload
    except jwt.ExpiredSignatureError:
        raise HTTPException(status_code=401, detail="Token has expired")
    except jwt.InvalidTokenError:
        raise HTTPException(status_code=401, detail="Invalid token")

@app.post("/token")
def get_token():
    expiration = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(hours=1)
    token = jwt.encode({"exp": expiration}, key, algorithm="HS256")
    return {"access_token": token, "token_type": "bearer"}

@app.post("/books", response_model=BookOut)
def create_book(book: Book, user: dict = Depends(authenticate_user)):
    global current_id
    new_book = book.model_dump()
    new_book["id"] = current_id
    books.append(new_book)
    current_id += 1
    return new_book

@app.get("/books", response_model=List[BookOut])
def list_books(user: dict = Depends(authenticate_user)):
    return books

Setting up EchoAPI for Testing

First, we can create a folder, let’s say Books, to make our requests more organized.

Next, we need to create individual requests within the “Books” folder.

We created three requests as shown in the image below.

According to the code, we need to generate an authentication token, so we need to send a POST request to http://127.0.0.1:8000/token to obtain an auth token and then we can use this token in other requests to authenticate a user.

Authentication

If we try to hit this URL (http://127.0.0.1:8000/books) to create a book without passing the auth token, we’ll get an error in the response.

We need to pass the previously generated auth token in the “Auth” section.

EchoAPI supports various authentication methods such as API Key, Basic Auth, and JWT Bearer. You can choose the type of authentication required in your project.

Debugging APIs

We can debug the APIs from the “Post-response” section. In this section, we can write custom scripts (Postman script) and set assertions to debug APIs, extract variables for dynamic testing, and simulate delays in API execution.

Let’s perform a simple debugging. We are setting an assertion for the response code equal to 200 and deliberately making a mistake in the request.

We can see detail in the response indicating that the price field is missing and it returned the response code 422, which is not equal to 200, so the assertion failed.

This is not limited to response code only, we can set assertions for different target objects.

More Features

Server sent event Requests:

Server-sent events (SSE) enable a server to push real-time updates to the client over a single HTTP connection. They are widely used for real-time notifications, live data feeds, status updates, or chat applications.

With Thunder Client, creating and testing SSE requests is locked behind a paywall. This makes it less appealing for developers who rely on free tools for basic functionality.

EchoAPI, however, provides full support for SSE requests free of charge. Here’s how EchoAPI simplifies SSE testing:

Ease of Setup: You can create and test SSE requests directly within EchoAPI’s user interface, without any advanced configuration.
Real-Time Stream Support: It allows you to see real-time data streams as they are sent from the server.
No Cost: Unlike Thunder Client, there are no charges or restrictions for using this feature.

For example, you can set up an SSE endpoint in your API, like /notifications, and use EchoAPI to connect to it. EchoAPI will display live updates sent by the server in a clean and intuitive interface.

Imports data from cURL:

cURL is a command-line tool used to send HTTP requests. Developers often use cURL commands to test APIs quickly in a terminal. However, when transitioning to a GUI-based tool like EchoAPI, manually re-entering the details of API requests can be tedious and error-prone.

Here’s how it works:

Copy cURL Command: Simply copy the cURL command from your terminal or documentation.
Paste in EchoAPI: Select the Import cURL option in EchoAPI, and paste the cURL command.
Automatic Conversion: EchoAPI automatically parses the command and converts it into a structured API request within its interface.

Import data from Postman:

Postman is one of the most popular API testing tools, often used for creating and managing collections of API requests. If you’re moving from Postman to EchoAPI, re-creating collections from scratch can be a daunting task.

Here’s how it works:

Export from Postman: In Postman, go to the collection you want to migrate, click on the options menu (three dots), and choose Export. Postman will generate a JSON file containing all the details of the collection.
Import to EchoAPI: Select the Import from Postman option in EchoAPI, and upload the Postman JSON file.
Seamless Integration: EchoAPI will parse the file and recreate the collection, including all requests, headers, parameters, and body configurations.

Team Collaboration and Data Sync

To collaborate with the team and sync data in EchoAPI, we need to sign up. After signing up for EchoAPI, we can push data to EchoAPI storage easily.

Click on the cloud icon and then select the workspace folder where the data will be stored and then select the collection to push to EchoAPI.

Now we can easily access the data from the EchoAPI desktop app or Webapp.

We can invite our team members through URL in the EchoAPI web app or desktop app. Click on the “Invite” button in the top right corner, and then click on the “Copy Link” button to copy the invitation link.

Conclusion

While Thunder Client once ruled as a lightweight API testing extension for Visual Studio Code, its shift to a paid model has left many developers searching for viable alternatives. EchoAPI emerges as a strong contender, offering a feature-rich, completely free, and ultra-lightweight solution tailored for seamless API testing and debugging.

With its no-login requirement, Postman script compatibility, limitless testing capabilities, and team collaboration features, EchoAPI sets itself apart as a powerful tool for both individual developers and teams. Its intuitive interface, support for server-sent events, and extensive customization options make it a go-to choice for API testing directly within VS Code.

If you're looking for an efficient, no-cost alternative to Thunder Client that doesn’t compromise on functionality, EchoAPI is worth exploring.

That’s all for now.

Keep Coding✌✌.

How to disable GIL (Global Interpreter Lock) in Python 3.13

Sachin — Wed, 20 Nov 2024 07:03:29 +0000

The Python organization has released the official version of Python 3.13, which includes many major changes.

One of the major changes we get is making GIL (Global Interpreter Lock) optional in Python 3.13.

In this article, you’ll see how to disable GIL in Python 3.13 to make multi-threaded programs faster.

Disable GIL in Python 3.13

First, download the official Python 3.13 on your local machine. Then proceed to install the Python executable file.

Follow the steps just as you install any Python version, you will see a step in which you are asked to choose the advanced installation options.

Now select the checkbox appearing at the end, this will install the different executable Python build with the name python3.13t or python3.13t.exe.

This will allow you to optionally disable or enable GIL in the runtime. Now proceed to install the setup.

After completing the setup for Python 3.13, if you look at the file location of Python 3.13, you will see that you also got a different Python build called python3.13t.

If you remember that “free-threaded binaries” mode was marked “experimental”, this means that you don’t get GIL disabled by default.

But how do I disable GIL? Python organization provided a different CPython build which is also called “Free-threaded CPython”.

Free-threaded CPython will help you disable GIL. Let’s see how to do it.

Let’s say, you have a Python script and you want it to run with GIL disabled. You can use the following command.

python3.13t main.py

This CPython build comes with GIL disabled by default.

Practical

I have the following Python script (main.py) that runs a multi-threaded task and calculates the time taken to complete the execution.

import sys
import sysconfig
import math
import time
import threading

def compute_factorial(n):
    return math.factorial(n)

# Multi-threaded
def multi_threaded_compute(n):
    threads = []
    # Create 5 threads
    for num in n:
        thread = threading.Thread(target=compute_factorial, args=(num,))
        threads.append(thread)
        thread.start()

    # Wait for all threads to complete
    for thread in threads:
        thread.join()

    print("Factorial Computed.")

def main():
    # Checking Version
    print(f"Python version: {sys.version}")

    # GIL Status
    status = sysconfig.get_config_var("Py_GIL_DISABLED")
    if status is None:
        print("GIL cannot be disabled")
    if status == 0:
        print("GIL is active")
    if status == 1:
        print("GIL is disabled")

    numlist = [100000, 200000, 300000, 400000, 500000]

    # Multi-threaded Execution
    start = time.time()
    multi_threaded_compute(numlist)
    end = time.time() - start
    print(f"Time taken : {end:.2f} seconds")

if __name__ == "__main__":
    main()

Now, we’ll run this script with and without free-threaded CPython.

With GIL enable (without free-threaded CPython)

python main.py

With GIL disable (with free-threaded CPython)

python3.13t main.py

That’s all for now.

Keep Coding✌✌

2 Ways to Stream Videos on the Frontend in FastAPI

Sachin — Sat, 16 Nov 2024 15:30:00 +0000

FastAPI is a fast and modern web framework known for its support for asynchronous REST API and ease of use.

In this article, we’ll see how to stream videos on frontend in FastAPI.

StreamingResponse

Stream Local Video

FastAPI provides a StreamingResponse class that is dedicated to streaming purposes. The StreamingResponse class takes a generator or iterator and streams the response.

Here’s a simple example of streaming local video to the browser.

# localvid.py
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

# Video path
vid_path = 'sample_video.mp4'

# Function to stream local video
def stream_local_video():
    with open(vid_path, 'rb') as vid_file:
        yield from vid_file

# Path to stream video
@app.get("/")
def video_stream():
    return StreamingResponse(stream_local_video(), media_type='video/mp4')

We imported the StreamingResponse class from the fastapi.responses module.

In the video_stream() path operation function, we returned the response using StreamingResponse. We passed the generator function stream_local_video() and media_type as an argument to the StreamingResponse class.

The generator function stream_local_video() reads the bytes of the video and yield from iterates over the bytes and each part iterated is then yielded.

The following command will run the server and stream the video on http://127.0.0.1:8000/.

> fastapi dev localvid.py

 ╭────────── FastAPI CLI - Development mode ───────────╮
 │                                                     │
 │  Serving at: http://127.0.0.1:8000                  │
 │                                                     │
 │  API docs: http://127.0.0.1:8000/docs               │
 │                                                     │
 │  Running in development mode, for production use:   │
 │                                                     │
 │  fastapi run                                        │
 │                                                     │
 ╰─────────────────────────────────────────────────────╯

Stream Online Video

# onlinevid.py
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import requests

app = FastAPI()

# Video URL
vid_url = 'https://cdn.pixabay.com/video/2023/07/28/173530-849610807_large.mp4'

# Function to stream online video
def stream_online_video(url):
    response = requests.get(url, stream=True)
    for portion in response.iter_content(chunk_size=1024*1024):
        yield portion

# Path to stream video
@app.get("/")
def video_stream():
    return StreamingResponse(stream_online_video(vid_url), media_type='video/mp4')

In this example, we used the requests library to fetch the content of the video from the URL and we set the stream=True (avoids reading the video at once into memory). Then we iterated and yielded the video content in chunks (1024 bytes at a time).

When we run the server, it will serve the video from the URL.

Instead of a hardcoded URL, we can specify the URL in the endpoint.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import requests

app = FastAPI()

# Function to stream online video
def stream_online_video(url):
    response = requests.get(url, stream=True)
    for portion in response.iter_content(chunk_size=1024*1024):
        yield portion

# Path to stream video
@app.get("/")
def video_stream(url):
    return StreamingResponse(stream_online_video(url), media_type='video/mp4')

In this example, we modified the path operation function video_stream() to accept a URL.

Now we can specify the URL of the video in the endpoint in the following format.

http://127.0.0.1:8000/?url=https://cdn.pixabay.com/video/2023/07/28/173530-849610807_large.mp4

We can also make the path operation function asynchronous using async.

FileResponse

Stream Local Video

The FileResponse class simply takes a file and streams the response.

from fastapi import FastAPI
from fastapi.responses import FileResponse

app = FastAPI()

# Video path
vid_path = 'video.mp4'

# Path to stream video
@app.get("/")
def video_stream():
    return FileResponse(vid_path, media_type='video/mp4')

In the above example, we simply passed a video file in the FileResponse class and returned the response.

We didn’t read and iterate over the bytes of the video to stream it like we did in the case of StreamingResponse.

The FileResponse class is ideal for relatively small or medium-sized files as it loads the file in the memory. Large files will consume more memory.

Stream Online Video

Streaming video from the URL using FileResponse isn’t the same as streaming the local video. As we know, the FileResponse class takes a file and streams it.

from fastapi import FastAPI
from fastapi.responses import FileResponse
import requests
import os

app = FastAPI()

# Video URL
vid_url = 'https://cdn.pixabay.com/video/2023/07/28/173530-849610807_large.mp4'
local_vid = 'sample_video.mp4'

# Function to save video from URL
def save_as_file(url, path):
    if not os.path.exists(path):
        response = requests.get(url)
        with open('sample_video.mp4', 'wb') as vid:
            vid.write(response.content)

save_as_file(vid_url, local_vid)

# Path to stream video
@app.get("/")
def video_stream():
    return FileResponse(local_vid, media_type='video/mp4')

In this example, instead of streaming the local video, we streamed the video from the URL using FileResponse.

This isn’t the best practice because we first saved the video from the URL in the local machine and then streamed the video.

The FileResponse class can be best utilized for local files, not web content.

That was all about streaming video directly to the browser but we don’t want to do that every time, instead, we want the video to be streamed on our website’s landing page.

Stream Video on Frontend

So, how to stream video on the frontend using FastAPI? We need to create a frontend using HTML and then stream the video.

First, create a directory named serve_video or whatever you want to name it, and create the following files and sub-directories.

serve_video/
   - templates/
      - display_video.html
   - app.py

The app.py file will contain our backend code and the display_video.html file will contain frontend code.

app.py

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse, HTMLResponse
from fastapi.templating import Jinja2Templates
import requests

app = FastAPI()
templates = Jinja2Templates(directory="templates")

vid_urls = [
    "https://cdn.pixabay.com/video/2023/07/28/173530-849610807_large.mp4",
    'https://cdn.pixabay.com/video/2024/03/31/206294_large.mp4',
    'https://cdn.pixabay.com/video/2023/10/11/184510-873463500_large.mp4',
    'https://cdn.pixabay.com/video/2023/06/17/167569-837244635_large.mp4'
]

# Stream the video from the URL
def stream_video(url):
    response = requests.get(url, stream=True)
    for portion in response.iter_content(chunk_size=1024*1024):
        yield portion

# Endpoint to render the HTML template with the video player
@app.get("/", response_class=HTMLResponse)
async def video_template(request: Request):
    return templates.TemplateResponse("display_video.html", {"request": request})

# Endpoint to stream the video
@app.get("/video/{video_id}")
async def video_stream(vid_id: int):
    if 0 <= vid_id < len(vid_urls):
        return StreamingResponse(stream_video(vid_urls[vid_id]), media_type="video/mp4")
    else:
        return HTMLResponse("Video not found", status_code=404)

In this example, we are streaming multiple videos on the frontend. We have a list of video URLs stored within vid_urls.

We have a stream_video() generator function that yields the video in smaller chunks.

We created an asynchronous path operation function video_template() that returns a response from HTML (@app.get("/", response_class=HTMLResponse)) file.

We are serving HTML from a template stored within the templates directory, so, we set up a Jinja2 template directory where HTML templates are stored (templates = Jinja2Templates(directory="templates")). This is where FastAPI will look for .html files.

Note: To render templates in FastAPI, you need to install the Jinja2 (pip install jinja2) library.

Then we returned the response using templates.TemplateResponse("display_video.html", {"request": request}). We passed our HTML template display_video.html and along with it, a dictionary containing Request object ({"request": request}). This Request object provides information about the incoming HTTP request, such as headers, cookies, and URLs, which can be accessed within the HTML template.

Next, we created an endpoint (@app.get("/video/{video_id}")) to stream individual videos based on the ID in the vid_ulrs. The video_stream(vid_id: int) path operation functions accept the index of the video in vid_urls.

Within the video_stream(), the if condition checks if the vid_id is within the range, if not, then raises an error otherwise the video is streamed based on the specified index.

display_video.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Stooooockzz</title>
</head>
<body>
    <h1>What we do?</h1>
    <p>We create awesome stock videos for free usage</p>
    <h3>Video Samples</h3>
    <div style="display: flex; justify-content: space-evenly;">
        <video width='20%' height='30%' style="border-radius: 10px;" controls autoplay>
            <source src="/video/0" type="video/mp4">
        </video>
        <video width='20%' height='30%' style="border-radius: 10px;" controls autoplay>
            <source src="/video/1" type="video/mp4">
        </video>
        <video width='20%' height='30%' style="border-radius: 10px;" controls autoplay>
            <source src="/video/2" type="video/mp4">
        </video>
        <video width='20%' height='30%' style="border-radius: 10px;" controls autoplay>
            <source src="/video/3" type="video/mp4">
        </video>
    </div>
</body>
</html>

This HTML file contains multiple <video> tags with the source pointing to the /video/{video_id} endpoint that streams the video based on ID.

Here, we passed <source src="/video/0" type="video/mp4"> within the <video> tag that represents streaming the first video from the vid_urls (in app.py) list. Similarly, we specified the same source but changed the ID within each <video> tag.

> fastapi dev app.py

Upon running the server using the above command, we get the following response on the frontend.

Conclusion

We’ve used StreaminResponse and FileResponse classes to stream local and web videos directly on the browser, and along with this we’ve also streamed multiple videos using StreamingResponse which was rendered using HTML files on the browser.

In this article, we’ve used URLs of the video residing on the internet but you can also fetch URLs from databases, local files, disks, etc.

That’s all for now.

Keep Coding✌✌.

10 Useful but Rarely Used OS Functions in Python

Sachin — Thu, 14 Nov 2024 13:55:55 +0000

You must have used functions provided by the os module in Python several times in your projects. These could be used to create a file, walk down a directory, get info on the current directory, perform path operations, and more.

In this article, we’ll discuss the functions that are as useful as any function in the os module but are rarely used.

os.path.commonpath()

When working with multiple files that share a common directory structure, you might want to find the longest shared path. os.path.commonpath() does just that. This can be helpful when organizing files or dealing with different paths across environments.

Here’s an example:

import os

paths = ['/user/data/project1/file1.txt', '/user/data/project2/file2.txt']
common_path = os.path.commonpath(paths)
print("Common Path:", common_path)

This code will give us the common path shared by these two paths.

Common Path: /user/data

You can see that os.path.commonpath() takes a list of path names, which might be impractical to manually write them down.

In that case, it is best to iterate over all of the directories, subdirectories, and file names and then look for the common path.

import os

def get_file_paths(directory, file_extension=None):
    # Collect all file paths in the directory (and subdirectories, if any)
    file_paths = []
    for root, dirs, files in os.walk(directory):
        for file in files:
            if file_extension is None or file.endswith(file_extension):
                file_paths.append(os.path.join(root, file))
    return file_paths


# Specify the root directory to start from
directory_path = 'D:/SACHIN/Pycharm/Flask-Tutorial'

# If you want to filter by file extension
file_paths = get_file_paths(directory_path, file_extension='.html')

# Find the common path among all files
if file_paths:
    common_path = os.path.commonpath(file_paths)
    print("Common Path:", common_path)
else:
    print("No files found in the specified directory.")

In this example, the function get_file_paths() traverses a directory from top to bottom and appends all the paths found in the file_paths list. This function optionally takes a file extension if we want to look out for specific files.

Now we can easily find the common path of any directory.

Common Path: D:\SACHIN\Pycharm\Flask-Tutorial\templates

os.scandir()

If you’re using os.listdir() to get the contents of a directory, consider using os.scandir() instead. It’s not only faster but also returns DirEntry objects, which provide useful information like file types, permissions, and whether the entry is a file or a directory.

Here’s an example:

import os

with os.scandir('D:/SACHIN/Pycharm/osfunctions') as entries:
    for entry in entries:
        print(f"{entry.name} : \n"
              f">>>> Is File: {entry.is_file()} \n"
              f">>>> Is Directory: {entry.is_dir()}")

In this example, we used os.scandir() and passed a directory and then we iterated over this directory and printed the info.

.idea : 
>>>> Is File: False 
>>>> Is Directory: True
main.py : 
>>>> Is File: True 
>>>> Is Directory: False
sample.py : 
>>>> Is File: True 
>>>> Is Directory: False

os.path.splitext()

Let’s say you’re working with files and need to check their extension, you can get help from os.path.splitext() function. It splits the file path into the root and extension, which can help you determine the file type.

import os

filename = 'report.csv'
root, ext = os.path.splitext(filename)
print(f"Root: {root} \n"
      f"Extension: {ext}")

Output

Root: report 
Extension: .csv

Look at some cases where paths can be weird, at that time how os.path.splitext() works.

import os

filename = ['.report', 'report', 'report.case.txt', 'report.csv.zip']
for idx, paths in enumerate(filename):
    root, ext = os.path.splitext(paths)
    print(f"{idx} - {paths}\n"
          f"Root: {root} | Extension: {ext}")

Output

0 - .report
Root: .report | Extension: 
1 - report
Root: report | Extension: 
2 - report.case.txt
Root: report.case | Extension: .txt
3 - report.csv.zip
Root: report.csv | Extension: .zip

os.makedirs()

There's already a frequently used function that allows us to create directories. But what about when you create nested directories?

Creating nested directories can be a hassle with os.mkdir() since it only makes one directory at a time. os.makedirs() allows you to create multiple nested directories in one go, and the exist_ok=True argument makes sure it doesn’t throw an error if the directory already exists.

import os

os.makedirs('project/data/files', exist_ok=True)
print("Nested directories created!")

When we run this program, it will create specified directories and sub-directories.

Nested directories created!

If we run the above program again, it won’t throw an error due to exist_ok=True.

os.replace()

Similar to os.rename(), os.replace() moves a file to a new location, but it safely overwrites any existing file at the destination. This is helpful for tasks where you’re updating or backing up files and want to ensure that old files are safely replaced.

import os

os.replace(src='main.py', dst='new_main.py')
print("File replaced successfully!")

In this code, main.py file will be renamed to new_main.py just as os.rename() function but this operation is like take it all or nothing. It means the file replacement happens in a single, indivisible step, so either the entire operation succeeds or nothing changes at all.

File replaced successfully!

os.urandom()

For cryptographic purposes, you need a secure source of random data. os.urandom() generates random bytes suitable for things like generating random IDs, tokens, or passwords. It’s more secure than the random module for sensitive data.

os.urandom() uses randomness generated by the operating system you are using from various resources to make bytes (data) unpredictable.

In Windows, it uses BCryptGenRandom() to generate random bytes.

import os

secure_token = os.urandom(16)  # 16 bytes of random data
print("Secure Token:", secure_token)
#Making it human-readable
print("Secure Token:", secure_token.hex())

Output

Secure Token: b'\x84\xd6\x1c\x1bKB\x7f\xcd\xf6\xb7\xc4D\x92z\xe3{'
Secure Token: 84d61c1b4b427fcdf6b7c444927ae37b

os.path.samefile()

The os.path.samefile() function in Python is used to check if two paths refer to the same file or directory on the filesystem. It’s particularly helpful in scenarios where multiple paths might point to the same physical file, such as when dealing with symbolic links, hard links, or different absolute and relative paths to the same location.

import os

is_same = os.path.samefile('/path/to/file1.txt', '/different/path/to/symlink_file1.txt')
print("Are they the same file?", is_same)

os.path.samefile() is designed to return True only if both paths reference the same file on disk, such as a file that’s hard-linked or symlinked to the same data on the filesystem.

os.path.relpath()

os.path.relpath() is a computation function that computes the relative path between two paths. This is particularly useful when building file paths dynamically or working with relative imports.

Consider the following example:

import os

# Target file path
target_path = "D:/SACHIN/Pycharm/osfunctions/project/engine/log.py"
# Starting point
start_path = "D:/SACHIN/Pycharm/osfunctions/project/interface/character/specific.py"

relative_path = os.path.relpath(target_path, start=start_path)
print(relative_path)

In this example, we have target_path which contains a path where we have to navigate and start_path contains a path from where we have to start calculating the relative path to target_path.

When we run this, we get the following output.

..\..\..\engine\log.py

This means we have to go up three directories and then down to engine/log.py.

os.fsync()

When we perform a file writing (file.write()) operation, the data isn’t saved to disk instantly instead the data is saved into the system’s buffer and if something unexpected happens before writing the data to the disk, the data gets lost.

os.fsync() forces the data to be written, ensuring data integrity. It’s especially useful in logging or when writing critical data that must not be lost.

import os

with open('data.txt', 'w') as f:
    f.write("gibberish!")
    os.fsync(f.fileno())  # Ensures data is written to disk

os.fsync(f.fileno()) is called to make sure the data is immediately written to the disk and not left in the buffer.

os.fsync() takes file descriptor that’s why we passed f.fileno() which is a unique integer assigned by the system to the file on which we are operating.

os.get_terminal_size()

If you’re creating CLI tools, formatting the output to fit the terminal width can make the output cleaner. os.get_terminal_size() gives you the current terminal width and height, making it easy to dynamically format content.

import os

size = os.get_terminal_size()
print(f"Terminal Width: {size.columns}, Terminal Height: {size.lines}")

When we run this code in the terminal, we get the size of the terminal on which we are running this script.

PS > py sample.py
Terminal Width: 158, Terminal Height: 12

Note: You may get an error when directly running the script on IDE where the program doesn’t have access to the terminal.

🏆Other articles you might be interested in if you liked this one

✅Streaming videos on the frontend in FastAPI.

✅How to fix circular imports in Python.

✅Template inheritance in Flask.

✅How to use type hints in Python?

✅How to find and delete mismatched columns from datasets in pandas?

✅How does the learning rate affect the ML and DL models?

That’s all for now.

Keep Coding✌✌.

Different Ways to Fix Circular Imports in Python

Sachin — Mon, 04 Nov 2024 11:56:27 +0000

Have you ever come across circular imports in Python? Well, it’s a very common code smell that indicates something’s wrong with the design or structure.

Circular Import Example

How does circular import occur? This import error usually occurs when two or more modules depending on each other try to import before fully initializing.

Let’s say we have two modules: module_1.py and module_2.py.

# module_1.py
from module_2 import ModY
class ModX:
    mody_obj = ModY()

# module_2.py
from module_1 import ModX
class ModY:
    modx_obj = ModX()

In the above code snippets, both module_1 and module_2 are mutually dependent on each other.

The initialization of mody_obj in module_1 depends on module_2 and the initialization of modx_obj in module_2 depends on module_1.

This is what we call a circular dependency. Both modules will stuck in the import loops while attempting to load each other.

If we run module_1.py, we’ll get the following traceback.

Traceback (most recent call last):
  File "module_1.py", line 1, in <module>
    from module_2 import ModY
  File "module_2.py", line 1, in <module>
    from module_1 import ModX
  File "module_1.py", line 1, in <module>
    from module_2 import ModY
ImportError: cannot import name 'ModY' from partially initialized module 'module_2' (most likely due to a circular import)

This error explains the situation of circular import. When the program attempted to import ModY from module_2, at that time module_2 wasn’t fully initialized (due to another import statement that attempts to import ModX from module_1).

How to fix circular imports in Python? There are different ways to get rid of circular imports in Python.

Fix Circular Imports in Python

Move code into a common file

We can move the code into a common file to avoid import errors and then try to import the modules from that file.

# main.py ----> common file
class ModX:
    pass

class ModY:
    pass

In the above code snippet, we moved the classes ModX and ModY into a common file (main.py).

# module_1.py
from main import ModY

class Mod_X:
    mody_obj = ModY()

# module_2.py
from main import ModX

class Mod_Y:
    modx_obj = ModX()

Now, module_1 and module_2 import the classes from main which fixes the circular import situation.

There is a problem with this approach, sometimes the codebase is so large that it becomes risky to move the code into another file.

Move the import to the end of the module

We can shift the import statement at the end of the module. This will give time to fully initialize the module before importing another module.

# module_1.py
class ModX:
   pass

from module_2 import ModY

class Mod_X:
   mody_obj = ModY()

# module_2.py
class ModY:
   pass

from module_1 import ModX

Importing module within the class/function scope

Importing modules within the class or function scope can avoid circular imports. This allows the module to be imported only when the class or function is invoked. It’s relevant when we want to minimize memory use.

# module_1.py
class ModX:
  pass

class Mod_X:
   from module_2 import ModY
   mody_obj = ModY()

# module_2.py
class ModY:
   pass

class Mod_Y:
   from module_1 import ModX
   modx_obj = ModX()

We moved the import statements within classes Mod_X and Mod_Y scope in module_1 and module_2 respectively.

If we run either module_1 or module_2, we’ll not get a circular import error. But, this approach makes the class accessible only within the class’s scope, so we can’t leverage the import globally.

Using module name/alias

Using the module name or just an alias like this solves the problem. This allows both modules to load fully by deferring circular dependency until runtime.

# module_1.py
import module_2 as m2

class ModX:
    def __init__(self):
        self.mody_obj = m2.ModY()

# module_2.py
import module_1 as m1

class ModY:
    def __init__(self):
        self.modx_obj = m1.ModX()

Using importlib library

We can also use the importlib library to import the modules dynamically.

# module_1.py
import importlib

class ModX:
    def __init__(self):
        m2 = importlib.import_module('module_2')
        self.mody_obj = m2.ModY()

# module_2.py
import importlib

class ModY:
    def __init__(self):
        m1 = importlib.import_module('module_1')
        self.mody_obj = m1.ModX()

Circular Imports in Python Packages

Usually, circular imports come from modules within the same package. In complex projects, the directory structure is also complex, with packages within packages.

These packages and sub-packages contain __init__.py files to provide easier access to modules. That’s where sometimes arises circular dependencies among modules unintentionally.

We have the following directory structure.

root_dir/
|- mainpkg/
|---- modpkg_x/
|-------- __init__.py
|-------- module_1.py
|-------- module_1_1.py
|---- modpkg_y/
|-------- __init__.py
|-------- module_2.py
|---- __init__.py
|- main.py

We have a package mainpkg and a main.py file. We have two sub-packages modpkg_x and modpkg_y within mainpkg.

Here’s what each Python file within modpkg_x and modpkg_y looks like.

mainpkg/modpkg_x/__init__.py

from .module_1 import ModX
from .module_1_1 import ModA

This file imports both classes (ModX and ModA) from module_1 and module_1_1.

mainpkg/modpkg_x/module_1.py

from ..modpkg_y.module_2 import ModY
class ModX:
    mody_obj = ModY()

The module_1 imports a class ModY from module_2.

mainpkg/modpkg_x/module_1_1.py

class ModA:
    pass

The module_1_1 imports nothing. It is not dependent on any module.

mainpkg/modpkg_y/__init__.py

from .module_2 import ModY

This file imports the class ModY from module_2.

mainpkg/modpkg_y/module_2.py

from ..modpkg_x.module_1_1 import ModA
class ModY:
    moda_obj = ModA()

The module_2 imports a class ModA from the module_1_1.

We have the following code within the main.py file.

root_dir/main.py

from mainpkg.modpkg_y.module_2 import ModY

def mody():
    y_obj = ModY()

mody()

The main file imports a class ModY from module_2. This file is dependent on module_2.

If we visualize the import cycle here, it would look like the following ignoring the __init__.py files within the modpkg_x and modpkg_y.

We can see that the main file depends on module_2, module_1 also depends on module_2 and module_2 depends on module_1_1. There is no import cycle.

But you know, modules depend on their __init__.py file, so the __init__.py file initializes first, and modules are re-imported.

This is what the import cycle looks like now.

This made module_1_1 depend on module_1, which is a fake dependency.

If this is the case, empty the sub-packages __init__.py files and using a separate __init__.py file can help by centralizing imports at the package level.

root_dir/
|- mainpkg/
|---- modpkg_x/
|-------- __init__.py  # empty file
|-------- module_1.py
|-------- module_1_1.py
|---- modpkg_y/
|-------- __init__.py  # empty file
|-------- module_2.py
|---- subpkg/
|-------- __init__.py
|---- __init__.py
|- main.py

In this structure, we added another sub-package subpkg within mainpkg.

mainpkg/subpkg/__init__.py

from ..modpkg_x.module_1 import ModX
from ..modpkg_x.module_1_1 import ModA
from ..modpkg_y.module_2 import ModY

This will allow internal modules to import from a single source, reducing the need for cross-imports.

Now we can update the import statement within the main.py file.

root_dir/main.py

from mainpkg.subpkg import ModY
def mody():
    y_obj = ModY()

mody()

This solves the problem of circular dependency between the modules within the same package.

Conclusion

Circular dependency or import in Python is a code smell which is an indication of serious re-structuring and refactoring of the code.

You can try any of these above-mentioned ways to avoid circular dependency in Python.

🏆Other articles you might be interested in if you liked this one

✅Template Inheritance in Flask with Example.

✅Difference between exec() and eval() with Examples.

✅Understanding the Use of global Keyword in Python.

✅Python Type Hints: Functions, Return Values, Variable.

✅Why Slash and Asterisk Used in Function Definition.

✅How does the learning rate affect the ML and DL models?

That’s all for now.

Keep Coding✌✌.

Making a Webapp is so EASY with Streamlit

Sachin — Wed, 23 Oct 2024 10:09:45 +0000

Streamlit is kind of popular among data scientists because you don't require frontend knowledge in general.

They provide simple and easy-to-implement elements and widgets without writing much code.

I used streamlit several times in my ML/AI projects, and the experience was great. You can focus more on writing the logic, and the frontend part (design, layout, and more) is handled very well by streamlit.

I created a demo webapp using streamlit and Python so that you can understand what I'm saying.

Webapp

This webapp is about converting an image format into another format, for instance, if your image is in PNG format, you can convert it into a JPEG image.

The following code makes the user interface of the webapp.

import streamlit as st
from imgconvrtr import convert_img_format
from PIL import Image

# Webpage setup
st.set_page_config(page_title="Image Convrtr")
st.title("Image Converter")
st.write("Convert your images in one _click_")

# File uploader
uploaded_file = st.file_uploader(
    "Upload an image",
    type=["png", "jpg", "jpeg", "jfif", "bmp"]
)

if uploaded_file is not None:
    # Show the uploaded image
    img = Image.open(uploaded_file)
    st.image(img, caption="Uploaded Image", use_column_width=True)

    # Show original image format
    st.write(f"Original format: {img.format}")

    # Output format selection
    format_options = ["PNG", "JPEG", "JFIF", "BMP"]
    output_format = st.selectbox("Choose output format", format_options)

    # Convert the image
    if img.format != output_format:
        if st.button("Convert"):
            converted_img = convert_img_format(uploaded_file, output_format.lower())
            st.write(f"Image converted to {output_format}")

            # Download button
            st.download_button(
                label=f"Download as {output_format}",
                data=converted_img,
                file_name=f"image.{output_format.lower()}",
                mime=f"image/{output_format.lower()}"
            )
    else:
        st.write("Select a different format... Yo!")

Now you already have a brief idea of what this webapp does. We can directly jump onto discussing the components used in this code.

In the beginning, you can see page elements like st.title and st.write which are used to set the page title and display text on the page, respectively.

Next, you can see a widget for uploading a file (in this case used for uploading an image). See how easy it is to create a file uploader.

st.image is used to display the uploaded image by the user.

Then we have a dropdown to select a variety of formats which is created using a selectbox (st.selectbox) widget.

Now, you can see we have two buttons (st.button and st.download_button). They both are the same but it's all about convenience.

The st.button displays a button widget that we used here for image conversion.

The st.download_button makes it useful when the user needs to directly download the file from the app.

Streamlit provides numerous elements and widgets for different purposes.

Now if you want to try this webapp, you need to install the required libraries:

pip install streamlit pillow

Here's the image conversion function:

from PIL import Image
import io

# Function to convert image format
def convert_img_format(image_file, frmat):
    with Image.open(image_file) as img:
        output_img = io.BytesIO()
        img.save(output_img, format=frmat.upper())
        output_img.seek(0)
        return output_img

Run the app using the following command:

streamlit run <script_name>.py

Replace <scipt_name> with the actual script name.

That's all for now.
Keep Coding✌✌

7 SQL Concepts You Should Know as a Data Scientist?

Sachin — Fri, 20 Sep 2024 13:00:04 +0000

If you are applying for data science roles, it is essential to have a solid understanding of key SQL topics and concepts.

Data retrieval and manipulation are critical skills for both Data Scientists and Data Analysts, as they form the foundation for effective data analysis.

Below are the SQL topics and concepts, prioritized from highest to lowest, that a data scientist should be proficient in.

1. SQL Querying

A strong foundation in SQL is crucial for data scientists, as it forms the basis for building efficient and effective data analysis processes. Proficiency in basic SQL querying, along with the ability to write complex yet readable and manageable queries, is essential. If you're not familiar with constructing queries or using the right commands, you risk facing constant challenges in your data tasks.

2. SQL Joins

SQL joins are a crucial concept for data scientists. In real-world scenarios, you rarely work with data from a single table. Instead, databases contain data from multiple sources, fields, or departments - often spread across different tables. These datasets contribute to various aspects of business growth, and as a data scientist, you need to combine and analyze them effectively.

For instance, if your organization is in the book business, you might have separate tables for members, books, orders, and more. To provide comprehensive insights or create a case study for your organization, you must understand how to join these tables and extract meaningful, relevant data.

3. CTE (Common Table Expressions)

You will need CTE (Common Table Expressions) to simplify complicated queries such as nested/subqueries, multiple joins, or aggregations.

CTEs are used when you don’t want to complicate things during data retrieval. This will help you break down complex queries into smaller queries that are more readable and manageable and your queries will look more structured.

Recursive CTE

Another form of CTE, recursive CTE is useful when you need to explore or traverse relationships between data, particularly when dealing with hierarchical structures or recursive relationships.

For instance, if a company has data about their employees and wants to know which manager each employee reports to, you can use a recursive CTE to find the relationships between managers and their subordinates.

Another example could involve sales data where different products belong to multiple categories in a hierarchical structure. You can use recursive CTEs to explore how these product categories are related, for example, by traversing from top-level categories down to subcategories.

4. Window Functions

Window functions play a crucial role in SQL. They allow you to perform a wide range of tasks, from data partitioning to time-series analysis, without altering the original structure of the data.

With window functions, you can carry out complex calculations, comparative analysis, and analyze data across rows, helping to derive meaningful insights while preserving row-level details.

5. Conditionals

Conditional expressions are useful they enable dynamic and flexible querying, data transformation, and feature engineering.

These expressions allow you to perform calculations, filter data, or modify values based on specific conditions, which is vital for data analysis, reporting, and preparation of data for machine learning models.

Conditional expressions (like CASE or IF()) allow you to clean, standardize, and transform data based on certain criteria. This is essential when dealing with inconsistent or incomplete data.

6. Aggregate Functions

Aggregate functions are essential for performing mathematical computations and summarizing data across different categories or fields. They allow you to compute metrics like sums, averages, counts, and more, based on specific groups within the data.

7. PL/SQL

While PL/SQL is less commonly used in everyday data science tasks, it's important to know how to create functions, stored procedures, triggers, and views in SQL.

PL/SQL (Procedural Language/SQL) extends SQL with procedural programming features like loops, conditions, and exception handling, allowing for more complex operations similar to any other programming language.

It is particularly useful for complex data processing, automating data tasks, reusing code, and optimizing query performance, ultimately saving time and resources.

That’s all for now.

Keep Coding✌✌.

NumPy's Argmax? How it Finds Max Elements from Arrays

Sachin — Fri, 14 Jun 2024 15:00:00 +0000

NumPy is most often used to handle or work with arrays (multidimensional, masked) and matrices. It has a collection of functions and methods to operate on arrays like statistical operations, mathematical and logical operations, shape manipulation, linear algebra, and much more.

Argmax function

numpy.argmax() is one of the functions provided by NumPy that is used to return the indices of the maximum element along an axis from the specified array.

Syntax

numpy.argmax(a, axis=None, out=None)

Parameters:

a - The input array we will work on
axis - It is optional. We can specify an axis like 1 or 0 to find the maximum value index horizontally or vertically.
out - By default, it is None. It provides a feature to insert the output to the out array, but the array should be of appropriate shape and dtype.

Return value

The array of integers is returned with the indices of max values from the array a with the same shape as a.shape with the dimension along the axis removed.

Finding the index of the max element

Let's see the basic example to find the index of the max element in the array.

Working with a 1D array without specifying the axis

# Importing Numpy
import numpy as np

# Working with a 1D array
inp_arr = np.array([5, 2, 9, 4, 2])

# Applying argmax() function
max_elem_index = np.argmax(inp_arr)

# Printing index
print("MAX ELEMENT INDEX:", max_elem_index)

Output

MAX ELEMENT INDEX: 2

Working with a 2D array without specifying the axis

When we work with 2D arrays in numpy and try to find the index of the max element without specifying the axis, the array we are working on has the element index the same as the 1D array or flattened array.

# Importing Numpy
import numpy as np

# Creating 2D array
arr = np.random.randint(16, size=(4, 4))

# Array preview
print("INPUT ARRAY: \n", arr)

# Applying argmax()
elem_index = np.argmax(arr)

# Displaying max element index
print("\nMAX ELEMENT INDEX:", elem_index)

Output

INPUT ARRAY: 
 [[ 5  5  4 12]
 [12 15 13  0]
 [11 13  2  6]
 [ 6  8  8  9]]

MAX ELEMENT INDEX: 5

Finding the index of the max element along the axis

Things will change when we specify the axis and try to find the index of the max element along it.

When the axis is 0

When we specify the axis=0, then the argmax function will find the index of the max element vertically in the multidimensional array that the user specified. Let's understand it by an illustration below.

In the above illustration, argmax() function returned the max element index from the 1^st column which is 1 and then returned the max element index from the 2^nd column which is again 1, and the same does for the 3^rd and the 4^th column.

# Importing Numpy
import numpy as np

# Creating 2D array
arr = np.random.randint(16, size=(4, 4))

# Array preview
print("INPUT ARRAY: \n", arr)

# Applying argmax()
elem_index = np.argmax(arr, axis=0)

# Displaying max element index
print("\nMAX ELEMENT INDEX:", elem_index)

Output

INPUT ARRAY: 
 [[ 8  6 10  3]
 [ 4  5  9  1]
 [ 6 15 13 13]
 [ 4 14 15 13]]

MAX ELEMENT INDEX: [0 2 3 2]

When the axis is 1

When we specify the axis=1, the argmax function will find the index of the max element horizontally in the multidimensional array that the user specified. Let's understand it by an illustration below.

In the above illustration, argmax() function returned the max element index from the 1^st row which is 2 and then returned the max element index from the 2^nd row which is 1, and the same does for the 3^rd and the 4^th row.

Code example

# Importing Numpy
import numpy as np

# Creating 2D array
arr = np.random.randint(12, size=(4, 3))

# Array preview
print("INPUT ARRAY: \n", arr)

# Applying argmax()
elem_index = np.argmax(arr, axis=1)

# Displaying max element index
print("\nMAX ELEMENT INDEX:", elem_index)

Output

INPUT ARRAY: 
 [[ 7  8  0]
 [ 3  0 11]
 [ 7  6  0]
 [10  8  1]]

MAX ELEMENT INDEX: [1 2 0 0]

Multiple occurrences of the highest value

Sometimes, we can come across multidimensional arrays with multiple occurrences of the highest values along the particular axis, then what will happen?

The argmax() function will return the index of the highest value that occurs first in a particular axis.

Illustration showing multiple occurrences of the highest values along axis 0.

Illustration showing multiple occurrences of the highest values along axis 1.

Code example

# Importing Numpy
import numpy as np

# Defining the array
arr = np.array([[2, 14, 9, 4, 5],
                [7, 14, 53, 10, 4],
                [91, 2, 41, 6, 91]])

# Displaying the highest element index along axis 0
print("MAX ELEMENT INDEX:", np.argmax(arr, axis=0))

# Displaying the highest element index along axis 1
print("\nMAX ELEMENT INDEX:", np.argmax(arr, axis=1))

# Flattening the array, making it a 1D array
flattened_arr = arr.flatten()
print("\nThe array is flattened into 1D array:", flattened_arr)

# Displaying the highest element index
print("\nMAX ELEMENT INDEX:", np.argmax(flattened_arr))

Output

MAX ELEMENT INDEX: [2 0 1 1 2]

MAX ELEMENT INDEX: [1 2 0]

The array is flattened into 1D array: [ 2 14  9  4  5  7 14 53 10  4 91  2 41  6 91]

MAX ELEMENT INDEX: 10

Explanation

In the above code, when we try to find the indices of the max elements along the axis 0, we got an array with values [2 0 1 1 2], if we look at the 2^nd column, 14 is the highest value at the 0^th and the 1^st index, we got 0 because the value 14 at the 0^th index occurred first when finding the highest value.

The same goes for the array we obtained in the second output when we provided the axis 1, in the 3^rd row, 91 is the highest value at the 0^th and the 4^th index, the value 91 at the 0^th index occurred first when finding the highest value hence we got output 0.

Using the out parameter

The out parameter in numpy.argmax() function is optional and by default it is None.

The out parameter stores the output array(containing indices of the max elements in a particular axis) in a numpy array. The array specified in the out parameter should be of shape and dtype, the same as the input array.

Code Example

# Importing Numpy
import numpy as np

# Creating array filled with zeroes which then be replaced
out_array = np.zeros((4,), dtype=int)
print("ARRAY w/ ZEROES:", out_array)

# Input array
arr = np.random.randint(16, size=(4, 4))
print("INPUT ARRAY:\n", arr)

# Storing the indices of the max elements(axis=1) in the out_array
print("\nAXIS 1:", np.argmax(arr, axis=1, out=out_array))

# Storing the indices of the max elements(axis=0) in the out_array
print("\nAXIS 0:", np.argmax(arr, axis=0, out=out_array))

Output

ARRAY w/ ZEROES: [0 0 0 0]
INPUT ARRAY:
 [[ 4  2 14 15]
 [ 6 15  2  1]
 [13  6 13  3]
 [ 5  1 13  9]]

AXIS 1: [3 1 0 2]

AXIS 0: [2 1 0 0]

Explanation

We created an array filled with zeroes named out_array, and we defined the shape and dtype same as the input array and then used the numpy.argmax() function to get the indices of the max elements along the axis 1 and 0 and stored them in the out_array that we defined earlier.

The numpy.zeros() has by default dtype float that's why we defined the dtype in the above code because our input array has the dtype=int.

If we didn't specify the dtype in the above code, it would throw an error.

# Importing Numpy
import numpy as np

# Creating array filled with zeroes without specifying dtype
out_array = np.zeros((4,))
print("ARRAY w/ ZEROES:", out_array)

# Input array
arr = np.random.randint(16, size=(4, 4))
print("INPUT ARRAY:\n", arr)

print("\nAXIS 1:", np.argmax(arr, axis=1, out=out_array))

Output

ARRAY w/ ZEROES: [0. 0. 0. 0.]
INPUT ARRAY:
 [[14  9  3  4]
 [ 9  2  4  8]
 [ 5  1  9  1]
 [ 6  0 10  7]]

TypeError: Cannot cast array data from dtype('float64') to dtype('int64') according to the rule 'safe'

Conclusion

That was the insight of the argmax() function in NumPy. Let's review what we've learned:

numpy.argmax() function returns the index of the highest value in an array. If the maximum value occurs more than once in an array(multidimensional or flattened array), then the argmax() function will return the index of the highest value which occurred first.
We can specify the axis parameter when working with a multidimensional array to get the result along a particular axis. If we specify axis=0, then we'll get the index of the highest values vertically in the multidimensional array, and for axis=1, we'll get the result horizontally in the multidimensional array.
We can store the output in another array specified in the out parameter; however, the array should be compatible with the input array.

That's all for now

Keep Coding✌✌

OOPs Concept: getitem, setitem & delitem in Python

Sachin — Wed, 12 Jun 2024 15:01:00 +0000

Python has numerous collections of dunder methods(which start with double underscores and end with double underscores) to perform various tasks. The most commonly used dunder method is __init__ which is used in Python classes to create and initialize objects.

In this article, we'll see the usage and implementation of the underutilized dunder methods such as __getitem__, __setitem__, and __delitem__ in Python.

getitem

The name getitem depicts that this method is used to access the items from the list, dictionary and array.

If we have a list of names and want to access the item on the third index, we would use name_list[3], which will return the name from the list on the third index. When the name_list[3] is evaluated, Python internally calls __getitem__ on the data (name_list.__getitem__(3)).

The following example shows us the practical demonstration of the above theory.

# List of names
my_list = ['Sachin', 'Rishu', 'Yashwant', 'Abhishek']

# Accessing items using bracket notation
print('Accessed items using the bracket notation')
print(my_list[0])
print(my_list[2], "\n")

# Accessing items using __getitem__
print('Accessed items using the __getitem__')
print(my_list.__getitem__(1))
print(my_list.__getitem__(3))

We used the commonly used bracket notation to access the items from the my_list at the 0th and 2nd index and then to access the items at the 1st and 3rd index, we implemented the __getitem__ method.

Accessed items using the bracket notation
Sachin
Yashwant 

Accessed items using the __getitem__
Rishu
Abhishek

Syntax

__getitem__(self, key)

The __getitem__ is used to evaluate the value of self[key] by the object or instance of the class. Just like we saw earlier, object[key] is equivalent to object.__getitem__(key).

self - object or instance of the class

key - value we want to access

getitem in Python classes

# Creating a class
class Products:
    def __getitem__(self, items):
        print(f'Item: {items}')


item = Products()
item['RAM', 'ROM']
item[{'Storage': 'SSD'}]
item['Graphic Card']

We created a Python class named Products and then defined the __getitem__ method to print the items. Then we created an instance of the class called item and then passed the values.

Item: ('RAM', 'ROM')
Item: {'Storage': 'SSD'}
Item: Graphic Card

These values are of various data types and were actually parsed, for example, item['RAM', 'ROM'] was parsed as a tuple and this expression was evaluated by the interpreter as item.__getitem__(('RAM', 'ROM')).

Checking the type of the item along with the items.

import math
# Creating a class
class Products:
    # Printing the types of item along with items
    def __getitem__(self, items):
        print(f'Item: {items}. Type: {type(items)}')


item = Products()
item['RAM', 'ROM']
item[{'Storage': 'SSD'}]
item['Graphic Card']
item[math]
item[89]

Output

Item: ('RAM', 'ROM'). Type: <class 'tuple'>
Item: {'Storage': 'SSD'}. Type: <class 'dict'>
Item: Graphic Card. Type: <class 'str'>
Item: <module 'math' (built-in)>. Type: <class 'module'>
Item: 89. Type: <class 'int'>

Example

In the following example, we created a class called Products, an __init__ that takes items and a price, and a __getitem__ that prints the value and type of the value passed inside the indexer.

Then we instantiated the class Products and passed the arguments 'Pen' and 10 to it, which we saved inside the obj. Then, using the instance obj, we attempted to obtain the values by accessing the parameters items and price.

# Creating a class
class Products:
    # Creating a __init__ function
    def __init__(self, items, price):
        self.items = items
        self.price = price

    def __getitem__(self, value):
        print(value, type(value))

# Creating instance of the class and passing the values
obj = Products('Pen',10)
# Accessing the values
obj[obj.items]
obj[obj.price]

Output

Pen <class 'str'>
10 <class 'int'>

setitem

The __setitem__ is used to assign the values to the item. When we assign or set a value to an item in a list, array, or dictionary, this method is called internally.

Here's an example in which we created a list of names, and attempted to modify the list by changing the name at the first index (my list[1] = 'Yogesh'), and then printed the updated list.

To demonstrate what the interpreter does internally, we modified the list with the help of __setitem__.

# List of names
my_list = ['Sachin', 'Rishu', 'Yashwant', 'Abhishek']

# Assigning other name at the index value 1
my_list[1] = 'Yogesh'
print(my_list)

print('-'*20)

# What interpreter does internally
my_list.__setitem__(2, 'Rishu')
print(my_list)

When we run the above code, we'll get the following output.

['Sachin', 'Yogesh', 'Yashwant', 'Abhishek']
--------------------
['Sachin', 'Yogesh', 'Rishu', 'Abhishek']

Syntax

__setitem__(self, key, value)

The __setitem__ assigns a value to the key. If we call self[key] = value, then it will be evaluated as self.__setitem__(key, value).

self - object or instance of the class

key - the item that will be replaced

value - key will be replaced by this value

setitem in Python classes

The following example demonstrates the implementation of the __setitem__ method in a Python class.

# Creating a class
class Roles:
    # Defining __init__ method
    def __init__(self, role, name):
        # Creating a dictionary with key-value pair
        self.detail = {
            'name': name,
            'role': role
        }

    # Defining __getitem__ method
    def __getitem__(self, key):
        return self.detail[key]

    # Function to get the role and name
    def getrole(self):
        return self.__getitem__('role'), self.__getitem__('name')

    # Defining __setitem__ method
    def __setitem__(self, key, value):
        self.detail[key] = value

    # Function to set the role and name
    def setrole(self, role, name):
        print(f'{role} role has been assigned to {name}.')
        return self.__setitem__('role', role), self.__setitem__('name', name)

# Instantiating the class with required args
data = Roles('Python dev', 'Sachin')
# Printing the role with name
print(data.getrole())

# Setting the role for other guys
data.setrole('C++ dev', 'Rishu')
# Printing the assigned role with name
print(data.getrole())
# Setting the role for other guys
data.setrole('PHP dev', 'Yashwant')
# Printing the assigned role with name
print(data.getrole())

We created a Roles class and a __init__ function, passing the role and name parameters and storing them in a dictionary.

Then we defined the __getitem__ method, which returns the key's value, and the getrole() function, which accesses the value passed to the key name and role.

Similarly, we defined the __setitem__ method, which assigns a value to the key, and we created the setrole() function, which assigns the specified values to the key role and name.

The class Roles('Python dev,' 'Sachin') was then instantiated with required arguments and stored inside the data object. We printed the getrole() function to get the role and name, then we called the setrole() function twice, passing it the various roles and names, and printing the getrole() function for each setrole() function we defined.

('Python dev', 'Sachin')
C++ dev role has been assigned to Rishu.
('C++ dev', 'Rishu')
PHP dev role has been assigned to Yashwant.
('PHP dev', 'Yashwant')

We got the values passed as an argument to the class but after it, we set the different roles and names and got the output we expected.

delitem

The __delitem__ method deletes the items in the list, dictionary, or array. The item can also be deleted using the del keyword.

# List of names
my_list = ['Sachin', 'Rishu', 'Yashwant', 'Abhishek']

# Deleting the first item of the list
del my_list[0]
print(my_list)

# Deleting the item using __delitem__
my_list.__delitem__(1)
print(my_list)

----------
['Rishu', 'Yashwant', 'Abhishek']
['Rishu', 'Abhishek']

In the above code, we specified the del keyword and then specified the index number of the item to be deleted from my_list.

So, when we call del my_list[0] which is equivalent to del self[key], Python will call my_list.__delitem__(0) which is equivalent to self.__delitem__(key).

delitem in Python class

class Friends:
    def __init__(self, name1, name2, name3, name4):
        self.n = {
            'name1': name1,
            'name2': name2,
            'name3': name3,
            'name4': name4
        }

    # Function for deleting the entry
    def delname(self, key):
        self.n.__delitem__(key)

    # Function for adding/modifying the entry
    def setname(self, key, value):
        self.n[key] = value


friend = Friends('Sachin', 'Rishu', 'Yashwant', 'Abhishek')
print(friend.n, "\n")

# Deleting an entry
friend.delname('name3')
print('After deleting the name3 entry')
print(friend.n, "\n")

# Modifying an entry
friend.setname('name2', 'Yogesh')
print('name2 entry modified')
print(friend.n, "\n")

# Deleting an entry
friend.delname('name2')
print('After deleting the name2 entry')
print(friend.n)

We defined the delname function in the preceding code, which takes a key and deletes that entry from the dictionary created inside the __init__ function, as well as the setname function, which modifies/adds the entry to the dictionary.

Then we instantiated the Friends class, passed in the necessary arguments, and stored them in an instance called friends.

Then we used the delname function to remove an entry with the key name3 before printing the updated dictionary. In the following block, we modified the entry with the key name2 to demonstrate the functionality of setname function and printed the modified dictionary, then we deleted the entry with the key name2 and printed the updated dictionary.

{'name1': 'Sachin', 'name2': 'Rishu', 'name3': 'Yashwant', 'name4': 'Abhishek'} 

After deleting the name3 entry
{'name1': 'Sachin', 'name2': 'Rishu', 'name4': 'Abhishek'} 

name2 entry modified
{'name1': 'Sachin', 'name2': 'Yogesh', 'name4': 'Abhishek'} 

After deleting the name2 entry
{'name1': 'Sachin', 'name4': 'Abhishek'}

Conclusion

We learned about the __getitem__, __setitem__, and __delitem__ methods in this article. We can compare __getitem__ to a getter function because it retrieves the value of the attribute, __setitem__ to a setter function because it sets the value of the attribute, and __delitem__ to a deleter function because it deletes the item.

We implemented these methods within Python classes in order to better understand how they work.

We've seen code examples that show what Python does internally when we access, set, and delete values.

🏆Other articles you might be interested in if you liked this one

✅How to use and implement the __init__ and __call__ in Python.

✅Types of class inheritance in Python with examples.

✅How underscores modify accessing the attributes and methods in Python.

✅Create and manipulate the temporary file in Python.

✅Display the static and dynamic images on the frontend using FastAPI.

✅Python one-liners to boost your code.

✅Perform a parallel iteration over multiple iterables using zip() function in Python.

That's all for now

Keep Coding✌✌