Forem: Vidyasagar SC Machupalli

Finding Today's Changed Files: A Quick Python Script for File Uploads

Vidyasagar SC Machupalli — Mon, 03 Nov 2025 03:03:08 +0000

My friend recently asked me for help with a common problem: they needed to upload files to a remote server, but didn't want to upload everything—just the files that had changed today. Manually checking modification dates for dozens of files? Not fun.

So I wrote a quick Python script to solve it.

The Problem

When you're syncing files to a remote server, uploading everything is inefficient and time-consuming. You really only need the files you've modified today—whether that's code you've been working on, documents you've edited, or data files you've processed.

The Solution

This get_latest_files_by_today() function scans a folder and returns only the files modified today, sorted with the most recent first:

def get_latest_files_by_today(folder_path: str) -> list:

It's straightforward:

Validates the folder path to avoid errors
Compares each file's modification date to today's date
Sorts results by time (newest first)
Returns clean filenames ready to use

The key insight is comparing just the date portion—ignoring the specific time—so anything touched today qualifies:

today = datetime.now().date()
modified_date = datetime.fromtimestamp(modified_timestamp).date()
if modified_date == today:
    today_files.append((file_path, modified_timestamp))

Real-World Usage

My friend now runs this script at the end of their workday:

latest_files = get_latest_files_by_today("/path/to/project")
print(latest_files)

The output gives them an instant list of exactly which files need uploading to the remote server. No guesswork, no manual checking, no accidentally missing a file.

Beyond File Uploads

While we built this for upload workflows, it's useful anytime you need to track today's file activity:

Creating daily backups of changed files
Reviewing what you worked on before leaving for the day
Monitoring build outputs or logs
Checking which documents were edited during a meeting

Simple Tools, Big Impact

This script uses only Python's standard library—no external dependencies, no complex setup. Just point it at a folder and get your results.

Sometimes the most helpful code isn't the most sophisticated. It's the utility that solves a real problem in 40 lines, saves you 15 minutes every day, and just works.

Got a folder that needs smart file filtering? Give this script a try.

import os
from datetime import datetime

def get_latest_files_by_today(folder_path: str) -> list:
    """
    Retrieves and sorts files in a given folder, returning only those modified today.

    Args:
        folder_path (str): The path to the folder to scan.

    Returns:
        list: A list of full filenames (name + extension) of files modified today,
              sorted by modification time (latest first).
    """
    if not os.path.isdir(folder_path):
        print(f"Error: Folder '{folder_path}' does not exist.")
        return []

    today = datetime.now().date()
    today_files = []

    for filename in os.listdir(folder_path):
        file_path = os.path.join(folder_path, filename)

        if os.path.isfile(file_path):
            modified_timestamp = os.path.getmtime(file_path)
            modified_date = datetime.fromtimestamp(modified_timestamp).date()

            if modified_date == today:
                today_files.append((file_path, modified_timestamp))

    # Sort files by modification time in descending order (latest first)
    today_files.sort(key=lambda x: x[1], reverse=True)

    # Extract only the full filenames
    sorted_filenames = [os.path.basename(file_path) for file_path, _ in today_files]

    return sorted_filenames

# Example usage:
current_directory = os.getcwd()
latest_files = get_latest_files_by_today(current_directory)
print(latest_files)

# Or with a specific folder:
# my_folder = "/path/to/your/folder"
# latest_files_in_folder = get_latest_files_by_today(my_folder)
# print(latest_files_in_folder)

output:

/Users/vmac/Documents/Code/Python/.venv/bin/python /Users/vmac/Documents/Code/Python/latestfile.py 
Files changed today ['latestfile.py', 'abc.py']

Process finished with exit code 0

Resolving Secrets Manager DNS Resolution Errors in Terraform and Pulumi IaC

Vidyasagar SC Machupalli — Fri, 08 Aug 2025 18:30:32 +0000

I love Terraform and as a curious explorer, I started looking for an alternative when Terraform introduced HCL2. An IaC alternative where I can use the programming language of my choice for declarative approach.

Infrastructure as Code (IaC) has revolutionized how organizations manage and provision cloud resources. By defining infrastructure through code rather than manual processes, teams can achieve consistency, version control, and automated deployment workflows. However, working with IaC tools can present unique challenges, particularly when integrating with cloud services like IBM Cloud Secrets Manager.

While creating a Pulumi project to provision Secrets Manager group instance, I came across an error

CreateSecretGroupWithContext failed: Post "https://0a21f2b7-e695-42e4-873f.us-south.secrets-manager.appdomain.cloud/api/v2/secret\_groups": dial tcp: lookup 0a21f2b7-e695-42e4-873f.us-south.secrets-manager.appdomain.cloud: no such host

halting the infrastructure provisioning entirely. This article provides a comprehensive guide to understanding, diagnosing, and resolving these DNS resolution errors while working with IBM Cloud Secrets Manager through both Terraform and Pulumi.

Understanding Infrastructure as Code (IaC)

Infrastructure as Code allows developers and operations teams to manage infrastructure through machine-readable definition files rather than physical hardware configuration or interactive configuration tools. This approach brings several benefits including repeatability, consistency, and the ability to version control infrastructure changes.

Terraform vs Pulumi: A Brief Overview

While Terraform uses HashiCorp Configuration Language (HCL) and has become the industry standard for IaC, Pulumi offers a modern alternative that allows infrastructure definition using familiar programming languages like Python, TypeScript, Go, .NET, and Java. Both tools support IBM Cloud through dedicated providers, but they differ significantly in their approach and capabilities.

Key differences include:

Language Support: Terraform uses HCL exclusively, while Pulumi supports multiple general-purpose programming languages
State Management: Terraform uses local or remote state files, while Pulumi uses its SaaS backend by default
Testing Capabilities: Pulumi provides native support for unit and integration testing through programming language ecosystems

Introduction to Pulumi

What is Pulumi?

Pulumi is a modern Infrastructure as Code platform that leverages existing programming languages and their native ecosystems to interact with cloud resources through the Pulumi SDK. Unlike traditional IaC tools that require learning domain-specific languages, Pulumi allows developers to use languages they already know and love.

Key Advantages of Pulumi over Traditional IaC Tools

Enhanced Developer Experience: Pulumi provides full IDE support with code completion, error checking, and refactoring capabilities that come naturally with general-purpose programming languages.

Advanced Testing Capabilities: Since Pulumi uses real programming languages, teams can leverage existing testing frameworks and methodologies to validate their infrastructure code.

Stronger Abstraction and Modularity: Programming language features like functions, classes, and packages enable more sophisticated infrastructure abstractions than simple configuration languages allow.

Native Integration with Application Code: Teams can embed infrastructure management directly in their applications or use the same languages for both application and infrastructure development.

Pulumi’s Multi-Language Support

Pulumi supports a wide range of programming languages:

TypeScript/JavaScript: Native Node.js support with full npm ecosystem
Python: Full Python 3.x support with pip package management
Go: Native Go modules and tooling
.NET: Support for C#, F#, and VB.NET
Java: Full JVM ecosystem support
YAML: For simpler, declarative scenarios

Setting Up IBM Cloud with Pulumi

Installing the IBM Cloud Provider for Pulumi

To work with IBM Cloud resources in Pulumi, you need to install the IBM Cloud provider. The IBM Cloud provider for Pulumi must be installed as a local package:

pulumi package add terraform-provider ibm-cloud/ibm

Understanding the Service Endpoint Configuration Error

Though I am exploring Pulumi, the IBM Pulumi provider is based on Terraform provider. So, I took a step back to see if the Terraform provider works but I hit the same error. While digging deeper, I found this GitHub issue that clarifies many questions

Error Analysis from GitHub Issue

The specific issue documented in GitHub issue reveals a fundamental misconfiguration problem. When users specify service_endpoints = "public-and-private" in their Terraform configuration for IBM Secrets Manager, the resulting instance is provisioned with only private endpoints, despite the configuration suggesting otherwise.

Root Cause Identification:

The issue stems from IBM’s April 2025 change to make private endpoints the default for Secrets Manager instances. However, the critical distinction is that Secrets Manager uses a different parameter structure than other IBM Cloud services for endpoint configuration.

Configuration Inconsistency:

Standard IBM Services: Use service_endpoints parameter
Secrets Manager: Uses allowed_network within the parameters block

Analyzing the Incorrect Configuration

The problematic Terraform configuration from the GitHub issue:

resource "ibm_resource_instance" "secrets_manager" {
  count = var.provision_sm_instance == true ? 1 : 0
  name = var.sm_instance_name
  service = "secrets-manager"
  plan = var.sm_service_plan
  location = var.region
  resource_group_id = data.ibm_resource_group.sm_instance.id
  tags = var.tags
  service_endpoints = "public-and-private" # ← This is INCORRECT for Secrets Manager

  timeouts {
    create = "20m"
  }
}

Pulumi implementation

secrets_manager = ibm.ResourceInstance(
    "vpn-secrets-manager",
    name="vpn-secrets-manager",
    service="secrets-manager",
    plan="standard",
    location=region,
    resource_group_id=resource_group_id,
    service_endpoints="public-and-private", 
    tags=["vpn", "certificates", "pulumi"],
    opts=secrets_manager_opts
)

Result : Despite specifying service_endpoints = "public-and-private", the instance is provisioned with only private endpoints, leading to DNS resolution failures when attempting to create secret groups.

Pulumi error

The Correct Configuration Approach

Based on IBM’s documentation and the GitHub issue resolution, the correct approach uses the parameters block:

resource "ibm_resource_instance" "secrets_manager" {
  count = var.provision_sm_instance == true ? 1 : 0
  name = var.sm_instance_name  
  service = "secrets-manager"
  plan = var.sm_service_plan
  location = var.region
  resource_group_id = data.ibm_resource_group.sm_instance.id
  tags = var.tags

  parameters = {
    "allowed_network" = "public-and-private" # ← This is CORRECT for Secrets Manager
  }

  timeouts {
    create = "20m"
  }
}

Pulumi code fix:

Pulumi Code repository

secrets_manager = ibm.ResourceInstance(
    "vpn-secrets-manager",
    name="vpn-secrets-manager",
    service="secrets-manager",
    plan="standard",
    location=region,
    resource_group_id=resource_group_id,
    #service_endpoints=secrets_manager_service_endpoints,
    parameters={
        "allowed_network" : secrets_manager_service_endpoints,    
    },
    tags=["vpn", "certificates", "pulumi"],
    opts=secrets_manager_opts
)

For complete Pulumi code examples, refer to my GitHub repository

Conclusion

The IBM Secrets Manager DNS resolution error stems from a critical configuration difference where Secrets Manager requires parameters = { "allowed_network" = "..." } instead of the standard service_endpoints parameter.

Key Solutions:

Always use allowed_network within the parameters block for Secrets Manager
Implement validation to catch configuration errors early
Use infrastructure modules to encapsulate correct patterns
Test endpoint accessibility after provisioning

Pulumi Advantages: Offers superior error handling, testing capabilities, and IDE support compared to traditional HCL-based approaches, making it easier to implement robust validation and error handling patterns.

By following these patterns and understanding the service-specific requirements, teams can avoid DNS resolution errors and build reliable infrastructure automation workflows for IBM Secrets Manager.

The Essential Role of Process Monitoring Scripts in Linux Environments

Vidyasagar SC Machupalli — Mon, 12 May 2025 15:23:32 +0000

In today’s complex computing environments, keeping tabs on running processes isn’t just good practice — it’s absolutely critical. Whether you’re managing a fleet of Linux servers, maintaining development workstations, or overseeing macOS production systems, understanding what’s running (and what should be running) forms the foundation of system health, security, and performance.

This comprehensive guide introduces a powerful yet lightweight Bash script that delivers cross-platform process monitoring with enterprise-grade capabilities. We’ll explore:

The diverse landscape of Linux distributions and their monitoring needs
How this script solves real-world administration challenges
Detailed usage examples for various scenarios
Advanced integration techniques for professional environments

Understanding the Linux Distribution Landscape

1. Enterprise-Grade Workhorses

Red Hat Enterprise Linux (RHEL), Ubuntu LTS, and SUSE Linux Enterprise dominate mission-critical environments. These distributions prioritize:

Stability over cutting-edge features
Long-term support (5+ years typically)
Certified compatibility with enterprise hardware/software

Monitoring Consideration: These systems often run business-critical services where unexpected process termination can cost thousands per minute of downtime.

2. Rolling Release Distributions

Arch Linux, openSUSE Tumbleweed, and Gentoo appeal to developers and enthusiasts with:

Continuous updates to the latest software versions
Minimal default installations
Customization-focused philosophies

Monitoring Consideration: Frequent updates increase the chance of service disruptions, making automated monitoring particularly valuable.

3. Specialized and Lightweight Systems

From Alpine Linux’s container-optimized 5MB footprint to Kali’s security toolkit, these distributions serve niche purposes:

Embedded systems (Raspberry Pi OS)
Security auditing (Kali, Parrot OS)
Edge computing (Fedora IoT)

Monitoring Consideration: Resource constraints demand ultra-efficient monitoring tools that don’t contribute to system load.

Introducing the Universal Process Monitor

Our solution addresses these diverse needs through intelligent design:

Key Features

Dual-Platform Support

Automatically detects and adapts to Linux or macOS environments
Handles differences in command syntax between systems

2. Flexible Output Options

Color-coded terminal output (default)
JSON format for programmatic use (--json flag)
Log file support (--log-file option)

3. Intelligent Process Detection

Uses pgrep for reliable process matching
Excludes the monitoring script itself from results
Normalizes output across different platforms

4. Lightweight Operation

Minimal dependencies (only basic Unix tools)
Fast execution with low resource usage

Usage Examples

Basic Process Check

./process_monitor.sh nginx mysql docker

Sample Output:

PROCESS CHECK - Ubuntu 22.04 LTS
TIME: 2023-07-15 14:30:45

nginx RUNNING 1234 www-data /usr/sbin/nginx -g daemon off;
mysql RUNNING 5678 mysql /usr/sbin/mysqld
docker NOT RUNNING No matching process found

On macOS,

Process monitor running on macOS checking for the processes

JSON Output Mode

./process_monitor.sh --json nginx docker

JSON output using the json flag

Sample Output:

{
    "hostname": "web-server",
    "timestamp": "2023-07-15 14:30:45",
    "os": "Ubuntu 22.04 LTS",
    "processes": [
        {
            "process": "nginx",
            "status": "RUNNING",
            "pid": "1234",
            "user": "www-data",
            "command": "/usr/sbin/nginx -g daemon off;"
        },
        {
            "process": "docker",
            "status": "NOT RUNNING",
            "pid": null,
            "user": null,
            "command": null
        }
    ]
}

Logging to File

./process_monitor.sh --log-file "/var/log/process_monitor.log" redis postgres

Integration Possibilities

System Monitoring

Schedule regular checks with cron
Parse JSON output with monitoring tools

2. CI/CD Pipelines

Verify service status before deployments
Check for conflicting processes

3. Troubleshooting

Quick verification of running services
Check process dependencies

Performance Characteristics

Typically completes in under 100ms for basic checks
Minimal CPU and memory usage
Efficient process detection without spawning unnecessary subshells
The script is production-ready and can be immediately deployed in any Linux or macOS environment. Future enhancements could include Windows Subsystem for Linux (WSL) support or additional filtering options.

Installation:

chmod +x process_monitor.sh 
./process_monitor.sh --help

Real-World Implementation Scenarios

1. Proactive Service Health Monitoring

Problem: Web servers occasionally crash without alerting

Solution:

# Cron job checking every 2 minutes
*/2 * * * * /usr/local/bin/monitor.sh --silent nginx || systemctl restart nginx

Outcome: Automatic recovery with failure timestamps in logs

2. Security Baseline Enforcement

Problem: Need to detect unauthorized processes

Solution:

# Daily diff against approved list
diff <(./monitor.sh --json | jq .) <(jq . /etc/approved-processes.json)

Outcome: Immediate visibility into any unexpected processes

3. Containerized Environment Monitoring

Problem: Process visibility inside Kubernetes pods

Solution:

# Kubectl integration example
kubectl exec -it pod-name -- /monitor.sh --json

Outcome: Consistent monitoring across container boundaries

Advanced Integration Patterns

1. Prometheus/Grafana Dashboarding

# Metrics endpoint creation
echo "# HELP process_up Process status" > /var/lib/node_exporter/process.prom
echo "# TYPE process_up gauge" >> /var/lib/node_exporter/process.prom
./monitor.sh --json | jq -r '.processes[] | "process_up{process=\"\(.process)\"} \(.status == "RUNNING" ? 1 : 0)"' >> /var/lib/node_exporter/process.prom

Result: Beautiful dashboards showing service status across your entire fleet

2. Centralized Log Analysis

# Syslog integration example
./monitor.sh | logger -p local0.notice -t process-monitor

Result: Correlation of process events with other system logs in SIEM tools

3. CI/CD Pipeline Gates

# GitLab CI example
verify_services:
  script:
    - if ! ./monitor.sh --json postgresql | grep -q "RUNNING"; then exit 1; fi

Result: Failed deployments when dependent services aren’t available

Conclusion: Building a Monitoring Culture

Implementing this script represents more than just technical deployment — it’s about cultivating observability best practices:

Start Small Begin with critical services before expanding coverage
Establish Baselines

./monitor.sh --json > /etc/process-baseline-$(date +%F).json

3. Iterate and Expand Add checks gradually as you identify new requirements

4. Document Expectations Maintain a living document of what should be running

This approach gives you a professional-grade monitoring foundation that scales from single servers to enterprise environments while remaining lightweight and adaptable.

Podman: Detailed Overview, Advantages, Disadvantages, and Setup

Vidyasagar SC Machupalli — Sun, 04 May 2025 13:34:31 +0000

Podman is an open-source container engine that enables users to create, manage, and run OCI containers and pods across Linux, macOS, and Windows. Unlike Docker, Podman is daemonless, running containers directly under the user's control, which brings unique benefits in security and flexibility.

Advantages of Podman

Daemonless Operation

Podman does not require a central background service. Each container runs as a child process of the user, reducing resource overhead and improving security by eliminating a single point of failure.

Rootless Containers

Podman supports running containers without root privileges. This greatly reduces the risk of privilege escalation and is ideal for multi-user systems or environments with strict security requirements.

Docker Compatibility

Podman’s command-line interface is largely compatible with Docker, allowing most Docker commands and workflows to be used without modification. It can build and run containers from Dockerfiles and work with Docker images.

Pods and Kubernetes Integration

Podman natively supports pods, which are groups of containers sharing resources, similar to Kubernetes. This makes it easier to transition workloads to Kubernetes environments.

Systemd Integration

Podman can generate systemd unit files for containers and pods, making it easy to manage container lifecycles and ensure containers start automatically on system boot.

Performance

Without a central daemon, Podman typically consumes fewer resources and starts containers faster, especially in environments with many containers or limited resources.

Disadvantages of Podman

Learning Curve

Podman introduces concepts like rootless mode, custom storage, and networking configurations that may require extra learning, especially for users transitioning from Docker.

Smaller Ecosystem

Podman’s ecosystem and third-party integrations are not as extensive as Docker’s, which can make finding solutions or integrating with other tools more challenging.

Networking Limitations

Some advanced networking features are less mature or require more manual setup compared to Docker, particularly for complex routing or load balancing scenarios.

Feature Maturity

Features like orchestration and high availability are still evolving in Podman and may require additional tools or manual configuration.

Podman Installation and Setup

Linux

Debian/Ubuntu:

sudo apt update && sudo apt upgrade -y
sudo apt install -y podman
podman --version

CentOS/RHEL:

sudo yum -y install podman
podman --version

macOS

Using Homebrew:

brew install podman
podman machine init
podman machine start
podman info

Alternatively, you can download the official installer from the Podman website or use Podman Desktop for a GUI-based experience. After installation, initialize and start a Podman machine, as containers on macOS run inside a lightweight virtual machine.

Podman Desktop (GUI):

Download the .dmg file from the Podman Desktop website.
Drag the Podman Desktop icon to the Applications folder.
Launch Podman Desktop and follow the prompts to install the Podman engine if needed.

Windows

Download the Podman installer from the official Podman website or GitHub releases.
Run the installer and follow the prompts.
Podman uses Windows Subsystem for Linux (WSLv2) to run containers.
After installation, initialize and start Podman using Podman Desktop or with:

  podman machine init
  podman machine start
  podman info

You can use Podman from PowerShell, CMD, or directly within the WSL environment.

Podman Basic Configuration and Settings

Podman’s behavior can be customized using configuration files and environment variables:

Key Configuration Files:

containers.conf: Sets engine-wide defaults (runtime, network, resource limits).
storage.conf: Configures storage drivers and options.
registries.conf: Lists container registries for pulling/pushing images.
policy.json: Manages image signature verification policies.

Common Environment Variables:

CONTAINERS_CONF: Path to containers.conf.
CONTAINERS_REGISTRIES_CONF: Path to registries.conf.
CONTAINERS_STORAGE_CONF: Path to storage.conf.
CONTAINER_HOST: URL for remote Podman service (e.g., unix:///run/user/$UID/podman/podman.sock).
TMPDIR: Directory for temporary image storage.
XDG_CONFIG_HOME, XDG_DATA_HOME, XDG_RUNTIME_DIR: Control where configuration, image storage, and runtime data are stored, especially in rootless mode.

Example: Creating and Managing Pods

Create an empty pod:

podman pod create --name demo-pod

List pods:

podman pod ls

List containers in a pod:

podman ps -a --pod

Add a container to a pod:

podman run -dt --pod demo-pod nginx

Summary Table: Podman vs Docker

Feature	Podman	Docker
Architecture	Daemonless	Central daemon required
Rootless Mode	Yes	Limited
CLI Compatibility	Docker-compatible	Standard
Systemd Integration	Native	Requires workarounds
Pod/Kubernetes Support	Native pods, easy Kubernetes integration	Requires additional tools
Security	Enhanced (rootless, user auditing)	Higher attack surface
Ecosystem	Growing, less mature	Large, mature
Performance	Lightweight, fast startup	Slightly higher overhead
Tooling	Fewer third-party tools	Extensive tooling

Conclusion

Podman is a powerful alternative to Docker, offering improved security, rootless operation, and native pod support. While its ecosystem is still growing and some advanced features are maturing, it is a strong choice for developers and administrators seeking a secure, flexible, and modern container engine. Installation is straightforward on all major platforms, with both CLI and GUI options available. Configuration is highly customizable, allowing Podman to fit a wide range of use cases.

S3cmd: CLI for Object Storage

Vidyasagar SC Machupalli — Wed, 05 Mar 2025 16:32:05 +0000

In the recent times, I am exploring and playing with Top Tools for Object Storage and Data Management. In this journey, one of the tools that I was introduced to was s3cmd.

What is s3cmd?

S3cmd is a powerful open-source command-line tool designed for managing Amazon S3 and S3-compatible object storage services like IBM Cloud Object Storage. It provides a user-friendly interface for performing various operations on S3 buckets and objects, including creating and removing buckets, uploading, downloading, and deleting files.

Key features of s3cmd include:

Bucket management: Create, list, and delete buckets.
Object operations: Upload, download, and delete files within buckets.
Synchronization: Sync local directories with S3 buckets.
Access control: Modify bucket policies and access control lists.
Website hosting: Create and manage static websites hosted on S3.
Multipart uploads: Handle large file uploads efficiently.
Encryption: Support for server-side encryption.

S3cmd is compatible with Python 2.6 or newer, including Python 3.x, and runs on Linux and macOS systems. Its versatility and ease of use make it an excellent choice for both beginners and advanced users who need to interact with S3-compatible storage services.

This tool simplifies the process of managing cloud storage, allowing users to automate tasks, integrate S3 operations into scripts, and efficiently handle large amounts of data. Whether you’re a developer, system administrator, or data analyst, s3cmd can streamline your workflow when working with S3 and similar object storage platforms.

s3cmd installation and configuration

Install S3cmd following theinstructions here

s3cmd command on warp

To configure, you need to have an S3 cloud instance or any S3 compatible instance with HMAC credentials — Accesskey and Secrets key and endpoint that starts with or similar to s3://…

To understand what are HMAC credentials, refer to my article here

Once you have the HMAC credentials and endpoint, you can run the following command for an interactive configuration.

s3cmd --configure

Key commands

Here are some commonly used s3cmd commands:

Create a bucket:

  s3cmd mb s3://bucket-name

List all buckets:

  s3cmd ls

Upload a file to a bucket:

  s3cmd put file.txt s3://bucket-name/

Download a file from a bucket:

  s3cmd get s3://bucket-name/file.txt

Delete a file from a bucket:

   s3cmd rm s3://bucket-name/file.txt

List objects in a specific bucket:

   s3cmd ls s3://bucket-name/

Synchronize a local directory with a bucket:

   s3cmd sync local_directory/ s3://bucket-name/

Set public access for a file:

   s3cmd setacl s3://bucket-name/file.txt --acl-public

Delete a bucket:

   s3cmd rb s3://bucket-name

These commands allow you to perform basic operations like creating and deleting buckets, uploading and downloading files, and managing access permissions. S3cmd offers many more advanced features for managing your S3-compatible storage, making it a powerful tool for both simple and complex storage management tasks.

403 error resolution

The most common error is 403 — Access denied. Which you may think is because of the AccessKeyID or the Secrets key you provided. But actually this is a generic error.

ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you provided does not exist in our records.

One of the many reasons for this error is this configuration parameter that you either forgot to set or wrongly configured

Use “%(bucket)s.s3.amazonaws.com” to the target Amazon S3. “%(bucket)s” and “%(location)s” vars can be used

if the target S3 system supports dns based buckets.

DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:

The right way to configure this parameter is

%(bucket)s.<END_POINT>

For example:
%(bucket)s.s3.eu-de.cloud-object-storage.appdomain.cloud

Once the right of credentials, endpoint & the DNS-style is setup, you won’t see the 403 (InvalidAccesskeyId) error.

If this article is helpful. please clap 👏 and ♻️ share with your network.

An Introduction to SymPy: A Python Library for Symbolic Mathematics

Vidyasagar SC Machupalli — Mon, 06 Jan 2025 20:57:45 +0000

SymPy is an open-source Python library for symbolic mathematics.

SymPy — Mathematics Python Library

SymPy aims to become a full-featured computer algebra system (CAS) while keeping the code simple, extensible, and free of external dependencies. With SymPy, you can perform algebraic manipulations, calculus operations, linear algebra, equation solving, discrete mathematics, and much more, all symbolically, rather than numerically.

In this article, we will explore the SymPy library, its capabilities, and some practical examples that demonstrate how it can be used effectively for symbolic computation.

What is SymPy?

At its core, SymPy provides functions for performing symbolic calculations, which means it works with mathematical expressions in their exact form, rather than approximating values as floating-point numbers. SymPy leverages Python’s object-oriented nature to represent mathematical objects, such as variables, functions, and matrices, as Python objects.

SymPy is built in pure Python and does not require any external libraries or software packages. This makes it easy to install and use, and it’s suitable for applications in scientific computing, education, engineering, and mathematics.

Key Features of SymPy

Symbolic computation : Perform algebraic operations like simplification, expansion, and factorization on symbolic expressions.
Calculus : Compute derivatives, integrals, limits, series expansions, and solve differential equations symbolically.
Linear algebra : Manipulate matrices, solve linear systems, and compute determinants, eigenvalues, and eigenvectors.
Equation solving : Solve algebraic equations (linear, polynomial, non-linear) symbolically.
Plotting : Generate plots of functions and equations, both symbolic and numerical.

Installing SymPy

To install SymPy, you can use pip:

pip install sympy

After installation, you can import the library into your Python script:

import sympy as sp

Basic Usage of SymPy

Let’s start by using SymPy to define symbolic variables and perform some basic algebraic manipulations.

Defining Symbolic Variables

You can define symbolic variables using the symbols function. For example:

import sympy as sp

# Define symbols
x, y, z = sp.symbols('x y z')

# Display the symbols
print(x, y, z)

Simple Algebraic Operations

Once you’ve defined symbolic variables, you can perform various algebraic operations such as addition, multiplication, and exponentiation.

# Define an expression
expr = x**2 + 2*x + 1

# Simplify the expression
simplified_expr = sp.simplify(expr)
print("Simplified expression:", simplified_expr)

# Expand a binomial
expanded_expr = sp.expand((x + 1)**2)
print("Expanded expression:", expanded_expr)

# Factor an expression
factored_expr = sp.factor(x**2 + 2*x + 1)
print("Factored expression:", factored_expr)

Simplification and Expansion

SymPy has built-in functions for simplifying and expanding expressions:

# Simplify the expression x**2 - 2*x + 1 - (x - 1)**2
expr_to_simplify = x**2 - 2*x + 1 - (x - 1)**2
simplified = sp.simplify(expr_to_simplify)
print("Simplified expression:", simplified)

Solving Equations

SymPy makes it easy to solve equations symbolically. Here’s how to solve a linear equation:

# Solve x + 3 = 7
solution = sp.solve(x + 3 - 7, x)
print("Solution:", solution)

For solving quadratic equations, SymPy will return all possible roots:

# Solve a quadratic equation
quadratic_solution = sp.solve(x**2 - 5*x + 6, x)
print("Quadratic solution:", quadratic_solution)

Calculus Operations

SymPy excels in symbolic calculus. Let’s look at some common operations, such as differentiation and integration.

Differentiation

To compute the derivative of an expression:

# Derivative of x **3 + 2*x** 2 + x with respect to x
derivative = sp.diff(x **3 + 2*x** 2 + x, x)
print("Derivative:", derivative)

Integration

To compute the integral of an expression:

# Integral of x**2 with respect to x
integral = sp.integrate(x**2, x)
print("Integral:", integral)

Limits

You can also compute limits of expressions:

# Limit of (sin(x)/x) as x approaches 0
limit = sp.limit(sp.sin(x)/x, x, 0)
print("Limit:", limit)

Series Expansion

SymPy allows you to expand an expression as a series around a given point:

# Series expansion of exp(x) around x = 0
series_expansion = sp.series(sp.exp(x), x, 0, 5)
print("Series expansion:", series_expansion)

Solving Differential Equations

SymPy also provides tools for solving ordinary differential equations (ODEs) symbolically.

# Solve the differential equation y'' - 2*y' + y = 0
y = sp.Function('y')
ode_solution = sp.dsolve(y(x).diff(x, x) - 2*y(x).diff(x) + y(x), y(x))
print("Differential equation solution:", ode_solution)

Linear Algebra with SymPy

SymPy has robust support for linear algebra operations. Let’s see how to perform matrix operations and solve linear systems.

Matrix Operations

# Define matrices
A = sp.Matrix([[1, 2], [3, 4]])
B = sp.Matrix([[5, 6], [7, 8]])

# Matrix addition
C = A + B
print("Matrix addition:\n", C)

# Matrix multiplication
D = A * B
print("Matrix multiplication:\n", D)

# Determinant of a matrix
det_A = A.det()
print("Determinant of A:", det_A)

# Eigenvalues and eigenvectors
eigen = A.eigenvals(), A.eigenvects()
print("Eigenvalues and eigenvectors:", eigen)

Solving a Linear System

You can solve systems of linear equations symbolically using the linsolve function.

# Solve the system of equations: x + y = 2, x - y = 0
system = [x + y - 2, x - y]
solution = sp.linsolve(system, x, y)
print("Solution to the system:", solution)

Plotting with SymPy

SymPy includes basic plotting capabilities for visualizing symbolic expressions. You can use the plot function for this purpose:

from sympy.plotting import plot

# Plot the function sin(x)
plot(sp.sin(x))

Advanced Features of SymPy

SymPy has many more advanced features such as:

Mathematical functions : Special functions like gamma, Bessel, and error functions.
Solving Diophantine equations : SymPy can handle integer solutions for certain equations.
SymPy physics module : For symbolic calculations in physics, including mechanics, optics, and quantum mechanics.
Symbolic Fourier transforms : Useful for signal processing and other applications.
Code generation : Generate Python, C, Fortran, or other language code for symbolic expressions.

Applications in Various Fields

SymPy finds applications in numerous domains:

Physics : Quantum mechanics calculations and equation derivations
Engineering : Solving complex mathematical models
Computer Science : Algorithm analysis and optimization
Finance : Option pricing and risk analysis

Code Generation

One of SymPy’s powerful features is its ability to generate code in various programming languages:

from sympy.utilities.codegen import codegen 
expr = x**2 + sin(x) [(c_name, c_code), (h_name, c_header)] = codegen(('f', expr), 'C', 'file', header=False, empty=False) 
print(c_code)

This feature allows users to convert symbolic expressions into efficient, executable code for numerical computation.

The Jupyter Notebook with the outputs is available on the GitHub repository. You can run the Jupyter Notebook on Colab following the instructions on the GitHub repository.

Conclusion

SymPy is a powerful tool for symbolic mathematics in Python. It can simplify complex expressions, solve equations, perform calculus operations, and handle linear algebra tasks symbolically. Whether you are a researcher, educator, engineer, or student, SymPy provides an easy-to-use yet powerful framework for performing a wide variety of symbolic computations.

With its simple installation process, ease of use, and extensive capabilities, SymPy has become a popular choice in the Python ecosystem for symbolic mathematics. By using the examples in this article as a starting point, you can begin to explore the full range of features SymPy has to offer.

Originally published at https://vmacwrites.substack.com.

Pandas: Conversion using loc and iloc

Vidyasagar SC Machupalli — Wed, 01 Jan 2025 22:00:39 +0000

Pandas is a powerful Python library used for data manipulation and analysis.

Created by Wes McKinney in 2008, it provides data structures and functions for working with structured data efficiently. Pandas allows users to analyze big data, clean messy datasets, and derive meaningful insights.

Photo by Kevin Canlas on Unsplash

Key Features of Pandas

Data Structures: Pandas introduces two primary data structures:

Series: A one-dimensional labeled array
DataFrame: A two-dimensional labeled data structure with columns of potentially different types

Data Manipulation: Pandas offers functions for analyzing, cleaning, exploring, and manipulating data.
Data Analysis: It enables users to perform complex operations like correlation analysis, grouping, and statistical calculations.
Data Visualization: Pandas integrates well with other libraries to create insightful visualizations.

Practical Examples

Indexing with loc

The loc function enables label-based indexing in DataFrames, allowing precise data selection:

import pandas as pd

# Create a sample DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}, index=['x', 'y', 'z'])

# Select rows with label 'y' and 'z', and columns 'A' and 'C'
result = df.loc[['y', 'z'], ['A', 'C']]
print(result)

The iloc function provides integer-based indexing for DataFrame selection:

import pandas as pd

# Create a sample DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]})

# Select rows 0 and 2, and columns 1 and 2
result = df.iloc[[0, 2], [1, 2]]
print(result)

Date Conversion with to_datetime

The to_datetimefunction transforms various date formats into standardized datetime objects:

import pandas as pd

# Convert string to datetime
date_string = "2023-09-17 14:30:00"
dt_object = pd.to_datetime(date_string)
print(dt_object)

# Convert multiple date strings
date_series = pd.Series(['20200101', '20200201', '20200301'])
dt_series = pd.to_datetime(date_series, format='%Y%m%d')
print(dt_series)

Output:

2023-09-17 14:30:00
0 2020-01-01
1 2020-02-01
2 2020-03-01
dtype: datetime64[ns]

Pandas simplifies data manipulation tasks, making it an essential tool for data scientists and analysts. Its versatile functions like loc, iloc, and to_datetime provide powerful ways to interact with and transform data, enabling efficient data processing and analysis in Python.

Something to consider while using loc or iloc

Let’s convert the object column date to datetime using loc

import pandas as pd

df = pd.DataFrame({'date': ['2023-01-01', '2023-02-15', '2023-03-31']})

df.loc[:, 'date'] = pd.to_datetime(df.loc[:, 'date'])

print(df)
print(df.dtypes)

Output: 

   date
0 2023-01-01 00:00:00
1 2023-02-15 00:00:00
2 2023-03-31 00:00:00
date object
dtype: object

If you observe, the dtype is object not datetime64[ns]. If you try to extract the date using df['date'].dt.date

You will see an error as the conversion was not successful.

Traceback (most recent call last):
  File "/HelloWorld.py", line 11, in <module>
    print(df.dt.date)
          ^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pandas/core/generic.py", line 6299, in __getattr__
    return object. __getattribute__ (self, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'dt'. Did you mean: 'at'?

The reason lies in the changes made in version 2.x.x of Pandas.

From What’s new in 2.0.0 (April 3, 2023):

Changed behavior in setting values with df.loc[:, foo] = bar or df.iloc[:, foo] = bar, these now always attempt to set values inplace before falling back to casting (GH 45333)

How to overcome:

The best way to address this issue is to either avoid using loc or iloc or as suggested on the Pandas documentation use DataFrame.__setitem__()

df = pd.DataFrame({'date': ['2023-01-01', '2023-02-15', '2023-03-31']})

df['date'] = pd.to_datetime(df.loc[:, 'date'])

print(df)
print(df.dtypes)

print(df['date'].dt.date)

Additional read:

https://stackoverflow.com/questions/76766136/pandas-pd-to-datetime-assigns-object-dtype-instead-of-datetime64ns

Reflecting on My 2024 Journey: Achievements and Growth

Vidyasagar SC Machupalli — Sun, 08 Dec 2024 17:13:01 +0000

As I think on 2024, I’m overwhelmed with gratitude and excitement for the incredible journey I’ve experienced. This year has been a remarkable chapter in my professional and personal growth. I have achieved significant milestones. I have received heartwarming support from my incredible network.

Professional Milestones

This year brought some extraordinary recognition that I’m truly humbled by:

The Most Admired Global Indian 2024 – https://www.passionvista.com/sarath-chandra-vidya-sagar-machupalli/
BCS Fellowship acknowledging my contributions to the IT industry in June 2024
Distinguished Architect Certification from The Open Group in March 2024
Member of Association of Enterprise Architects AEA

My Passion and Purpose

At IBM, I’ve continued to dive deep into my core passions:

Quantum computing
Data science
Machine learning
Cloud technologies
Solution architecture

Reflecting on my recent experiences, I had the incredible opportunity to attend the Quantum Ideas Summer School. This immersive event featured sessions led by renowned quantum professors and researchers, focusing on the fundamentals of quantum computation and its potential applications. Engaging with experts in the field not only deepened my understanding of quantum technologies but also inspired me to explore innovative solutions that harness these advancements.

Throughout my journey, I have continuously sought to expand my skill set, particularly in emerging technologies. My commitment to lifelong learning has led me to earn several certifications that enhance my expertise. Notably, I completed a Practical Introduction to Quantum-Safe Cryptography certification from IBM in April 2024, which has equipped me with valuable insights into the intersection of quantum computing and cybersecurity.

In addition to my quantum-focused skills, I have developed a comprehensive portfolio of technical competencies:

AI and Machine Learning : I hold the “watsonx.ai Data Science and MLOps Sales Foundation” certification from IBM, which has broadened my understanding of machine learning models and their deployment.
Cloud Technologies : My experience with IBM Cloud continues to grow, allowing me to design and implement cloud-native solutions effectively.
Software Development : With over 17 years in software development, I am proficient in multiple programming languages and frameworks, enabling me to tackle diverse projects.
Solution Architecture : As a Senior Certified Solutions Architect at IBM, I focus on creating scalable and efficient architectures that meet client needs.

These skills not only empower me to contribute effectively to my current role but also position me as a thought leader in the tech community. I remain dedicated to sharing knowledge through speaking engagements, writing articles, and participating in open-source projects. My goal is to inspire others while driving digital transformation through innovative technology solutions.

My journey has always been about more than just technical expertise. It’s about creating innovative solutions that bridge technology and human potential.

Community Engagement

In 2024, I had the privilege of serving as a judge for multiple prestigious awards, including:

Globee® Awards for Technology in May 2024
Globee® Awards for Customer Excellence in May 2024
Golden Bridge Awards® in June 2024

I continued to engage with the community. I served as a core member and organizer of the BlueCoders Meetup Community. I fostered collaboration and innovation among peers.

I’m excited to share some of the technical articles I published in 2024:

“Latest Data Architectures, Tools, and Technologies: Emerging Trends” (October 23, 2024)
“Nurturing the Developer Within: A Journey of Growth and Community” (December 1, 2024)

These articles show my ongoing commitment to sharing knowledge and insights with the tech community. I’m particularly proud of reaching over 60,000 total views on my Dev.to posts, a milestone that underscores the value of community engagement and knowledge sharing.

Throughout the year, I continued to contribute to DZone as an expert, with my articles accumulating 431.9K pageviews (as of 1st Jan, 2025) across 72 publications. This platform has been an excellent avenue for discussing cloud technologies, AI, and various other tech topics.

I’m grateful for the opportunity to contribute to the tech community through these publications. I look forward to sharing more insights in the coming year.

— VidyaSagar Machupalli (@vidyasagarmsc.bsky.social) 2024-12-07T17:17:51.550Z

Personal Reflection

As a multi-faceted professional – an architect, speaker, writer, and photographer – I’ve been fortunate to explore technologies that can truly transform our world. My goal remains consistent: to develop solutions that solve real-world problems, mentor emerging talents, and contribute to open-source projects.

Social Media Highlights

Throughout the year, I shared insights and updates on my social media platforms – https://vidyasagarmsc.github.io/#connect

A Heartfelt Thank You

To my mentors, colleagues, friends, and family:

Your unwavering support has been my greatest strength. Each challenge, each opportunity has been a stepping stone, and I’m grateful for every moment of this incredible journey.

To the tech community that has embraced me:

Your collaboration, feedback, and shared passion continue to inspire me. Together, we’re not just developing technology – we’re shaping the future.

Here’s to continuous learning, innovation, and creating a better tomorrow!

Blogs

Nurturing the Developer Within: A Journey of Growth and Community

Vidyasagar SC Machupalli — Sun, 01 Dec 2024 17:01:49 +0000

In the ever-evolving landscape of technology, keeping the developer inside you alive
is not just a choice—it's a necessity. As we navigate through our careers, it's easy
to get caught up in the day-to-day tasks and forget the essence of what makes us
developers: our curiosity, our drive to solve problems, and our passion for creating.

The Essence of the Inner Developer

At its core, the developer spirit is about more than just writing code. It's about:

Continuous Learning : Embracing new technologies and methodologies.
Problem-Solving : Approaching challenges with creativity and persistence.
Innovation : Thinking outside the box to create novel solutions.
Collaboration : Working with others to achieve greater outcomes.

Why Keeping Your Developer Spirit Alive Matters

Maintaining your inner developer is crucial for several reasons:

Career Longevity : The tech industry moves fast. Staying current ensures you remain valuable.
Personal Growth : Challenging yourself leads to new skills and perspectives.
Job Satisfaction : Engaging with new ideas keeps your work exciting and fulfilling.
Leadership Opportunities : A developer mindset is invaluable in guiding teams and projects.

The Power of Community and Sharing

One of the most effective ways to keep your developer spirit thriving is by engaging with the community and sharing your knowledge. This is where platforms like Dev.to come into play.

My Dev.to Journey: A Celebration of 60K+ Views

I'm incredibly excited and grateful to announce that my posts on Dev.to have surpassed 60,000 total views! This milestone is not just a personal achievement; it's a testament to the power of community engagement and knowledge sharing.

A heartfelt thank you to everyone who has read, commented, shared, and engaged with my content. Your support and interaction have been invaluable in this journey.

Why Sharing Matters

Reinforces Learning : Explaining concepts to others deepens your own understanding.
Builds Your Brand : Consistent sharing establishes you as a thought leader.
Helps Others : Your insights might be exactly what another developer needs.
Creates Opportunities : Sharing can lead to collaborations, job offers, and speaking engagements.
Gives Back : Contributing to the community that has helped you grow.

Strategies to Keep Your Inner Developer Alive

Set Personal Coding Challenges : Take on projects outside of work that excite you.
Attend Tech Meetups and Conferences : Connect with like-minded individuals and learn about cutting-edge technologies.
Contribute to Open Source : Collaborate with developers worldwide on projects that matter.
Read and Write Technical Blogs : Stay informed and share your own experiences.
Experiment with New Technologies : Don't be afraid to play with new tools and frameworks.
Mentor Others : Teaching is one of the best ways to reinforce your own knowledge.

The Impact of Community Engagement

Engaging with the developer community, whether through platforms like Dev.to, GitHub, or local meetups, has numerous benefits:

Exposure to Diverse Perspectives : Learn how others approach problems.
Networking : Build relationships that can lead to collaborations or career opportunities.
Motivation : Seeing others' achievements can inspire you to push further.
Support : Find help when you're stuck on a problem.

Looking Forward

As we celebrate milestones like reaching 60K+ views, it's important to remember that the journey of a developer is ongoing. Each article written, each problem solved, and each interaction with the community is a step forward in our collective growth.

I invite you to join me in this journey of continuous learning and sharing. Check out my articles on Dev.to, where I cover topics ranging from cloud technologies to DevOps practices, and let's continue to grow together.

Final Thoughts

Keeping the developer inside you alive is about maintaining that spark of curiosity and the drive to create. It's about seeing every challenge as an opportunity to learn and grow. By engaging with the community, sharing our knowledge, and staying open to new ideas, we not only enhance our own careers but also contribute to the advancement of our field as a whole.

Thank you once again for being part of this incredible journey. Here's to many more milestones, breakthroughs, and shared learnings in the future!

IceCream: A Sweet Alternative to Print Debugging in Python

Vidyasagar SC Machupalli — Wed, 20 Nov 2024 22:57:03 +0000

Tired of cluttering your code with print statements for debugging? Enter IceCream, a Python library that makes debugging effortless and more readable. Let's explore how IceCream can sweeten your debugging experience.

Installation

First, install IceCream using pip:

pip install icecream

Basic Usage

To use IceCream, import the ic function:

from icecream import ic

Now, let's compare traditional print debugging with IceCream:

# Traditional print debugging
x: int = 5
y: int = 10
print("x:", x)
print("y:", y)
print("x + y:", x + y)


# Using IceCream
ic(x)
ic(y)
ic(x + y)

Output:

x: 5
y: 10
x + y: 15

ic| x: 5
ic| y: 10
ic| x + y: 15

As you can see, IceCream automatically prints both the variable names and their values, making the output more informative and easier to read.

Debugging Functions

IceCream really shines when debugging functions:

def square(num: int) -> int:
    return num * num

# Traditional print debugging
print("square(4):", square(4))

# Using IceCream
ic(square(4))

Output:

square(4): 16

ic| square(4): 16

IceCream displays the function call along with its result, providing more context.

Pretty-Printing Data Structures

IceCream formats complex data structures for better readability:

data: dict = {"name": "Alice", "age": 30, "scores": [85, 90, 92]}

# Traditional print debugging
print("data:", data)

# Using IceCream
ic(data)

Output:

data: {'name': 'Alice', 'age': 30, 'scores': [85, 90, 92]}

ic| data: {
    'name': 'Alice',
    'age': 30,
    'scores': [85, 90, 92]
}

The IceCream output is much easier to read, especially for nested structures.

Including Context

IceCream can optionally include file, line number, and function context:

ic.configureOutput(includeContext=True)

def example_function():
    x = 42
    ic(x)

example_function()

Output:

ic| example.py:3 in example_function()- x: 42

This feature is invaluable when debugging larger codebases.

Conclusion

IceCream offers a more efficient and readable alternative to traditional print debugging. By automatically including variable names, formatting complex structures, and optionally providing context, IceCream can significantly speed up your debugging process. Give it a try in your next Python project and experience the difference for yourself!

Chain of Trust: Decoding SSL Certificate Security Architecture

Vidyasagar SC Machupalli — Mon, 18 Nov 2024 14:01:50 +0000

The chain of trust is a fundamental concept in digital security that underpins the reliability and authenticity of SSL/TLS certificates used to secure online communications. It establishes a hierarchical structure of trust, starting from a trusted root Certificate Authority (CA) and extending down to the end-entity certificate used by a website or service.

This chain is important for several reasons. First of all, it allows for the scalable distribution of trust across the internet. Instead of requiring every device to trust millions of individual certificates, they only need to trust a small number of root CAs. Second, it provides a mechanism for verifying the authenticity of certificates. When a browser encounters a certificate, it can trace the chain back to a trusted root, ensuring that each link in the chain is valid and trustworthy. This process helps prevent man-in-the-middle attacks and other forms of certificate fraud. Additionally, the chain of trust enables the revocation of compromised certificates without undermining the entire system. If an intermediate CA is compromised, only the certificates issued by that CA need to be revoked, rather than all certificates issued by the root CA. This flexibility and security make the chain of trust an essential component of modern internet security.

Understanding the Chain of Trust

The chain of trust is a hierarchical structure of digital certificates that validates the authenticity of a certificate. This chain typically consists of three main components:

Root Certificate Authority (Root CA)
Intermediate Certificate Authority (Intermediate CA)
End-entity Certificate (Server or Client Certificate)

Root Certificate Authority (Root CA)

The Root CA sits at the top of the trust hierarchy. It is a self-signed certificate, meaning it vouches for its own authenticity. Root CAs are implicitly trusted by web browsers and operating systems. They are responsible for issuing certificates to Intermediate CAs and, in some cases, directly to end-entities.

Key characteristics of Root CAs:

Self-signed certificates
Stored securely offline to prevent compromise
Long validity periods (typically 20–30 years)
Highly trusted and recognized globally

Intermediate Certificate Authority (Intermediate CA)

Intermediate CAs bridge the gap between Root CAs and end-entity certificates. They are issued by Root CAs and, in turn, issue certificates to end-entities or other Intermediate CAs. The use of Intermediate CAs enhances security by allowing the Root CA to remain offline while the Intermediate CA handles day-to-day certificate issuance.

Key characteristics of Intermediate CAs:

Issued by Root CAs
Can issue certificates to end-entities or other Intermediate CAs
Shorter validity periods than Root CAs (typically 10–15 years)
Enhance security and flexibility in certificate management

End-entity Certificate

This is the certificate issued to a specific entity, such as a website or email server. It’s the certificate that end-users interact with when visiting a secure website or sending encrypted email.

Key characteristics of End-entity Certificates:

Issued by Intermediate CAs or, occasionally, Root CAs
Shortest validity period (typically 1–2 years)
Contains information about the entity it represents (e.g., domain name for websites)

SSL Chain of trust generated using Eraser

Role in Security

The chain of trust plays a crucial role in ensuring the security and integrity of online communications:

Authentication : It verifies the identity of the certificate holder, ensuring that you’re communicating with the intended party.
Integrity : It ensures that the certificate hasn’t been tampered with or altered in transit.
Encryption : While not directly responsible for encryption, a valid certificate chain is necessary for establishing encrypted connections.
Revocation : If a certificate in the chain is compromised, it can be revoked, invalidating all certificates below it in the chain.

Tools to Check Chain of Trust and Certificate Validity

Several tools are available to verify the chain of trust and validate certificates:

OpenSSL

OpenSSL is a versatile command-line tool that can perform various SSL/TLS-related tasks.

To check a certificate chain:

openssl s_client -connect example.com:443 -showcerts

This command will display the entire certificate chain and highlight any issues.

Certificate Checker Tools

Many online services provide comprehensive certificate checking capabilities:

SSL Labs Server Test : Offers in-depth analysis of SSL/TLS configurations.
DigiCert Certificate Inspector : Scans for certificate issues across your entire network.
Qualys SSL Server Test : Provides detailed information about SSL/TLS configuration and potential vulnerabilities.

Browser Developer Tools

Modern web browsers include developer tools that can display certificate information:

Click on the padlock icon in the address bar.
Select “Certificate” or a similar option.
Review the certificate details, including the chain of trust.

Certificate Validation Process

When a client connects to a server, it performs several checks to validate the certificate:

Signature Verification : The client verifies each certificate’s digital signature in the chain, starting from the end-entity certificate up to the Root CA.
Trust Anchor Check : The client checks if the Root CA is in its list of trusted root certificates.
Validity Period : The client ensures all certificates in the chain are within their validity periods.
Revocation Check : The client may check if any certificates in the chain have been revoked using Certificate Revocation Lists (CRLs) or Online Certificate Status Protocol (OCSP).
Name Matching : For the end-entity certificate, the client verifies that the certificate’s Subject Alternative Name (SAN) or Common Name (CN) matches the domain being accessed.

Common Issues in Certificate Chain Validation

Incomplete Chain : The server doesn’t provide all necessary intermediate certificates.
Expired Certificates : One or more certificates in the chain have expired.
Revoked Certificates : A certificate in the chain has been revoked.
Name Mismatch : The server’s certificate doesn’t match the domain name being accessed.
Untrusted Root : The Root CA isn’t in the client’s list of trusted roots.

The testssl.sh tool is a powerful, open-source command-line utility for testing SSL/TLS enabled servers. It provides comprehensive information about a server’s SSL/TLS configuration, including supported protocols, ciphers, and certificate details. This tool is particularly useful for administrators and security professionals looking to assess the security of their SSL/TLS implementations.

./testssl.sh --jsonfile-pretty SSL_test.json https://example.com:443/

While running ./testssl.sh, if you see an error Chain of trust — NOT ok , you need to check whether intermediate certificates are supplied and are part of the chain of trust.

If you are supplying all the certificates including the intermediate certificate but still seeing the error, supply the certificates to the testssh using the add-ca flag pointing to the folder with all the certificates in the chain of trust

./testssl.sh --add-ca ~/Downloads/amc-private-certificate --jsonfile-pretty SSL_test.json https://example.com:443/

The result should now be

{
    "id": "cert_chain_of_trust",
    "severity": "OK",
    "finding": "passed."
}

Conclusion

Understanding the chain of trust and regularly validating certificates is crucial for maintaining a secure online environment. By utilizing the tools and knowledge discussed, you can ensure that your digital certificates are properly configured and trusted, thereby safeguarding your online communications and transactions.

Git push: fatal: the remote end hung up unexpectedly

Vidyasagar SC Machupalli — Tue, 10 Sep 2024 04:26:05 +0000

Everthing was working as expected when I closed my laptop yesterday and today morning when I try running mkdocs gh-deploy. I see the below error

Error

Enumerating objects: 101, done.
Counting objects: 100% (101/101), done.
Delta compression using up to 16 threads
Compressing objects: 100% (32/32), done.
send-pack: unexpected disconnect while reading sideband packet
Writing objects: 100% (54/54), 1.30 MiB | 1.73 MiB/s, done.
Total 54 (delta 23), reused 0 (delta 0), pack-reused 0
fatal: the remote end hung up unexpectedly
Everything up-to-date
Traceback (most recent call last):
  File "/Users/vmac/.pyenv/versions/3.9.13/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/mkdocs/__main__.py", line 318, in gh_deploy_command
    gh_deploy.gh_deploy(
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/mkdocs/commands/gh_deploy.py", line 129, in gh_deploy
    ghp_import.ghp_import(
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/ghp_import.py", line 285, in ghp_import
    git.check_call('push', opts['remote'], opts['branch'])
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/site-packages/ghp_import.py", line 119, in check_call
    sp.check_call(['git'] + list(args), **kwargs)
  File "/Users/vmac/.pyenv/versions/3.9.13/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'push', 'origin', 'gh-pages']' returned non-zero exit status 1.

What is this error?

The error message "fatal: The remote end hung up unexpectedly" typically occurs during a Git operation when the server unexpectedly terminates the connection. This issue is often related to the size of the data being pushed or network-related problems. Here are some common causes and solutions:

Common Causes

Large File Size: The most frequent reason for this error is that the size of the files being pushed exceeds the server's configured limits. This can happen if the repository contains large files or if you're trying to push a large number of changes at once.
Network Issues: High-latency networks or unstable connections can lead to this error, especially if the connection drops during the data transfer.
Proxy Server Configuration: If you're behind a proxy, it may not support the chunked transfer encoding used by Git, leading to connection issues.

Solutions

Increase the Git Buffer Size: You can increase the buffer size to accommodate larger files. Run the following commands in your terminal:

For HTTP:

   git config --global http.postBuffer 524288000

For SSH:

   git config --global ssh.postBuffer 524288000

You may adjust the buffer size up to 2 GB if necessary.

Check Proxy Settings: If you're using a proxy, try bypassing it or adjusting the proxy settings. You can push directly to the server's IP address and port to see if that resolves the issue.
Reduce the Size of the Push: If possible, try breaking down your push into smaller commits. This can help avoid exceeding the server limits.
Network Stability: Ensure your internet connection is stable. If you're on a high-latency network, consider using a more reliable connection.
Server Configuration: If you have control over the server, check the server's configuration to ensure it allows for larger file uploads. This may involve adjusting settings like client_max_body_size in Nginx or similar configurations in other web servers.

By following these steps, you should be able to resolve the "fatal: The remote end hung up unexpectedly" error and successfully push your changes to the remote repository.

Additional configurations

If setting the http.postBuffer configuration alone doesn't work, try the additional commands

git config http.postBuffer 524288000
git config http.lowSpeedTime 600
git config --global pack.window 1
git config --global core.compression 0

http.postBuffer

Maximum size in bytes of the buffer used by smart HTTP transports when POSTing data to the remote system. For requests larger than this buffer size, HTTP/1.1 and Transfer-Encoding: chunked is used to avoid creating a massive pack file locally. Default is 1 MiB, which is sufficient for most requests.

For other git-config options check the documentation here

Forem: Vidyasagar SC Machupalli

Finding Today's Changed Files: A Quick Python Script for File Uploads

The Problem

The Solution

Real-World Usage

Beyond File Uploads

Simple Tools, Big Impact

Resolving Secrets Manager DNS Resolution Errors in Terraform and Pulumi IaC

Understanding Infrastructure as Code (IaC)

Terraform vs Pulumi: A Brief Overview

Introduction to Pulumi

What is Pulumi?

Key Advantages of Pulumi over Traditional IaC Tools

Pulumi’s Multi-Language Support

Setting Up IBM Cloud with Pulumi

Installing the IBM Cloud Provider for Pulumi

Understanding the Service Endpoint Configuration Error

Error Analysis from GitHub Issue

Analyzing the Incorrect Configuration

The Correct Configuration Approach

Conclusion

Related Documentation

The Essential Role of Process Monitoring Scripts in Linux Environments

Understanding the Linux Distribution Landscape

1. Enterprise-Grade Workhorses

2. Rolling Release Distributions

3. Specialized and Lightweight Systems

Introducing the Universal Process Monitor

Key Features

Usage Examples

Integration Possibilities

Performance Characteristics

Installation:

Real-World Implementation Scenarios

1. Proactive Service Health Monitoring

2. Security Baseline Enforcement

3. Containerized Environment Monitoring

Advanced Integration Patterns

1. Prometheus/Grafana Dashboarding

2. Centralized Log Analysis

3. CI/CD Pipeline Gates

Conclusion: Building a Monitoring Culture

Podman: Detailed Overview, Advantages, Disadvantages, and Setup

Advantages of Podman

Disadvantages of Podman

Podman Installation and Setup

Linux

macOS

Windows

Podman Basic Configuration and Settings

Example: Creating and Managing Pods

Summary Table: Podman vs Docker

Conclusion

S3cmd: CLI for Object Storage

What is s3cmd?

Key features of s3cmd include:

s3cmd installation and configuration

Key commands

403 error resolution

An Introduction to SymPy: A Python Library for Symbolic Mathematics

SymPy is an open-source Python library for symbolic mathematics.

What is SymPy?

Key Features of SymPy

Installing SymPy

Basic Usage of SymPy

Defining Symbolic Variables

Simple Algebraic Operations

Simplification and Expansion

Solving Equations

Calculus Operations

Differentiation

Integration

Limits

Series Expansion

Solving Differential Equations

Linear Algebra with SymPy

Matrix Operations

Solving a Linear System

Plotting with SymPy

Advanced Features of SymPy