Forem: Alan Varghese

Build a Lightweight File Integrity Monitor with Bash, SQLite, and Docker

Alan Varghese — Tue, 24 Mar 2026 10:57:51 +0000

In the world of server security, File Integrity Monitoring (FIM) is a critical layer of defense. It's the "silent alarm" that tells you when a configuration file, a system binary, or a sensitive database has been tampered with.

While there are enterprise grade tools like Tripwire or OSSEC, sometimes you need something lightweight, transparent, and easy to deploy.

In this post, I’ll walk you through a project I built: a Bash based File Integrity Checker that uses SQLite for baseline storage and Docker for a fully isolated testing environment.

🚀 The Core Concept: Baseline vs. Reality

The tool works on a simple but powerful principle:

Initialize (--init): Scan your critical files, calculate their SHA-256 hashes, and store them in a persistent SQLite database. This is your "known-good" state.
Check (--check): Periodically re-scan those same files. If a single bit has changed, the hashes won't match, and an alert is triggered.

🛠️ The Tech Stack

Bash: The engine. It handles the file traversal, hashing logic, and alerting.
SQLite3: Instead of messy text files, I used SQLite to store the baseline. It’s fast, structured, and handles hundreds of files with ease.
Docker & Docker Compose: To make testing easy, I containerized the entire app.
MailHog: A "superpower" for development. It's a mock SMTP server that catches all outgoing alert emails so you can verify them in a web UI without spamming your real inbox.

🔍 A Deep Dive into the Code

1. The Hashing Logic

We use sha256sum to ensure high cryptographic security. Even a tiny change to a file (like adding a space) results in a completely different hash.

# Calculate hash and store in database
current_hash=$(sha256sum "$file_path" | awk '{print $1}')
sqlite3 "$DB_PATH" "INSERT INTO file_hashes (file_path, hash) VALUES ('$file_path', '$current_hash');"

2. Multi Layered Alerting

One of the biggest challenges in shell scripting is ensuring email delivery. My script uses a "fallback" strategy:

Primary: ssmtp (sendmail) via the local system.
Secondary: curl to talk directly to the MailHog API.
Last Resort: Standard mail command.

🐳 The Docker Testing Workflow

Testing a security tool can be nerve wracking you don't want to accidentally modify your host's /etc/ files! That’s where Docker shines.

I created a docker-compose.yml that spins up:

An Ubuntu-based checker container with mock system files.
A MailHog container for email visualization.

How to test a "Malicious" change:

Start the environment:

   docker-compose up -d

Create the baseline:

   docker exec -it file_integrity_checker bash /app/file_integrity_checker.sh --init

Simulate a hack:

   docker exec -it file_integrity_checker bash -c "echo 'MALICIOUS_CODE' >> /etc/myapp/config/app.yaml"

Run the check:

   docker exec -it file_integrity_checker bash /app/file_integrity_checker.sh --check

See the alert: Open http://localhost:8025 and watch the security alert arrive in your MailHog inbox!

💡 Key Lessons Learned

Volume Mounts & Permissions: When mounting files from macOS to a Linux container, file permissions (like the execute bit) can be tricky. I learned that executing via bash <script> is often more robust than relying on the +x bit in a shared volume.
Architecture Matters: Adding platform: linux/amd64 to the docker-compose.yml was essential for ensuring the MailHog image (which is AMD64 only) runs smoothly on modern ARM64 chips like the Apple M1/M2.

🏁 Conclusion

Building your own security tools is one of the best ways to understand how systems work. This project taught me about hashing, database persistence in shell, and the power of Docker for creating reproducible security labs.

What's next for this project? I'm looking into adding Slack/Discord webhook support and real time monitoring via inotify.

Thanks for reading! If you have questions about Bash security or Docker setups, let's chat in the comments! 🛡️

Monitoring Remote Network Health: Building a Lightweight Connectivity Tool with Bash and Docker

Alan Varghese — Mon, 09 Mar 2026 12:26:06 +0000

Have you ever needed to verify if your remote servers have outbound internet access? Whether you're managing a cluster of web servers or a set of edge devices, ensuring they can "talk to the world" is a fundamental task.

In this post, I'll walk you through a lightweight Network Connectivity Monitoring Tool I built using Bash. It's simple, portable, and comes with a Docker-based test environment to get you started safely.

The Problem

Manually SSH-ing into dozens of servers to run a ping command is tedious and error-prone. I needed a way to:

Check connectivity from multiple servers at once.
Support non-standard SSH ports.
Log results for historical tracking.
Get a quick summary of which hosts are "Reachable" vs "Unreachable."

The Solution: A Bash-Powered Monitor

The core of this tool is a Bash script that uses SSH to execute a ping command on remote targets. Here's a look at the key features and how it works.

1. Robust Scripting with `set -euo pipefail`

To make the script reliable, I used strict error handling:

-e: Exit immediately if a command fails.
-u: Treat unset variables as errors.
-o pipefail: Ensure that if any part of a pipeline fails, the whole pipeline fails.

2. Intelligent Connectivity Checks

The script doesn't just check if the server is "up"; it distinguishes between an SSH failure (can't connect to the box) and a Network failure (connected to the box, but the box can't reach the internet).

# A snippet of the core logic
if ssh -o StrictHostKeyChecking=no -o BatchMode=yes -p "${port}" "${ssh_target}" \
       "ping -c 3 8.8.8.8 > /dev/null 2>&1"; then
    log_result "$host" "REACHABLE" "$TIMESTAMP"
    echo "  ✓ $host - REACHABLE"
else
    log_result "$host" "UNREACHABLE" "$TIMESTAMP"
    echo "  ✗ $host - UNREACHABLE (network issue)"
fi

3. Automated Logging & Summaries

Every run generates a timestamped log in connectivity_log.txt, making it easy to spot patterns over time. At the end of each run, you get a clean summary:

========================================
  Summary
========================================
  Total hosts checked: 3
  Reachable:          2
  Unreachable:        1
========================================

Testing Safely with Docker

One of the best parts of this project is the included Test Lab. Using docker-compose, you can spin up three Ubuntu-based SSH servers locally to test the script without risking your production environment.

I've included a setup.sh script that:

Starts the containers.
Installs ping (not present in minimal Ubuntu images).
Configures SSH keys for passwordless login.
Validates the environment.

How to Try It Out

Clone the project.
Generate an SSH key if you haven't already: ssh-keygen -t rsa.
Launch the test lab:
```
./setup.sh
```
Run the monitor:
```
./monitor_connectivity.sh
```

Lessons Learned

While building this, I ran into a few classic Bash "gotchas":

Loops vs set -e: Using a while read loop with set -e can cause the script to exit prematurely when it hits the end of a file. Switching to a for loop with grep solved this.
Docker Ephemerality: Since containers reset on restart, I automated the SSH key injection into the setup.sh script to ensure a smooth developer experience.

Conclusion

Sometimes, the simplest tools are the most effective. This Bash script provides a no-nonsense way to monitor network health across your infrastructure.

Check out the full source code here! github

What are your favorite "mini-tools" for server management? Let me know in the comments!

How I Built a Lightweight Cron Job Health Monitor with Bash and Docker

Alan Varghese — Sun, 08 Mar 2026 18:16:58 +0000

Tired of silent cron failures? Here's a lightweight Bash-based solution to monitor and alert on your scheduled tasks across multiple servers.

Why Your Cron Jobs Need Monitoring

We’ve all been there. You set up a "mission-critical" backup or data sync as a cron job, and then you forget about it. Six months later, you realize it hasn't run in weeks because of a silent failure, a disk space issue, or an SSH key change.

Cron is great for execution, but it's terrible at visibility.

In this post, I'll walk through a lightweight, Bash-powered monitoring solution I built to keep tabs on cron jobs across multiple servers without needing a heavy agent like Zabbix or Datadog.

🛠 The Architecture

The goal was simple:

Discover jobs automatically from remote servers via SSH.
Monitor their last execution time.
Alert via Slack or Email if a job is "missed" or overdue.
Test everything locally using Docker.

┌───────────────────────────────┐
│        Monitoring Host        │
│  (Bash + cron_health_monitor) │
└───────────────┬───────────────┘
                │
        ┌───────┴───────┐
        ▼               ▼
  ┌──────────┐    ┌──────────┐
  │ Server A │    │ Server B │
  │ (Docker) │    │ (Docker) │
  └──────────┘    └──────────┘

🚀 Key Features

1. Automated SSH Discovery

The script scans crontab -l on remote servers, parses the schedules, and automatically adds them to its tracking list. No more manual entry for every single task.

2. State-Based Tracking

Instead of checking logs (which can be messy), the monitor looks at "last run" timestamps. Jobs can report their own completion via a simple CLI command:

./cron_health_monitor.sh record backup_job server1

3. Dockerized Testing Environment

To ensure the monitor works before deploying it to production, I included a docker-compose.yml that spins up three Ubuntu servers. This allows you to simulate real-world cron failures in a safe sandbox.

🧩 The "Aha!" Moments (and Bugs)

While building this, I ran into a few classic engineering hurdles:

The Date Dilemma: I initially used BSD-style date commands (macOS default), which broke completely on the Linux target servers. I had to switch to the more universal GNU date -d syntax.
Docker Networking: On macOS, connecting to localhost inside a container can be tricky. Switching to 127.0.0.1 fixed several "Host not found" errors.
Silent Failures: I learned that if the SSH connection fails, the script should alert on the connection failure, not just the missing cron job.

🛠 Quick Start

If you want to try it out:

Clone & Setup:

   ./setup.sh # Generates keys & starts Docker test servers

Discover Jobs:

   ./cron_health_monitor.sh discover

Run Health Check:

   ./cron_health_monitor.sh check

💡 Lessons Learned

Bash is incredibly powerful for infrastructure glue code. By combining standard tools like ssh, awk, and grep, you can build a monitoring system that is:

Zero-agent: Nothing to install on target servers.
Low-overhead: Runs in milliseconds.
Portable: Works on almost any Linux distro.

Check out the full source code and my "bug log" in the repository!

https://github.com/alanvarghese-dev/Bash_Scripting/tree/main/cron_job_health_monitor

What are you using to monitor your legacy cron jobs? Let's discuss in the comments!

Stop SSH-ing One by One: Building a Parallel Command Executor in Bash

Alan Varghese — Thu, 05 Mar 2026 20:29:19 +0000

Learn how to build a robust, multi-server SSH command runner using Bash, Docker, and parallel processing.

As developers or system administrators, we've all been there: You need to check the disk space, uptime, or service status on 10 different servers.

The "manual" way is painful:

ssh user@server1 -> df -h -> exit
ssh user@server2 -> df -h -> exit
...repeat 8 more times. 😫

Sure, tools like Ansible exist, but sometimes you just want a lightweight, zero-dependency script to fire off a quick command and see what's happening right now.

In this post, I'll walk you through how I built a Multi-Server SSH Executor using pure Bash. We'll explore parallel processing, robust file parsing, and how to simulate a server cluster locally using Docker.

🎯 The Goal

We want a script that takes a command (e.g., uptime) and runs it on a list of servers defined in a config file.

Requirements:

Parallel Execution: Use threading (background processes) so checking 10 servers takes as long as the slowest one, not the sum of all of them.
Robust Config Parsing: Handle comments, weird whitespace, and different ports/users.
Local Testing Ground: A way to test this without buying 5 VPS instances (spoiler: we use Docker).

🏗️ The Architecture

The project consists of three main parts:

servers.conf: A simple file defining our target servers.
multi_ssh.sh: The brains of the operation.
docker-compose.yml: A simulated lab environment with 4 SSH-enabled containers.

1. The Configuration

I wanted a simple format that's easy to read but flexible:
name:hostname:port:username

# Production Servers
web01:192.168.1.10:22:admin
db01:192.168.1.20:22:dbadmin

# Docker Lab (Localhost mapped ports)
web1:localhost:2221:root
web2:localhost:2222:root

2. The Simulation (Docker Lab)

Testing SSH scripts on production servers is... brave. Instead, I used docker-compose to spin up lightweight Ubuntu containers running sshd.

services:
  web1:
    image: rastasheep/ubuntu-sshd:18.04
    ports: ["2221:22"]
  web2:
    image: rastasheep/ubuntu-sshd:18.04
    ports: ["2222:22"]

Now I have "real" servers running on localhost ports 2221, 2222, etc.

⚡ The "Secret Sauce": Parallelism in Bash

The core challenge is running commands simultaneously. In Bash, we do this by putting a command in the background with &.

Here is the simplified logic:

# Loop through servers
for server in "${servers[@]}"; do
    # Run SSH in the background
    ssh $user@$host "$command" > "/tmp/result_$server.txt" &

    # Save the Process ID (PID)
    pids+=($!)
done

# Wait for all background jobs to finish
wait

This simple trick reduces execution time from (N * Timeout) to (Max(Timeout)).

🧠 Lessons Learned & "Gotchas"

Writing the script revealed a few common Bash pitfalls that I had to fix to make it production-ready.

Lesson 1: `for` loops vs. `while read`

Initially, I used a for loop to read lines from the config file.
The Trap: If a line has spaces (like a description), for splits it into multiple items.
The Fix: Use a while loop with a custom Internal Field Separator (IFS).

# Robust way to read lines
while IFS=':' read -r name hostname port username || [[ -n "$name" ]]; do
    # Process server...
done < "$config_file"

Note the || [[ -n "$name" ]] part—this ensures we don't skip the last line if the file doesn't end with a newline character!

Lesson 2: Race Conditions & Temp Files

When running parallel jobs, you can't just write to output.txt. Multiple processes will write at the same time, garbling the text.
The Fix: Give each process its own temporary file (e.g., /tmp/ssh_result_web1.txt), let them finish, and then aggregate the results sequentially.

I used mktemp to ensure my temporary files never collided with other running instances of the script.

SERVERS_LIST_TMP=$(mktemp /tmp/ssh_multi_servers.XXXXXX)

Lesson 3: SSH is picky

Running SSH non-interactively requires specific flags to avoid hanging:

-o BatchMode=yes: Fail instead of asking for a password.
-o ConnectTimeout=X: Don't wait forever if a server is down.
-o StrictHostKeyChecking=no: Crucial for automated environments where IPs might change (like Docker containers).

🚀 The Result

Running ./multi_ssh.sh "df -h" gives me a beautiful, color-coded summary of disk space across my entire fleet in seconds.

📥 Try It Yourself

I've open-sourced this tool along with the setup script that automatically generates SSH keys and configures the Docker containers for you.

Prerequisites:

Docker & Docker Compose
sshpass (for the initial setup script)

Installation:

git clone https://github.com/alanvarghese-dev/Bash_Scripting/tree/main/ssh_multi_server_executor.git

cd ssh-multi-server-executor
./ssh_install.sh  # Sets up the Docker lab
./multi_ssh.sh "uptime"

Let me know in the comments if you prefer Bash for these tasks or if you stick to heavier tools like Ansible!

Happy scripting! 💻✨

Automating User Management in Linux with Bash Scripts

Alan Varghese — Mon, 02 Mar 2026 05:17:16 +0000

As a DevOps engineer or system administrator, you often find yourself performing repetitive tasks. One of the most common is managing user accounts—especially when onboarding a new team or cleaning up after a project.

Manually running useradd for 20 people isn't just boring; it's prone to errors. That's why I built a simple User Management Automation tool using Bash.

In this post, I'll walk you through how these scripts work and how you can use them to streamline your workflow.

🚀 The Goal

The objective was to create a system that can:

Read a list of usernames from a text file.
Bulk create users with a default password and force a password change on first login.
Bulk delete users and their home directories.
Log every action for auditing purposes.

🛠️ The Scripts

1. The User List (`users.txt`)

Instead of hardcoding names, we use a simple text file. Just add one username per line:

dev1
dev2
ronald

2. User Creation (`create_users.sh`)

This script handles the heavy lifting of onboarding. It checks if a user exists, creates them if they don't, sets a temporary password, and expires it immediately to ensure security.

#!/bin/bash

USER_FILE="users.txt"
PASSWORD="DevOps@1234!"
LOG_FILE="user_creation.log"

echo "User Creation Started: $(date)" >> $LOG_FILE

while read USERNAME
do
    if id "$USERNAME" &>/dev/null
    then
        echo "User $USERNAME already exists" | tee -a $LOG_FILE
    else 
        sudo useradd -m $USERNAME
        echo "$USERNAME:$PASSWORD" | sudo chpasswd
        sudo passwd -e $USERNAME
        echo "User $USERNAME created succesfully" | tee -a $LOG_FILE
    fi
done < $USER_FILE

echo "User Creation Completed: $(date)" >> $LOG_FILE

Key Features:

useradd -m: Creates the home directory automatically.
chpasswd: Efficiently sets passwords from a string.
passwd -e: Forces the user to change their password at the first login—a crucial security step!

3. User Deletion (`del_user.sh`)

When it's time to offboard, this script makes it a one-command job.

#!/bin/bash

USER_LIST="users.txt"
LOG_FILE="user_deletion.log"

echo "User Deletion Started: $(date)" >> $LOG_FILE

while read USERNAME
do 
    if id "$USERNAME" &>/dev/null
    then
        sudo userdel -r "$USERNAME"
        echo "User $USERNAME Deleted Successfully" | tee -a $LOG_FILE
    else
        echo "User $USERNAME does not exist" | tee -a $LOG_FILE
    fi
done < $USER_LIST

echo "User Deletion Completed: $(date)" >> $LOG_FILE

Key Features:

userdel -r: Removes the user and their home directory, keeping the system clean.
Error Handling: Checks if the user exists before trying to delete them.

📈 Logging for Auditing

Both scripts generate log files (user_creation.log and user_deletion.log). This is essential for tracking who was created and when, which is a standard requirement in production environments.

💡 How to Use It

Clone the repository.
Populate users.txt with your desired usernames.
Make the scripts executable: chmod +x *.sh
Run ./create_users.sh to onboard or ./del_user.sh to offboard.

🔒 Security Note

For this demonstration, the password is hardcoded. In a real-world production scenario, you should consider:

Using an environment variable for the default password.
Using a secret management tool.
Prompting for a password during script execution.

🏁 Conclusion

Bash scripting is a superpower for any Linux user. With just a few lines of code, we turned a tedious manual process into a reliable, logged, and automated workflow.

How do you handle user management in your environment? Let me know in the comments!

Check out the full project on my GitHub

Automate Your Server Maintenance: A Simple Bash Script for Log Cleanup

Alan Varghese — Sat, 28 Feb 2026 04:34:12 +0000

Tired of logs eating up your disk space? Learn how to build a simple, automated log cleanup script using Bash.

Every developer or sysadmin has been there: you log into a server to investigate an issue, only to find that the disk is 100% full. More often than not, the culprit is a mountain of old log files that haven't been touched in months.

While tools like logrotate are the industry standard, sometimes you need a lightweight, custom solution that you can understand and deploy in seconds.

In this post, I'll walk you through a simple Bash script I built to automate log cleanup and keep your storage breathing easy.

The Problem

Applications generate logs—lots of them. Whether it's access logs, error logs, or debug traces, these files grow silently. Without a retention policy, they will eventually consume all available disk space, potentially crashing your applications.

The Solution: `auto_clean_log.sh`

I created a script that identifies .log files older than a specific number of days, deletes them, and generates a report of the actions taken.

The Code

Here is the core of the script:

#!/bin/bash

# Configuration
LOG_DIR="/path/to/your/logs"
RETENTION_DAYS=7
REPORT="cleanup_report.log"

echo "------ Log Cleanup Report $(date) ---------" >> $REPORT

# Find files older than X days
FILES=$(find $LOG_DIR -name "*.log" -type f -mtime +$RETENTION_DAYS)

COUNT=0

if [ -z "$FILES" ]
then
    echo "No old log files found." >> $REPORT
else
    for file in $FILES
    do 
        echo "Deleting $file" >> $REPORT
        rm -f $file
        COUNT=$((COUNT+1))
    done
    echo "$COUNT files deleted." >> $REPORT
fi
echo "---------------------------------------------------" >> $REPORT

How It Works

find $LOG_DIR -name "*.log" -type f -mtime +$RETENTION_DAYS: This is the heart of the script.
- -name "*.log": Targets only log files.
- -type f: Ensures we only look at files, not directories.
- -mtime +$RETENTION_DAYS: Filters for files modified more than $X$ days ago.
The Loop: It iterates through the found files, deletes them using rm -f, and tracks the count.
Reporting: Every action is appended to cleanup_report.log, giving you a clear audit trail of what happened and when.

Setup & Automation

To make this truly "set it and forget it," follow these steps:

1. Make it Executable

chmod +x auto_clean_log.sh

2. Automate with Cron

You shouldn't have to run this manually. Add it to your crontab to run every night at midnight:

crontab -e

Add this line:

0 0 * * * /path/to/your/script/auto_clean_log.sh

Why This Matters

Performance: Prevents disk I/O issues related to full partitions.
Cost: Reduces storage costs on cloud providers.
Security/Compliance: Helps adhere to data retention policies by ensuring old logs aren't kept indefinitely.

Conclusion

Automation doesn't always require complex tools or heavy frameworks. Sometimes, a 20-line Bash script is exactly what you need to solve a recurring headache.

Feel free to check out the full project on my GitHub!

How do you handle log management in your environment? Let me know in the comments!

The Evolution of a Bash Tool: Comparing 3 Levels of Log Analysers

Alan Varghese — Fri, 20 Feb 2026 17:44:11 +0000

Log analysis is a bread and butter task for any DevOps engineer or SysAdmin. While there are massive enterprise tools for this, sometimes a quick Bash script is all you need.

In this post, I’ll be comparing three versions of a Bash based Log Analyser I've been working on, showing the transition from a "beginner" script to a "professional grade" CLI tool.

🚀 The Three Contenders

We have three distinct projects in the repository:

log_analyser_simple: The "Keep It Simple, Stupid" (KISS) approach.
log_analyser_scrpt: The "Professional" upgrade with flags and colors.
log_analyser_adv: The "Advanced" version with robust validation and modularity.

1. `log_analyser_simple`: The Bare Essentials

This version is perfect for anyone just starting with Bash. It focuses purely on the logic of parsing a file without the "bells and whistles" of a CLI.

Logic: Uses a simple for loop over hardcoded levels (ERROR, WARNING, INFO).
Argument Handling: Uses positional parameters ($1).
Key Feature: Identifies top unique messages using a classic grep | awk | sort | uniq | head pipeline.

Best for: Learning the basics of text processing in Unix.

2. `log_analyser_scrpt`: The "Professional" SetUp

This is where the script starts feeling like a real tool. It moves away from positional arguments and introduces a much better user experience.

CLI Experience: Uses getopts to handle flags like -f (file), -s (search), and -o (output).
Visuals: Adds ANSI color coding (RED, GREEN, BLUE) to make the terminal output readable at a glance.
Analytics: Introduces User Activity Tracking, extracting usernames from logs to see who the "noisiest" users are.
Reporting: Uses tee to show results on screen while simultaneously saving them to a file.

Best for: Daily use in a development environment where you need quick, readable reports.

3. `log_analyser_adv`: The "Production Ready" Tool

The advanced version takes the professional version and hardens it. It’s built for robustness and edge case handling.

Enhanced Scope: Adds DEBUG and FATAL log levels.
Validation: Includes strict validation for inputs. For example, it checks if the provided log level via the -l flag is actually valid before running.
Modularity: Uses functions to organize logic, making the code much easier to maintain.
Helpful: Includes a proper help message (-h)—a must-have for any shared tool.

Best for: Sharing with a team or using in automated cron jobs where validation is critical.

📊 Feature Comparison

Feature	Simple	Professional	Advanced
Argument Parsing	Positional	`getopts` (Flags)	`getopts` + Validation
Colors	❌	✅	✅
Search Function	❌	✅	✅
Report Export	❌	✅	✅
Log Levels	3	3	5
User Tracking	❌	✅	✅
Help Menu	❌	❌	✅

💡 Which one should you use?

Use simple if you are learning Bash and want to understand how awk and sed work.
Use professional if you want a tool that "just works" and looks good in your terminal.
Use advanced if you need a reliable, validated tool that can handle various log levels and provide a clean help interface.

Conclusion

The journey from a simple loop to a modular, flag driven CLI tool is a great way to master Bash scripting. It shows that even small scripts can be evolved into powerful utilities by focusing on user experience, error handling, and modularity.

What's your favorite Bash trick for log parsing? Let me know in the comments!

Building a Robust Log Analyzer with Bash: From Messy Logs to Actionable Insights

Alan Varghese — Fri, 20 Feb 2026 16:55:59 +0000

As developers and DevOps engineers, we often find ourselves staring at massive log files, trying to pinpoint that one elusive error or understand user behavior. While there are enterprise-grade tools like ELK or Datadog, sometimes you just need a lightweight, fast, and portable solution right in your terminal.

That's why I built the Professional Bash Log Analyzer. In this post, I'll walk you through how it works and how you can use it to make sense of your logs in seconds.

🚀 The Problem

Traditional grep and awk commands are powerful, but chaining them together every time you want a summary can be tedious. I wanted a tool that:

Provides a quick summary of log levels.
Identifies the most frequent errors.
Tracks the most active users.
Filters by specific levels or keywords.
Works out of the box on any Linux/macOS system.

✨ Key Features

My script, log_analyser.sh, comes packed with features designed for real world use:

Colorized CLI Output: Highlighting errors in red and info in green makes reports instantly readable.
Log Level Filtering: Support for INFO, WARNING, ERROR, DEBUG, and FATAL.
Keyword Search: Quickly find specific entries (e.g., "Database" or "Timeout").
Automated Summaries:
- Total entry count.
- Breakdown by log level.
- Top 5 most frequent error messages.
- Top 5 most active users.
Report Export: Easily save your analysis to a text file for sharing or auditing.
Professional CLI Experience: Built using getopts for robust argument parsing.

🛠 How It Works

The script follows a clean, modular structure. Here's a look at how it handles the core analysis logic:

analyze() {
    echo -e "${BLUE}--- Analysis Report for: $LOG_FILE ---${NC}"
    echo "Generated on: $(date)"

    # ... Count log levels ...
    echo -e "${RED}ERROR:   $(grep -i "ERROR" "$LOG_FILE" | wc -l)${NC}"

    # Extract Top 5 Error Messages
    echo -e "
${RED}--- Top 5 Error Messages ---${NC}"
    grep -i "ERROR" "$LOG_FILE" | awk '{ $1=$2=$3=""; print $0 }' | sort | uniq -c | sort -nr | head -n 5

    # Identify Top 5 Active Users
    echo -e "
${GREEN}--- Top 5 Active Users ---${NC}"
    grep -i "User" "$LOG_FILE" | awk '{ print $5 }' | tr -d "'" | sort | uniq -c | sort -nr | head -n 5
}

Smart Parsing with `getopts`

I used getopts to ensure the tool feels like a standard Linux utility. You can mix and match flags effortlessly:

./log_analyser.sh -f sample.log -l ERROR -s "Database" -o report.txt

📖 Usage Examples

1. Basic Analysis

Get a quick overview of your log file:

./log_analyser.sh -f server.log

2. Filter for Critical Issues

Focus only on ERROR logs and search for specific failure points:

./log_analyser.sh -f server.log -l ERROR -s "Connection"

3. Generate a Permanent Report

Save the output to a file while still seeing it in your terminal:

./log_analyser.sh -f server.log -o daily_report.txt

🧠 Lessons Learned

Developing this tool reinforced a few core Bash principles:

Modularity: Using functions like analyze() and usage() makes the script much easier to maintain.
Validation: Always validate user input. Checking if the file exists and if the log level is valid prevents cryptic shell errors later.
UX Matters: Adding ANSI color codes might seem small, but it significantly improves the user experience when scanning through data.

🎁 Wrap Up

This Log Analyzer is open-source and ready for you to tweak! Whether you're debugging a microservice or monitoring a legacy server, I hope this tool saves you some "grep-ping" time.

Check out the code and let me know what features you'd add next!

Follow me for more DevOps and automation tips!

Stop Reading Logs Manually: Build a Professional Log Analyzer in Bash

Alan Varghese — Fri, 20 Feb 2026 16:53:20 +0000

We've all been there staring at a massive log file, trying to figure out why a service is failing or which user is causing the most errors. Manually searching through thousands of lines using less or grep is tedious and error prone.

In this post, I'll show you how I built a Professional Log Analyzer using Bash. It's lightweight, color coded, and gives you instant insights into your application's health.

🚀 The Problem

Modern applications generate a lot of data. When things go wrong, you need answers fast:

How many errors happened in the last hour?
Which error is the most frequent?
Which users are most active (or causing the most trouble)?

🛠️ The Solution: `log_analyzer.sh`

I developed a script that transforms messy log data into a structured, readable report. Here are the core features:

Automated Summaries: Instantly counts INFO, WARNING, and ERROR levels.
Error Ranking: Shows the Top 5 most frequent error messages.
User Activity Tracking: Identifies the Top 5 most active users.
Custom Keyword Search: Quickly filter logs for specific issues (e.g., "Database" or "Timeout").
Professional Output: Uses ANSI color codes for readability and supports saving reports to a file.

💻 How It Works

The script uses standard Unix utilities (awk, grep, sort, uniq) and getopts for a professional CLI experience.

Parsing Arguments with `getopts`

I used getopts to handle command line flags, making the script feel like a real tool:

while getopts "f:s:o:" opt; do
    case $opt in
        f) LOG_FILE=$OPTARG ;;
        s) SEARCH_KEY=$OPTARG ;;
        o) OUTPUT_FILE=$OPTARG ;;
        *) usage ;;
    esac
done

The Analysis Logic

The heart of the script lies in combining pipe lined commands. For example, to find the most active users:

grep -i "User" "$LOG_FILE" | awk '{ print $5 }' | tr -d "'" | sort | uniq -c | sort -nr | head -n 5

This single line searches for user entries, extracts the username, cleans it up, counts occurrences, sorts them, and grabs the top 5.

📊 Sample Output

When you run the script, you get a clean, colorized report:

--- Analysis Report for: sample.log ---
Generated on: Fri Feb 20 14:30:00 UTC 2026
Total log entries: 1250

--- Log Level Counts ---
INFO:    850
WARNING: 300
ERROR:   100

--- Top 5 Error Messages ---
  45 Connection timeout to database
  20 Disk space low
  15 Invalid API key
  10 Unauthorized access attempt
   5 Cache sync failed

🧠 What I Learned

Building this tool reinforced several key concepts:

The Power of Pipes: Unix pipes are incredibly efficient for processing text data.
CLI UX Matters: Adding colors and clear flag based arguments makes a script much more usable for other developers.
Regex is your Friend: Using grep and awk effectively can replace complex Python or Node.js scripts for simple log processing.

📂 Try it Yourself!

If you want to automate your own log analysis, check out the project structure:

log_analyzer.sh: The main engine.
sample.log: For testing your regex and logic.

Question for you: How do you currently handle log analysis in your workflow? Do you use a full ELK stack, or do you have some "secret sauce" Bash scripts of your own?

Let's discuss in the comments! 👇

If you found this helpful, feel free to give it a ❤️ and follow for more DevOps and scripting tips!

Automate Your Log Analysis with This Simple Bash Script

Alan Varghese — Fri, 20 Feb 2026 16:50:22 +0000

If you've ever spent too much time staring at a wall of text in a log file, trying to figure out why your application is crashing, this post is for you.

While there are powerful tools like ELK or Datadog for enterprise scale logging, sometimes you just need something quick, local, and "no nonsense" to parse a log file on your machine or a remote server.

Today, I'm sharing a simple Log Analyser script I built in Bash that categorizes logs and surfaces the most frequent issues automatically.

The Problem

Scanning logs manually is tedious and error prone. You might grep "ERROR" file.log and then realize you have 500 lines of the same database connection error, hiding a single "File not found" error that is the actual root cause.

The Solution: `log_analyser.sh`

I created a lightweight script that does three things:

Counts total occurrences of INFO, WARNING, and ERROR levels.
Aggregates unique messages so you can see which specific error is occurring most often.
Displays a clean summary.

The Code

Here is the heart of the script:

#!/bin/bash

# ... basic validation logic ...

LOG_LEVELS=("ERROR" "WARNING" "INFO")

for LEVEL in "${LOG_LEVELS[@]}"; do
  COUNT=$(grep -ic "$LEVEL" "$LOG_FILE")
  echo -e "
[$LEVEL] Total occurrences: $COUNT"

  if [ "$COUNT" -gt 0 ]; then
    echo "Top unique $LEVEL messages:"
    grep -i "$LEVEL" "$LOG_FILE" | awk '{$1=$2=$3=""; print $0}' | sed 's/^[    ]*//' | sort | uniq -c | sort -rn | head -n 5
  fi
done

How It Works: The "Power Pipeline"

The most interesting part of this script is the command pipeline used to extract unique messages:

grep -i "$LEVEL" "$LOG_FILE" | awk '{$1=$2=$3=""; print $0}' | sed 's/^[ ]*//' | sort | uniq -c | sort -rn | head -n 5

Let's break it down:

grep -i "$LEVEL": Finds the log level (case-insensitive).
awk '{$1=$2=$3=""; print $0}': This is a neat trick! It clears the first three fields (usually Date, Time, and Level) so we only look at the actual message content.
sed 's/^[ ]*//': Trims the leading whitespace left behind by awk.
sort | uniq -c: Sorts the messages and then counts how many times each unique message appears.
sort -rn: Sorts the results numerically in reverse order (highest count first).
head -n 5: Only shows us the top 5 most frequent messages.

How to Use It

Clone the script (or copy it from above).
Make it executable: chmod +x log_analyser.sh
Run it against any log file: ./log_analyser.sh sample.log

Example Output

--- Log Analysis Summary for: sample.log ---

[ERROR] Total occurrences: 4
Top unique ERROR messages:
   3 Database connection failed.
   1 File not found: /var/www/html/index.php

[WARNING] Total occurrences: 2
Top unique WARNING messages:
   1 Memory usage high.
   1 Disk usage at 85%.

...

Conclusion

Bash is incredibly powerful for these kinds of "glue" tasks. By combining a few standard Unix tools, we've created a tool that saves minutes of manual work every time we debug a service.

What are your favorite "one liner" Bash tricks for log analysis? Let me know in the comments!

This project was a great exercise in learning Bash best practices and command line data processing. You can find the full project on my https://github.com/alanvarghese-dev/Bash_Scripting

"Bash Backup Battle: Minimalist vs. Feature-Rich Scripts"

Alan Varghese — Wed, 18 Feb 2026 10:22:23 +0000

In the world of automation, there's rarely a "one size fits all" solution. Sometimes you need a quick script to throw into a cron job, and other times you need a robust, configuration-driven tool that can handle remote transfers and detailed logging.

In this post, I’m comparing two Bash backup projects I've been working on. Both get the job done, but they take very different paths to get there.

The Contenders

1. The "DevOps-Ready" Script (`file_backup_script`)

This project is built for complexity and reliability. It treats configuration as code by separating the logic from the settings.

Key Features:

Configuration File (backup.conf): No need to touch the code to change paths or retention days.
Dry Run Mode (-d): A safety-first approach that tells you exactly what would happen without moving a single byte.
Remote Transfer: Built-in support for scp to push backups to a remote server.
Advanced Logging: Every action, warning, and error is timestamped and recorded in a dedicated log file.

2. The "Minimalist" Script (`File_backup_scripts`)

This project focuses on speed and simplicity. It’s the kind of script you can call on the fly from the terminal.

Key Features:

Argument-Driven: Pass your source and destination directly: ./backup.sh /src /dest.
Exclusion Patterns: Quickly skip .git folders or log files using glob patterns.
Automatic Cleanup: A hardcoded 30-day retention policy ensures your disk doesn't fill up.
Low Overhead: No config files to manage; just the script and your directories.

🔍 Feature Comparison

Feature	DevOps-Ready	Minimalist
Input Method	Configuration File	CLI Arguments
Remote Support	Yes (`scp`)	No (Local only)
Testing	Dry Run Flag	Manual
Multiple Sources	Supported	Single Source
Logging	Detailed File Logs	Standard Output
Complexity	Medium	Low

Which One Should You Use?

Use the DevOps-Ready Script if...

You are managing a server or a production environment. The Dry Run feature alone makes it much safer to deploy, and the Remote Transfer capability ensures your data is safe even if the local disk fails. It’s perfect for those who want to "set it and forget it."

Use the Minimalist Script if...

You need to grab a quick backup of a project folder before making major changes. It’s lightweight, requires zero setup, and handles the exclusion of messy folders like node_modules or .git with ease.

💡 Lessons Learned

Writing both of these taught me that context is king.

When I was building the DevOps version, I focused heavily on error handling and idempotency. I wanted to make sure that if a remote transfer failed, the script wouldn't just crash—it would log the error and move on.

With the minimalist version, the focus was on ergonomics. I wanted the shortest possible command to yield a reliable result.

Conclusion

Both scripts are available in my repository! Whether you need a robust automation tool or a quick-and-dirty backup utility, there's a Bash solution here for you.

Check out the code and let me know: Do you prefer config files or CLI arguments for your scripts?

https://github.com/alanvarghese-dev/Bash_Scripting

https://www.linkedin.com/in/alanvarghese-dev

Automate Your Backups Like a Pro: A Robust Bash Script for DevOps Enthusiasts.

Alan Varghese — Wed, 18 Feb 2026 10:17:33 +0000

In the world of DevOps, there's a golden rule: If it’s not backed up, it doesn’t exist.

While there are many enterprise-grade backup solutions available, sometimes you need something lightweight, highly customizable, and easy to integrate into your existing workflows. That's where a well-crafted Bash script comes in.

In this post, I'll walk you through a Robust File Backup Script I built that handles compression, remote transfers, retention policies, and detailed logging—all while following core DevOps principles.

Key Features

Our backup script isn't just a simple cp command. It's designed to be production-ready with features like:

Smart Compression: Uses tar and gzip to minimize storage space.
Configuration Decoupling: All settings live in a separate backup.conf file (Infrastructure as Code lite).
Dry Run Mode: A -d flag to see exactly what would happen without actually doing it.
Automatic Retention: Keeps your disk clean by deleting local backups older than N days.
Secure Remote Transfer: Optionally sends your archives to a remote server via scp.
Comprehensive Logging: Every action, warning, and error is timestamped and logged for auditing.

🛠 The Technical Breakdown

1. Separation of Concerns

We keep our logic (backup.sh) separate from our settings (backup.conf). This makes the script portable across different environments (Dev, Stage, Prod) without modification.

backup.conf example:

SOURCE_PATHS=(
    "/var/www/html"
    "/etc/nginx/conf.d"
)
BACKUP_DIR="./backups"
RETENTION_DAYS=7
ENABLE_REMOTE="true"
REMOTE_HOST="backup-server.local"

2. Flexible Argument Parsing

Using getopts, the script feels like a professional CLI tool. You can specify custom config files or trigger a dry run easily.

while getopts ":c:dh" opt; do
    case ${opt} in
        c ) CONFIG_FILE=$OPTARG ;;
        d ) DRY_RUN=true ;;
        h ) usage ;;
    esac
done

3. The Power of `find` for Retention

Managing disk space is crucial. We use find with the -mtime flag to identify and remove old archives automatically.

find "$BACKUP_DIR" -type f -name "backup_*.tar.gz" -mtime +"$RETENTION_DAYS" -exec rm {} \;

4. Secure Remote Offloading

A backup on the same disk isn't a true backup. Our script supports secure transfer to a remote host:

scp "$BACKUP_DIR/$BACKUP_NAME" "$REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH"

📖 Lessons Learned & DevOps Principles

Building this project reinforced several key concepts:

Automation over Manual Work: Human error is the #1 cause of data loss. Automating the backup removes the "I forgot" factor.
Idempotency & Resilience: The script checks if directories exist and if source paths are valid before starting.
Visibility: "In DevOps, if it wasn't logged, it didn't happen." Detailed logs are essential for debugging scheduled cron jobs.
Safety First: The Dry Run mode is a lifesaver when testing new configurations on a production server.

How to Use It

Clone the Repo: [Link to your repo here]
Configure: Edit backup.conf with your paths.
Make it Executable: chmod +x backup.sh
Test it: ./backup.sh -d (Dry run)
Run it: ./backup.sh
Schedule it: Add it to your crontab to run nightly!

0 2 * * * /path/to/backup.sh >> /path/to/logs/cron.log 2>&1

Future Enhancements

Adding AWS S3 support using the AWS CLI.
Implementing Slack/Email notifications on failure.
Adding Checksum verification to ensure data integrity after transfer.

What does your backup strategy look like? Do you prefer simple scripts or complex tools? Let’s discuss in the comments! 👇

https://github.com/alanvarghese-dev/Bash_Scripting

https://www.linkedin.com/in/alanvarghese-dev

Forem: Alan Varghese

Build a Lightweight File Integrity Monitor with Bash, SQLite, and Docker

🚀 The Core Concept: Baseline vs. Reality

🛠️ The Tech Stack

🔍 A Deep Dive into the Code

1. The Hashing Logic

2. Multi Layered Alerting

🐳 The Docker Testing Workflow

How to test a "Malicious" change:

💡 Key Lessons Learned

🏁 Conclusion

Monitoring Remote Network Health: Building a Lightweight Connectivity Tool with Bash and Docker

The Problem

The Solution: A Bash-Powered Monitor

1. Robust Scripting with set -euo pipefail

2. Intelligent Connectivity Checks

3. Automated Logging & Summaries

Testing Safely with Docker

How to Try It Out

Lessons Learned

Conclusion

How I Built a Lightweight Cron Job Health Monitor with Bash and Docker

Why Your Cron Jobs Need Monitoring

🛠 The Architecture

🚀 Key Features

1. Automated SSH Discovery

2. State-Based Tracking

3. Dockerized Testing Environment

🧩 The "Aha!" Moments (and Bugs)

🛠 Quick Start

💡 Lessons Learned

Stop SSH-ing One by One: Building a Parallel Command Executor in Bash

🎯 The Goal

🏗️ The Architecture

1. The Configuration

2. The Simulation (Docker Lab)

⚡ The "Secret Sauce": Parallelism in Bash

🧠 Lessons Learned & "Gotchas"

Lesson 1: for loops vs. while read

Lesson 2: Race Conditions & Temp Files

Lesson 3: SSH is picky

🚀 The Result

📥 Try It Yourself

Automating User Management in Linux with Bash Scripts

🚀 The Goal

🛠️ The Scripts

1. The User List (users.txt)

2. User Creation (create_users.sh)

3. User Deletion (del_user.sh)

📈 Logging for Auditing

💡 How to Use It

🔒 Security Note

🏁 Conclusion

Automate Your Server Maintenance: A Simple Bash Script for Log Cleanup

The Problem

The Solution: auto_clean_log.sh

The Code

How It Works

Setup & Automation

1. Make it Executable

2. Automate with Cron

Why This Matters

Conclusion

The Evolution of a Bash Tool: Comparing 3 Levels of Log Analysers

🚀 The Three Contenders

1. log_analyser_simple: The Bare Essentials

2. log_analyser_scrpt: The "Professional" SetUp

3. log_analyser_adv: The "Production Ready" Tool

📊 Feature Comparison

💡 Which one should you use?

Conclusion

Building a Robust Log Analyzer with Bash: From Messy Logs to Actionable Insights

🚀 The Problem

✨ Key Features

🛠 How It Works

Smart Parsing with getopts

📖 Usage Examples

1. Basic Analysis

2. Filter for Critical Issues

3. Generate a Permanent Report

1. Robust Scripting with `set -euo pipefail`

Lesson 1: `for` loops vs. `while read`

1. The User List (`users.txt`)

2. User Creation (`create_users.sh`)

3. User Deletion (`del_user.sh`)

The Solution: `auto_clean_log.sh`

1. `log_analyser_simple`: The Bare Essentials

2. `log_analyser_scrpt`: The "Professional" SetUp

3. `log_analyser_adv`: The "Production Ready" Tool

Smart Parsing with `getopts`

🛠️ The Solution: `log_analyzer.sh`

Parsing Arguments with `getopts`

The Solution: `log_analyser.sh`

1. The "DevOps-Ready" Script (`file_backup_script`)

2. The "Minimalist" Script (`File_backup_scripts`)

3. The Power of `find` for Retention