Forem: DanLin

How I Got Lost in Three Logs — and Still Found the User

DanLin — Mon, 01 Dec 2025 13:26:44 +0000

~~... or why last, lastlog, and journalctl behave like three different Linux~~
~~... and why writing a simple script turned into a small investigation of Linux~~

🧩 Prologue — I thought it would be easy

This time, I wanted to write a script that could determine who last logged into Linux and when, and then generate a report in CSV file format.
Now... I understand why SRE engineers are gray-haired.

In theory, it's clear:

Command	meaning
last	login history (wtmp)
lastlog	only the most recent entry
journalctl	systemd login logs

But in practice — they behaved like three half-blind witnesses:

each knows something,
each lies about something,
and sometimes… they just remain silent.

😯 Why `last` isn't always telling the truth

last -w -n 1 $USER

👎 still logged in — even if the user already logged out
👎 if wtmp is rotated → history is gone
👎 SSH/LDAP logins may not be detected
👎 PAM config affects results — different distros show different output

last doesn't guarantee that it knows the truth at all.

🤥 lastlog: seems reliable… until it isn’t

lastlog -u $USER

Real user	lastlog
Logged in 3 days ago	(Never logged in)
Never logged in	(Never logged in)

And how can you tell which one is real?
No way. Code fallback!

🛰 journalctl: almost a hero... if you're lucky

journalctl _UID=$(id -u $USER) --since="1 month ago"

Issue	What's happening
Various distributions	systemd not available → useless command
Log rotation	history disappears
Sudo required	some logs are hidden
Different PAM	can produce completely different events

journalctl sometimes knows everything...
...and sometimes it's as if it has never seen this user before.

🧪 Engineering solution: fallback logic

if last:
    #use that date
else if lastlog:
    #use its data
else:
    #return  “never” or “invalid”

This is not “beautiful code.”
This is behavior in chaos.
This is already “little SRE.”

🛠 Script (user-last-login-tool)

Workflow:

last → if there is a date — great
if not → lastlog
If still empty → return "never" or "invalid"

🔧 What I added:

set -euo pipefail
--help
--version
CSV-report
logging to /var/log/sre-tools
fallback logic for reliability
packaged for AUR / DEB / RPM
Dockerfile
GitHub Actions + Makefile CI
man-page

🧭 What surprised me the most

I expected to write a simple Bash script.
Instead, I discovered that Linux has no single “source of truth” about user logins.

That raised an interesting question:

How do SREs make decisions
when data is incomplete or contradictory?

That question turned this script
from a util — into an investigation.

👉 My project

https://github.com/DanLinX2004X/user-last-login-tool

💡 Why I’m Sharing This

LLMs can write code.
But they don’t feel the pain
when lastlog returns (Never logged in)
for a user who logged in yesterday.

That “pain” — is engineering experience.
And that can’t (yet) be generated.

🗣 Your Turn

Have you seen different behavior of these commands in:

Debian?
CentOS / RHEL?
LDAP / SSSD setups?

If yes — I’d really love to hear about it.
It might help make the tool actually useful.

Is this a universal Linux problem — or just my personal adventure?
Let me know 👇

How I Learned IaC Without Losing My Mind 🤯

DanLin — Sun, 09 Nov 2025 12:11:17 +0000

Prologue: The Quest for IaC Experience

I decided to prepare for an SRE/DevOps internship and learn Infrastructure as Code.

The challenge: build a portfolio project demonstrating IaC skills, but without using clouds or paid VPS — student budget is tight, and I don't trust those “free trials” that suddenly turn into bills.

I asked AI assistants for advice and got:

"Use Vagrant + Ansible + Terraform!"

Sounded impressive. In practice… well, let's just say it was an adventure.

Chapter 1: The Rise and Fall of Vagrant

Installed Vagrant on my CachyOS and added plugins as recommended by Arch Wiki.

Created a basic Vagrantfile — what could possibly go wrong?

Error #1: The Ghost of Plugins Past

undefined method 'exists?' for class File (NoMethodError)
path && File.exists?(path)

The vagrant-vbguest plugin had been abandoned since 2023 and was incompatible with modern Ruby.
The solution was surprisingly simple:

config.vbguest.auto_update = false

After some ritual dancing, the VM finally booted! 🥳

Chapter 2: Terraform and the Over-engineering Trap

Next phase: two VMs via Terraform + Ansible configuration.
This is where the real fun began:

Ansible installed Docker, Prometheus, and Grafana on both VMs — why?
Docker-ce refused to install due to dependency hell
At some point Ansible just froze without explanation

The most ironic moment: after hours of struggle, the AI assistant finally declared:

"Vagrant is legacy crap, just drop it"

I felt like a character in Harlan Ellison's “I Have No Mouth, and I Must Scream” — trapped in a digital nightmare of my own making. 🤡

Chapter 3: Monitron and the Golden Screwdriver

I decided to improve my existing Monitron project (a monitoring stack in Docker Compose) by adding Ansible.
The AI insisted: “This will give you idempotency and portability!”

In practice, I got a playbook that… copied files and ran docker-compose up.
Ansible became a golden screwdriver for a simple task.

Chapter 4: The PostgreSQL Wars

In my final attempt to create ansible-lab, I faced epic PostgreSQL battles:

Permission Error

chmod: invalid mode: 'A+user:postgres:rx:allow'

Authentication Nightmare

FATAL: Peer authentication failed for user "postgres"

The AI suggested everything — installing extra collections, solving non-existent permission issues, and even thought I was on Windows!

The solution turned out to be laughably simple — replace complex modules with shell commands:

- name: Create PostgreSQL user
  command: sudo -u postgres psql -c "CREATE USER myapp_user WITH PASSWORD 'secret';"

And miracle of miracles — it worked on the first try!

Epilogue: ansible-lab and Lessons Learned

I finally built a working pet project:

Vagrant creates two VMs (db + app)
Ansible manages them via roles
No Terraform, no unnecessary complexity

Lessons Learned:

Trust, but Verify AI assistants are tools, not gurus. Always ask:

"Why do I need this technology for my specific use case?"

Ansible Without VMs is Over-engineering
Use Bash for your local machine, Docker for containers.
Ansible shines when you have multiple servers to manage.
Sometimes Simple is Better
My final solution was simpler than the original plans, but it actually works and demonstrates exactly what I needed.
Real Experience > Perfect Tutorials
In interviews, real problem-solving stories are valued more than memorized syntax.

💡 Advice for Beginners:
Start with small but working projects. Sometimes a simple Vagrant lab is more valuable than a complex "everything-but-the-kitchen-sink" setup.

~~P.S. Dear AI assistants, I still love you. But sometimes you're absolute trolls. 😄~~

👉 Check out my ansible-lab on GitHub

💀 I built a full system monitor in Bash — and fought awk along the way

DanLin — Tue, 04 Nov 2025 21:16:25 +0000

I wanted to see how far I could push pure Bash before it collapses under its own syntax.
So, naturally, I decided to write a system monitoring tool — in Bash.
And thus, system-monitor was born:
👉 GitHub repo

Yes, it’s fully functional. Yes, it uses colors. Yes, awk is involved.

No, I don’t recommend doing this sober. 😅

💡 The idea

I wanted a single script that could show:

CPU load and number of cores
RAM usage and percentage
Disk space for /
Network I/O
Process count Basically, the “lazy Linux admin toolkit” in one file. Something like this:

./system-monitor.sh

and boom — everything you’d usually get from top, free, df, and ip combined.

⚙️ The features

What started as a 10-line script turned into a 250+ line CLI tool with:

Command-line arguments (--help, --brief, --no-color, --version, -i N)
Colorized output for warnings/critical thresholds
Continuous monitoring mode
Safe shutdown with signal trapping (Ctrl+C)
Brief (machine-readable) mode for piping into other scripts
Dependency checks and graceful error handling

Example:

./system-monitor.sh -i 5 --no-color

Updates every 5 seconds without colors.
Press Ctrl+C to stop — yes, it even says goodbye politely.

🧩 Some code highlights

Color management:

RED='\033[0;31m'
YELLOW='\033[1;33m'
GREEN='\033[0;32m'
NC='\033[0m'

Because monitoring your system is serious business — but who doesn’t love a bit of RGB?

Load detection magic:

CPU_LOAD_1MIN=$(cat /proc/loadavg | awk '{print $1}')
CPU_CORES=$(nproc)
load_percent=$(echo "scale=0; ($CPU_LOAD_1MIN * 100) / $CPU_CORES" | bc)

That moment when you realize you’re using awk, bc, and /proc in one line and it actually works.

Network metrics:

interface=$(ip route get 8.8.8.8 | awk '{print $5}' | head -1)
rx=$(cat /sys/class/net/$interface/statistics/rx_bytes)
tx=$(cat /sys/class/net/$interface/statistics/tx_bytes)

I like to imagine Bash crying softly every time it runs this.

🧠 What I learned

Bash can do a lot — if you’re patient (and slightly masochistic).
Quoting is a survival skill. Forget one and you’re debugging for hours.
awk is an ancient curse. It always works, but never for the reason you expect.
Color helps debugging. Visual feedback saves your sanity.
Document your code. Because 3 a.m. you will have no clue what that $7 in awk '{print $7}' was.

🧰 The brief mode

I added a --brief flag that outputs all metrics in a machine-readable format — perfect for logging or Prometheus-style integration.

./system-monitor.sh --brief

Output:

timestamp=1730765432
cpu_load_percent=12
mem_usage_percent=43.5
disk_usage_percent=56
network_rx_mb=12.4
process_count=187

Basically, JSON for people who hate themselves.

🧯 Safety first: signal handling

If you hit Ctrl+C, it doesn’t just die — it politely says goodbye:

trap cleanup INT TERM

cleanup() {
    echo -e "\nMonitoring stopped. Goodbye!"
    exit 0
}

Because professionalism.

🚀 AUR packaging

After testing it on Arch, I thought — why not go all in?
So I wrote a PKGBUILD, threw it into the AUR, and now you can literally install it with:

yay -S system-monitor

And that’s the moment I realized:

“Oh no. I’ve become that guy who writes Bash utilities for other people.”

🧭 Final thoughts

Was this efficient? Probably not.
Did I learn a lot? Absolutely.

Here’s what I got from it:

Deep understanding of Bash syntax and traps
Practical use of /proc and awk
Appreciation for proper CLI design
And a newfound respect for people who don’t do this in Bash

If you’re thinking of writing your first utility — do it.
Even if it’s messy. Even if awk haunts your dreams.

Because once you see your tool working, it’s worth every headache.

🔗 Links

🐧 GitHub: https://github.com/DanLinX2004X/system-monitor

📦 AUR: https://aur.archlinux.org/packages/system-monitor