Forem: Oyoh Edmond

How I Spent a Day Trying to Recover a Crashed OpenStack Environment — And What I Learned

Oyoh Edmond — Thu, 02 Apr 2026 15:39:35 +0000

A real-world incident report for engineers dealing with filesystem corruption on production Linux servers

The Problem

It started with a simple complaint: our company's OpenStack Horizon portal was unreachable. The browser returned ERR_CONNECTION_TIMED_OUT. No warning, no gradual degradation — just gone.

We had two physical HPE ProLiant DL380 Gen10 servers running the environment, accessible only via HP iLO 5 remote console. No physical access. No one near the data centre. Just me, a browser, and an iLO HTML5 console.

This is the story of what happened, what we tried, what failed, and what every engineer should know before they find themselves in the same situation.

The Environment

Controller Node: HPE ProLiant DL380 Gen10 (12-core)
Compute Node: HPE ProLiant DL380 Gen10 (10-core)
OS: Ubuntu 22.04 LTS
Storage: LVM on top of hardware RAID (HPE Smart Array P408i-a)
Access: HP iLO 5 remote console (HTML5)
VPN: FortiClient VPN required to reach internal network

Step 1 — Diagnosing the Problem

The first thing I noticed was that pinging the servers returned Destination host unreachable even on VPN. This ruled out a simple service crash — something was fundamentally wrong at the OS level.

Opening the iLO console for the controller node revealed the server was stuck in a BusyBox initramfs emergency shell with the following critical errors:

UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY
Failure: File system check of the root filesystem failed
The root filesystem requires a manual fsck

Lesson #1: Always check iLO/IPMI console first. The OS may be completely down while the management interface is still accessible.

Step 2 — The Filesystem Corruption

The root filesystem was on an LVM logical volume. The initramfs had tried to run an automatic fsck and failed. The errors pointed to:

Superblock corruption — the filesystem size recorded in the superblock was larger than the actual LVM volume
Journal corruption — e2fsck could not set superblock flags
Thousands of corrupted inodes — invalid flags, bad extended attributes, wrong inode sizes

The size mismatch error was particularly telling:

The filesystem size is 288358400 blocks
The physical size of the device is 285474816 blocks
Either the superblock or the partition table is likely to be corrupt

Lesson #2: A filesystem size larger than the physical device usually means the LVM volume was shrunk without first shrinking the filesystem, or the superblock was corrupted during an unclean shutdown.

Step 3 — Recovery Attempts in initramfs

The initramfs environment is extremely limited. Here is what we tried and the results:

Activating LVM Volumes

vgchange -ay

✅ This worked and activated all volume groups.

Running e2fsck with Backup Superblock

e2fsck -y -b 32768 /dev/mapper/<your-lv-name>

⚠️ This started working but kept getting killed by the OOM (Out of Memory) killer because initramfs has very limited RAM available for processes.

Extending the LVM Volume

lvm lvextend -l +100%FREE /dev/<vg-name>/<lv-name>

✅ This successfully extended the volume to match what the filesystem expected.

Rewriting the Superblock

mke2fs -S -b 4096 /dev/mapper/<your-lv-name>

✅ The superblock was rewritten. e2fsck then started making real progress fixing inodes.

Creating Swap to Help with OOM

dd if=/dev/zero of=/swapfile bs=1048576 count=4096
mkswap /swapfile
# swapon /swapfile — NOT AVAILABLE in initramfs

❌ swapon is not available in initramfs. This is a critical limitation.

Lesson #3: The initramfs environment is missing many essential tools including swapon, resize2fs, tune2fs, debugfs, and lvextend. Plan for this limitation before you need it.

Step 4 — The OOM Problem

Every time e2fsck got deep into repairing the large volume, the kernel OOM killer terminated it:

Out of memory: Killed process (e2fsck)

The server had significant RAM but initramfs was only making a small portion available for user processes. Without swap, e2fsck couldn't complete the repair.

Lesson #4: For large filesystems (500GB+), e2fsck requires significant RAM. Always ensure swap is available before running fsck on large volumes. If you're in initramfs without swap, you need a different approach.

Step 5 — Attempting to Boot from Live ISO

We tried to boot Ubuntu 20.04 Live Server from an ISO mounted via iLO Virtual Media. This would have given us a full Ubuntu environment with all tools.

The challenges we encountered:

iLO Virtual Media URL-based ISO streaming was too slow
Local ISO file mounting via iLO HTML5 console worked better
The ISO was detected as a Virtual CD-ROM by the kernel
However, the server's UEFI boot order did not include the virtual CD-ROM
The virtual CD-ROM did not appear in the UEFI one-time boot menu

Lesson #5: Test your iLO Virtual Media boot process BEFORE you need it in an emergency. Know whether your server's UEFI will boot from iLO virtual media and in what order.

Step 6 — UEFI Shell to the Rescue (Partially)

We discovered the HPE Embedded UEFI Shell under:
System Utilities → Embedded Applications → Embedded UEFI Shell

From there we could launch the GRUB bootloader directly:

fs0:
cd EFI\ubuntu
shimx64.efi

This gave us access to the GRUB menu and boot parameter editing. We modified the boot parameters to skip fsck:

linux /vmlinuz-<version>-generic root=/dev/mapper/<lv-name> ro fsck.mode=skip

Unfortunately the filesystem was too corrupted to mount even with fsck skipped.

Lesson #6: The HPE Embedded UEFI Shell is a powerful recovery tool. Learn how to use it. It can launch bootloaders directly from the EFI partition without needing a working boot order.

Step 7 — The Final Verdict

After extensive repair attempts, the final error was:

EXT4-fs error: inode #2: special inode unallocated
get root inode failed
mount failed

Inode #2 is the root directory inode — the most critical inode in any ext4 filesystem. When this is destroyed, the filesystem cannot be mounted under any circumstances without specialist data recovery tools.

Lesson #7: If inode #2 is corrupted, you need either a backup restore or professional data recovery. No amount of e2fsck will fix a destroyed root inode.

What Should Have Been Done Differently

Before the Incident

Regular backups — snapshots of the LVM volume or VM-level backups
Monitoring — disk health monitoring (smartctl), filesystem error monitoring
Documentation — record all credentials, architecture diagrams, and recovery procedures
Test recovery — periodically test that backups can actually be restored
Swap space — ensure servers have adequate swap configured

During the Incident

Boot from USB first — don't spend hours in initramfs; immediately boot from a live USB with full tools
Create swap immediately — before running e2fsck on large volumes, ensure swap is available
Use a higher-level backup superblock — if 32768 doesn't work, try 98304 or 163840
Document every command — keep a log of everything you try

Tools You Need Available

A bootable Ubuntu Live USB drive (or ISO ready for iLO virtual media)
resize2fs, tune2fs, debugfs — not available in initramfs
swapon — not available in initramfs
Adequate RAM (at least 8GB free) for e2fsck on large volumes

Key Commands Reference

# Activate LVM volumes from initramfs
vgchange -ay

# List mapper devices
ls /dev/mapper/

# Find backup superblocks
dumpe2fs /dev/mapper/<device> | grep -i superblock

# Run fsck with backup superblock
e2fsck -y -b 32768 /dev/mapper/<device>

# Extend LVM volume (using lvm wrapper in initramfs)
lvm lvextend -l +100%FREE /dev/<vg-name>/<lv-name>

# Rewrite superblock (does NOT destroy data)
mke2fs -S -b 4096 /dev/mapper/<device>

# Create swap file
dd if=/dev/zero of=/swapfile bs=1048576 count=4096
mkswap /swapfile
swapon /swapfile  # (not available in initramfs)

# Mount filesystem read-only
mount -o ro /dev/mapper/<device> /mnt

# Chroot into recovered system
chroot /mnt /bin/bash

Conclusion

Filesystem corruption at the inode level is one of the most serious failures a Linux system administrator can face. The key takeaways from this incident are:

Backups are not optional — this entire incident would have been resolved in minutes with a good backup
Know your recovery tools — understand the limitations of initramfs before you need it
iLO/IPMI is your lifeline — invest time in learning your server's management interface
Large filesystems need special care — e2fsck on a 1TB+ volume needs RAM, swap, and time
Document everything — credentials, architecture, and recovery procedures must be documented and accessible

If you find yourself in a similar situation, I hope this article saves you some of the hours I spent learning these lessons the hard way.

If this article helped you, please clap and share. If you have questions or have been through a similar experience, leave a comment below.

Tags: #Linux #OpenStack #SysAdmin #DevOps #DisasterRecovery #Ubuntu #LVM #Filesystem #HPE #iLO

CKA

Oyoh Edmond — Tue, 15 Apr 2025 19:17:03 +0000

Cloud Resume Challenge- Week 1

Oyoh Edmond — Mon, 08 Jan 2024 14:53:56 +0000

Introduction

In this piece, I'll delve into the AWS Cloud Resume Challenge, offering firsthand insights from my experience. In the ever-evolving tech scene, effectively highlighting your skills has become crucial. The challenge(Cloud resume challenge) serves as a fantastic avenue to exhibit your proficiency in cloud technologies and craft an impactful online resume. Conceived by Forrest Brazeal, this initiative presents a distinctive chance to showcase your expertise. Come along as I lead you through the initial phases of this engaging project.

Setting the Foundation with HTML and CSS:

During this stage of the project, I utilized a pre-made template(html5up) and made adjustments to both the HTML and CSS elements to customize it according to my preferences. To complete this task, familiarity with HTML and CSS is essential since a significant portion of the code is already in place. Your main responsibility is to edit the HTML files with your personal information. I then uploaded the modified code to (my GitHub repository) and leveraged GitHub Pages to serve and host the static content.
My Website

Upload Resume Folder to Amazon S3:

To proceed with this phase, I created an AWS account, set up an IAM group with a policy specifically tailored for this project, generated an IAM user, and then added the user to the previously configured IAM group. This series of steps enabled me to access the management console, where I initiated the creation of a bucket and uploaded the folder containing my HTML and CSS files to the S3 bucket.

Set up CloudFront, Amazon Route 53, and Certificate Manager:

I enhanced the performance of my resume website by utilizing CloudFront, with S3 as the origin, and strengthened security by obtaining an SSL certificate through AWS Certificate Manager to ensure the website's safety. Furthermore, I configured my custom domain using Amazon Route 53 to improve the overall presentation of my resume website.

Conclusion

I acquired valuable insights throughout the completion of the task in the first week. Despite encountering certain challenges, I managed to overcome them by engaging in troubleshooting and conducting comprehensive research. Consequently, I successfully addressed and resolved the issues that arose.

How Install and Configure the AWS CLI on Windows Terminal

Oyoh Edmond — Wed, 15 Nov 2023 15:50:33 +0000

Lab overview

The AWS Command Line Interface (AWS CLI) is a command line tool that provides an interface for interacting with products and services from Amazon Web Services (AWS).

You can install the AWS CLI on your local machine or a virtual machine such as an Amazon Elastic Compute Cloud (Amazon EC2) instance.

In this activity, you install and configure the AWS CLI on a Red Hat Linux instance because this instance type does not have the AWS CLI pre-installed. Some instance types, such as Amazon Linux, do come pre-installed with the AWS CLI.

During this activity, you establish a Secure Shell (SSH) connection to the instance. You configure the installation with an access key that can connect to an AWS account. Finally, you practice using the AWS CLI to interact with AWS Identity and Access Management (IAM).

When you finish the activity, it will reflect the following diagram:

In the preceding diagram, you can access the AWS Cloud through an SSH connection. Within the AWS Cloud, a virtual private cloud (VPC) with a Red Hat EC2 instance is configured with the AWS CLI. IAM is configured, and you use the AWS CLI to interact with IAM.

Objectives

Install and configure the AWS CLI.
Connect the AWS CLI to an AWS account.
Access IAM by using the AWS CLI.

Accessing the AWS Management Console

Based on the task given, a lab was provided.

Connect to the Red Hat EC2 instance by using SSH

In this task, an EC2 instance was already in existence.

As a Windows user, PuTTy was recommended for usage.

PuTTY is an SSH and telnet client, developed originally by Simon Tatham for the Windows platform.

Download PuTTY to use an SSH utility to connect to the EC2 instance. If you do not have PuTTY installed on your computer, download it.
A private key (.ppk format) was provided based on the already existing EC2 instance created.

A Public IP was provided based on the already existing EC2 instance created.

After downloading PuTTY, Open PuTTy.exe
Configure the PuTTY timeout to keep the PuTTY session open for a longer period of time:
- Choose Connection.
- For Seconds between keepalives, enter 30.

Configure your PuTTY session:
- Choose Session.
- For the Host Name (or IP address), enter the PublicIP address that you copied from the previous steps.

In PuTTY in the Connection list, choose SSH to expand it.
Choose Auth, then choose Credentials
Choose Browse.
Browse and select the Private key file that you downloaded.
To choose the file, choose Open.
Choose Open again.

Finally, click Open.
When prompted with login as, enter ec2-user and press Enter.
This step connects you to the EC2 instance.

Install the AWS CLI on a Red Hat Linux instance

From the terminal window to install the AWS CLI on a Red Hat Linux instance.

To write the downloaded file to the current directory, run the following curl command with the -o option:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

To unzip the installer, run the following unzip command with the -u option. In this command, the unzip command prompts you to overwrite any existing files. To skip these prompts, the command includes the -u option.

unzip -u awscliv2.zip

To run the install program, run the following command. This sudo command grants write permissions to the directory. The installation command in the code snippet uses a file named install in the unzipped AWS directory to install the AWS CLI.

sudo ./aws/install

To confirm the installation, run the following command:

aws --version

The following is an example of the output:

aws-cli/2.13.35 Python/3.11.6 Linux/4.14.327-246.539.amzn2.x86_64  exe/x86_64.amzn.2 prompt/off

Note: The version numbers that are installed change over time and might not reflect this example.

To verify that the AWS CLI is now working, run the following AWS help command. The help command displays the information for the AWS CLI.

aws help

At the: prompt, enter q to exit.

Configure the AWS CLI to connect to your AWS Account

In the SSH session terminal window, run the configure command for the AWS CLI:

aws configure

At the prompt, configure the following:

The following below was provided as a result of the existing Ec2 instance.
- AWS Access Key ID: Copy and paste the AccessKey value into the terminal window.
- AWS Secret Access Key: Copy and paste the SecretKey value into the terminal window.
- Default region name: Enter us-west-2
- Default output format: Enter json

In the terminal window, test the IAM configuration by running the following command:

aws iam list-users

Fellow me for more practical steps

Overview of AWS Security Group

Oyoh Edmond — Tue, 25 Jul 2023 01:17:36 +0000

In the world of Cloud Computing, we design everything with Security in mind. While you may have heard about AWS Security Groups — have you ever stopped to think about what a security group is, and what it actually does?

What is a Security Group?

A security group acts as a virtual firewall for your instance to control inbound and outbound traffic. For each security group, you add rules that control the inbound traffic to instances, and a separate set of rules that control the outbound traffic.

Security groups are acting as a "firewall" on EC2 instances.
They regulate:

Access to ports
Authorized IP ranges (IPv4 and IPv6)
Controls inbound network (from other to the instance)
Controls outbound network (from instance to other)

What to know about Security Groups?

It can be attached to multiple instances within a security group.
Locked down to a region / VPC combination.
It lives "outside" the EC2 - if traffic is blocked the EC2 instance won't see it.
If an application is not accessible (time out), then it's a security group issue.
If an application gives a connection error, then it's an application error, or the application isn't launched.
By default, all inbound traffic is blocked and all outbound traffic is authorized.

NB: It is good to maintain one separate security group for SSH access

Security group rules

Inbound: Inbound rules control the incoming traffic to reach the instances that are associated with the security group.

Outbound: Outbound rules control the traffic to leave the instances that are associated with the security group.

NB: For each inbound and outbound rule, you can add, update, or remove rules that take effect immediately in all instances associated with the security group.

Each rule consists of the following key elements:
Name: The name for the security group (for example: webserver-security-group, database-security-group).

Protocol: The network protocols to allow (for example: TCP, UDP, ICMP, etc.).

Port range: A specific port or the range of ports to allow traffic on (for example: 80, 22, 7000-8000, etc.).

Source or destination: The source (inbound rules) or destination (outbound rules) for the traffic that can specify by an IP, IP range, or other security groups.

Ports to know

22 = SSH (Secure Shell) - Log into a Linux instance
3389 = RDP (Remote Desktop Protocol) - Log into a Windows instance.
22 = SFTP (Secure File Transfer Protocol) - Upload files using SSH.
21 = FTP (File Transfer Protocol) - Upload files into a file share.
80 = HTTP (Hyper Text Transfer Protocol ) - Access unsecured websites.
443 = HTTPS (Hyper Text Transfer Protocol Secure) - Access secured websites.

Let's practice on our AWS console

1. Create an EC2 instance and launch it, check if it is running.

2. Under Network & Security, click Security Groups
You see two Security Groups, Default Security Group created by default and Launch-wizard-1 Security Group created when the EC2 instances was launched.

3. Click on the security group to edit the inbound rules

4. Inbound has two rules
HTTP which allows port 80 in the EC2 instances from anywhere
SSH which allows port 22 in the EC2 instances from anywhere

5. Add more rules

6. Delete the rule

7. Edit the rule

Cloud to the world

How To Create an EC2 Instance on AWS

Oyoh Edmond — Sat, 08 Jul 2023 08:07:31 +0000

In this guide, i will be demonstrating the process of setting up a Linux virtual machine on the cloud, specifically using Amazon EC2. Amazon Elastic Compute Cloud (Amazon EC2) is a service that offers scalable and secure compute resources in the cloud, functioning as Infrastructure as a Service (IaaS).

Pre-requisites

An AWS account
Familiarity with the AWS management console

Create EC2 Instance

- Sign in to the Amazon Management Console.

- From the navigation bar, choose a suitable geographic region that meets your requirements.
Tip: Choose a nearer geographic region from your location to get faster responses

- Search EC2 in the search bar, and click on it.

- In the EC2 dashboard, select instances.

- Click Launch instances.

- Give your instance a name.

- Select the appropriate AMI for your instance.

An Amazon Machine Image (AMI) is a pre-configured virtual machine image that is used to create EC2 instances.

It contains the software configuration (operating system, application server, and applications) required to easily and quickly launch your instance that is pre-configured with their desired operating system and application stack, saving significant time and effort in configuring and deploying new instances.

- Select an instance type and a key pair.

NB: Read for more

- Select create new key pair.

A key pair is a set of security credentials that are used to securely authenticate a user's login credentials to an EC2 instance.

It consists of two parts: a public key and a private key. The public key is shared with the instance, while the private key is kept secure by the user.

- Enter Key pair name.

AWS key pairs can be downloaded in .pem or .ppk formats.
The .pem format is commonly used with SSH clients on Unix/Linux systems.
The *.ppk file format * is a proprietary format used by PuTTY, a popular SSH client for Windows.

It is important to keep your private keys secure and should not be shared with others as anyone with access to your private key can gain access to your instance.

Key pair types
RSA encryption works for both windows and MacOs/Linux instances. ED25519 keys work with Linux and Mac instances only.'

- Enter your preferred network settings.

Your network settings include setting your VPC (Virtual Private Cloud) which is a virtual network infrastructure provided by AWS that allows users to launch resources in a logically isolated section of the AWS Cloud.

Your VPC enables you to create your own IP address ranges, subnets, route tables, security groups, etc.

AWS assigns a default VPC to every account, we'll use the default VPC and create a new security group that only allows SSH access to the instance.

Select Allow HTTPS traffic from the internet
Select Allow HTTP traffic from the internet to allow access to the EC2 webpage.

- Select the number of instances you want to create and review your selection, then click launch instance.

- The instance is running and has passed the checks

Conclusion
In this article, we discussed how to create an EC2 instance on AWS.

We covered the basic steps involved in launching an EC2 instance, including selecting an Amazon Machine Image (AMI), choosing an instance type, configuring security settings, and creating a key pair to connect to the instance.

We also discussed some important concepts related to EC2 instances, such as instance types, key pairs, and the free tier offering. Additionally, we provided a brief overview of VPCs and their importance in creating a secure and isolated environment in the cloud.

If you are new to AWS or cloud computing, creating an EC2 instance can be a great way to get started and learn about the benefits of cloud computing.

So, if you're interested in trying it out for yourself, sign up for an AWS account and launch your own EC2 instance today.

Amazon EC2 Basics

Oyoh Edmond — Tue, 04 Jul 2023 00:40:00 +0000

Hello there! I'm Edmond, an advocate of transparent learning, where I openly share my knowledge and experiences with the community while eagerly absorbing insights from others. Currently, I'm working on a blog post focused on a subject within AWS Cloud Computing.

What is Cloud computing?

Cloud computing provides users with the ability to access abundant computing resources via the internet, surpassing the limitations of individual hardware. It encompasses crucial resources like processing power, memory, and other components necessary for successful program execution. This scalability and accessibility enable users to leverage larger resource quantities than what a single local server or laptop can accommodate, resulting in enhanced computational capabilities.

Compute resources

Compute resources are measurable quantities of compute power that can be requested, allocated, and consumed for computing activities. Examples of compute resources are:
CPU and Memory.

CPU: Application developer can specify how many allocated CPUs are required for running their application and to process data. It is measured in millicores.
Memory: It is responsible for holding data that is actively being processed by the CPU, as well as the program code and variables necessary for program execution.

AWS compute options

Amazon Web Services (AWS) offers a broad range of functionality for your compute workloads or services.
For compute in AWS, the three most commonly used services are as follows:

Compute (AWS EC2)
Container services (AWS ECS & EKS)
Serverless services (Lambda)

Each of these options serves a specific function, provides configurable options, and exists to meet a variety of workloads within the AWS Cloud.

Amazon EC2(Elastic Compute Cloud)

Virtual machines(VMs) are basic building blocks that gets computing power from the cloud. A VM is software that can perform all the functions as a physical computer, including running applications and operating systems. It is a digital version of a physical computer.

In AWS compute services, VMs are called instances.

Amazon EC2 is a virtual machine that provides a diverse selection of instance types that can be tailored with different combinations of CPU, memory, storage, and networking resources. This allows users to configure instances according to their specific workload needs. The variety of options enables scalability and efficient resource management for applications hosted on Amazon EC2.

Instance types

Instance types are grouped together into instance families. Each instance family is optimized for specific types of use cases.

Instance families have sub-families, which are grouped according to the combination of processer and storage used.

A virtual central processing unit (vCPU) is a measure of processing ability. For most instance types, a vCPU represents one thread of the underlining physical CPU core. For example, if an instance type has two CPU cores and two threads per core, it will have four vCPUs.

The AWS instances are currently categorized into five distinct families.

General Purposes Instances
It provides a balance of compute, memory, and networking resources and can be used for a wide range of workloads. You can use General Purpose Instances for gaming servers, small databases, personal projects, etc

NB: If high performance CPUs are not required for your applications, you can go got General Purpose Instance.

The following diagram shows the icons for the general purpose family and sub-families as of this publication.

Compute Optimized Instances
it is an ideal for compute-bound applications that benefit from high-performance processors. Instances belonging to this family are well suited for compute-intensive operations, such as the following:

Batch processing workloads
Media transcoding
High performance web servers
High performance computing (HPC)
Scientific modeling
Dedicated gaming servers and ad server engines
Machine learning (ML) inference

NB: The following diagram shows the icons for the compute optimized family and sub-families as of this publication.

Memory Optimized Instances
They are designed to deliver fast performance for workloads that process large data sets in memory.
Memory here defines RAM which allows us to do multiple tasks at a time. Memory is a temporary storage area.
It loads from storage, holds the data, and process it before the computer can run it.

NB: The following diagram shows the icons for the memory optimized family and sub-families as of this publication.

Storage optimized instances
They are designed for workloads that require high, sequential read and write access to very large data sets on local storage.
Distributed file systems, data warehousing applications, and high frequency online transaction processing (OLTP) systems are examples of workloads that are suited for storage instances.
They are optimized to deliver tens of thousands of low-latency, random input/output (I/O) operations per second (IOPS) to applications.

NB: The following diagram shows the icons for the storage optimized family and sub-families as of this publication.

Accelerated Computing Instances
It uses hardware accelerators, or co-processors, to perform some functions more efficiently than is possible in software running on CPUs. Examples of such functions include floating point number calculations, graphics processing, and data pattern matching. Accelerated computing instances facilitate more parallelism for higher throughput on compute-intensive workloads.
Graphics applications, game streaming, and application streaming are all good candidates for Accelerated Computing Instances.
Data pattern matching can be done more efficiently with this instance type.

NB: The following diagram shows the icons for the Accelerated optimized family and sub-families as of this publication.