Forem: beto-bit

A Function Pointer Wrapper in C++

beto-bit — Tue, 14 Nov 2023 04:20:01 +0000

So you probably already know what are function pointers and how to use them. You also probably know that function pointers have a really ugly syntax.

int func(int a, double b);

int foo() {
    int (*ptr)(int, double) = func;
    return ptr(3, 0.4);
}

But what to do when you want to return a function pointer?
You could use a typedef in order to do that.

int fun(int a, double b);

typedef int (*func_type)(int, double);

func_type bar() {
    return func;
}

But if you don't want to name your function type, you end with something like this.

// The function name is baz and it takes no arguments!
int( *baz(void) )(int, double) {
    return func;
}

But we are talking about C++, so you have probably just used auto, or even worse, returned a slow std::function.
Instead of that, you can use this zero cost template.

It looks overcomplicated (as everything in C++) but is basically a nice type alias with CTAD. So you can do this.

FuncPtr<int(int, double)> foo() {
    FuncPtr temporal = func; // CTAD
    return temporal;
}

I don't really remember when was the first time I saw something like this. So if you find the original, please leave it in the comments! Thanks for reading.

How are threads created in Linux x86_64

beto-bit — Sat, 23 Sep 2023 02:05:58 +0000

Introduction

So I was trying to use atomics in C. Got a little working example.

// multithrd.c
#include <stdio.h>
#include <threads.h>
#include <stdatomic.h>

atomic_int acnt;
int cnt;

int f(void* thr_data) {
    for (int n = 0; n < 10000; ++n) {
        ++cnt;
        ++acnt;
    }

    return 0;
}

int main() {
    thrd_t thr[10];
    for (int n = 0; n < 10; ++n)
        thrd_create(thr + n, f, NULL);

    for (int n = 0; n < 10; ++n)
        thrd_join(thr[n], NULL);

    printf("Atomic counter is: %u\n", acnt);
    printf("Non Atomic counter is: %u\n", cnt);
}

And, I got results like this

$ ./multithrd
Atomic counter is: 100000
Non Atomic counter is: 98860 # Depends on the weather

Great! But how is a thread actually spawned? We currently don't know many things, but we know something; threads are created by the OS.

Hello, World! (in assembly)

As you may know, a program can do a lot of things by itself. It can add numbers, pass values around functions, compute digits of pi, etc. But it can't print out something to the screen. That's the kernel's job.
Well, a program can print out something to the screen, after all the Linux kernel is a bunch of instructions for the CPU to execute. It just has a lot of privileges. It can turn on and off your USB ports and communicate with your NIC and turn on specific pixels on your screen in order to print "Hello, World!".

So how does a normal program, an userland program, how does it prints something to the screen. Well, let's figure it out writing a x86_64 Linux assembly "Hello, World" program!

We're gonna be using NASM because I don't want to deal with GAS.

; main.asm
section .text

global _start
_start:
    mov rsi, msg    ; msg
    mov rdx, msg_l  ; len
    mov rax, 1      ; write syscall
    mov rdi, 1      ; stdout
    syscall

    mov rax, 60     ; exit syscall
    mov rdi, 0      ; exit code = 0
    syscall


section .data
msg:    db "Hi mum", 10
msg_l:  equ $ - msg

And then compile it!

$ nasm -f elf64 main.asm -o main.o    # Compile it
$ ld main.o -o main                   # Link it
$ ./main                              # Run it!
Hi mum

You can see the syscall instruction there. It basically says to the OS "make this for me, thx". We use it for writing to stdout and exiting the program. So, whenever the program can't do something by itself, say opening a TCP Socket, it asks the OS for it. That's how threads are created! But how to know which syscall does it uses?

Strace Adventures

The man page for strace says "[...] It intercepts and records the system calls which are called by a process and the signals which are received by a process." So this is the tool we need.

So let's run it against our new program!

$ strace ./main
execve("./main", ["./main"], 0x7ffccab40ae0 /* 67 vars */) = 0
write(1, "Hi mum\n", 7)                 = 7
exit(0)                                 = ?
+++ exited with 0 +++

As you can see, it shows us which syscalls were called. execve is the syscall that actually runs the program! It is called by our shell, and, in fact, the second argument is equivalent to argv in a C main function, and the third to envp.

int main(int argc, char *argv[], char *envp[]);

Then we can see the other 2 syscalls we invoked, write and exit.

Stracing `multithrd.c`

So let's dive in and directly strace our program!

$ strace -o calls.strace ./multithrd
$ cat calls.strace
execve("./multithrd", ["./multithrd"], 0x7ffee0802fc0 /* 72 vars */) = 0
brk(NULL)                               = 0x55b1c8444000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fffb92cf8a0) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9dad49e000
...

And it goes like this for 99 lines.

So yeah, let's start for something simpler. What about a program that does nothing? And I will make it without a file for the flex!

$ echo "int main() {}" | gcc -O2 -x c -
$ strace 2>&1 ./a.out | wc -l
34

Much better! 65 lines less! It does a bunch of syscalls even when we are doing nothing, so we will chop the lines that seem similar.

And if you close your eyes, in our multithread program you can notice the clone3 call being invoked multiple times. Let's check how many times.

$ grep -c clone3 calls.strace
10

The same number of threads we create, great!

The `clone3` syscall

So it seems that clone3 is the syscall that creates new threads. Let's use man (better if you use batman) to find about it.

$ man clone3

The man page is quite large. It gives us information about the glibc wrapper, clone and about the syscall itself. Let's check the signature of clone.

int clone(int (*fn)(void *), void *stack, int flags, void *arg, ...);

The man page says "When the child process is created with the clone() wrapper function, it commences execution by calling the function pointed to by the argument fn. [...] When the fn(arg) function returns, the child process terminates".

It also says something really important about the stack: "The stack argument specifies the location of the stack used by the child process. Since the child and calling process may share memory, it is not possible for the child process to execute in the same stack as the calling process. The calling process must therefore set up memory space for the child stack and pass a pointer to this space to clone()".

The stack is basically memory. That's about it. And it's needed for functions to work, because the return address, that is, the address that you should jump to give control back to the caller function, is pushed into the stack. Also you can use it to store local variables and pass more than 6 arguments to functions.

So we just need to pass around values and we have it!

`glibc`'s `clone` function

But how is the function I pass to it executed? I don't see any func_ptr field in clone3 (the syscall)!

Well, clone (the function) has a function pointer argument, let's check what it does.

The source code for that is here.

Surprisingly, it is somewhat well commented! First some sanity checks, then an ABI compliance thing, moving around some stuff to make the syscall, storing the start function and its argument in the stack and when everything is ready, the syscall is made.

And with this, the two threads are in the exact same position. The only differences are the stack pointer (in the child thread is the stack we passed to it) and the rax register.
In the parent thread, the rax register gets the value the thread ID.
In the child thread, it gets to 0.

With this logic, we do a little branching and it just works.

Unfortunately, this process isn't as easy to visualize without doing it inside a debugger. But eventually, it calls the function that you passed to it. And then it just exits with the value you returned from the thread.

So I basically did that by myself!

The repo is here: https://github.com/beto-bit/mt_asm

Aside from not using the standard library, it wasn't actually that hard. The glibc implementation is much more complex, but this a (somewhat) working implementation.

How to make your own Godbolt (for C++ and Rust)

beto-bit — Mon, 28 Aug 2023 16:14:58 +0000

Introduction

I hope you are aware of the existence of Compiler Explorer, a.k.a. Godbolt. It is an incredible tool for inspecting how your code gets translated to assembly, just one layer above machine code.

So, for example, if you want to check if range-based loops in C++ incur some additional runtime cost, you can check the assembly output. But what if you want to run that on your machine? I mean, you can for sure use Compiler Explorer locally, but let's do it the fun way.

Generating Assembly

First, I will assume you have some GNU Make experience, if not, check this resource.
Let's start by creating a src directory and some C++ file.

Then, edit your file to something interesting!

// src/func.cpp

#include <array>

int sum_manual(const std::array<int, 10>& arr) {
    int sum = 0;
    for (std::size_t i = 0; i < arr.size(); ++i)
        sum += arr[i];

    return sum;
}

int sum_range(const std::array<int, 10>& arr) {
    int sum = 0;
    for (int num: arr)
        sum += num;

    return sum;
}

Then we will generate the assembly output using gcc. For this purpose and because writing the whole compile command is error prone, we create a Makefile, and then edit it.

# Makefile
CC := g++
CFLAGS := -O2 -std=c++20 \
          -march=x86-64 -masm=intel -fno-stack-protector \
          -fno-dwarf2-cfi-asm -fno-asynchronous-unwind-tables 

all: out/func.asm

out/func.asm: src/func.cpp
    @ mkdir -p out              # Create directory
    ${CC} -S ${CFLAGS} $< -o $@ # Compile to assembly
    cat $@ | c++filt | tee $@ >/dev/null    # Make it pretty

.PHONY: clean
clean:
    rm -rf out

The flags -masm=intel -fno-stack-protector -fno-dwarf2-cfi-asm -fno-asynchronous-unwind-tables are for getting a nice looking assembly. c++filt is for demangling C++ symbols.

And when running make, it generates a... 168 lines assembly, in my computer. This is because, in theory, you can run GAS (GNU Assembler) on the generated output and get a fully working program.

But we don't want a fully working program, we want a small readable assembly. So we need to filter away the things we don't care about.

Filtering

So we could use grep and inverse filtering.
But basically, we need to:

Include lines that start with .L{number}.
Include lines that start with .quad or .string or .ascii or .asciz.
Remove lines that start with .L{something_else}.
Remove all lines after :0 or .Lframe1.
If the line starts with .LFE change it with a newline. That's because that indicates the end of the function.

And, after trying to make it in C++ and failing to optimize it, I made it in Rust. I'm not gonna explain the code a lot, you just need to know it is quite fast. And it reads and writes to stdin.

// rspper/src/main.rs

use std::io::{self, BufWriter, Write};

type File = Box<dyn Iterator<Item = String>>;

struct AssemblyFile {
    lines: File,
}

impl AssemblyFile {
    fn new(lines: File) -> Self {
        Self { lines }
    }

    fn add_function_separator(self) -> Self {
        let lines = self.lines.map(|l| {
            if l.starts_with(".LFE") {
                String::new()
            } else {
                l
            }
        });

        Self::new(Box::new(lines))
    }

    fn remove_last_section(self) -> Self {
        let lines = self
            .lines
            .take_while(|l| !l.starts_with("0:") && !l.starts_with(".Lframe1"));

        Self::new(Box::new(lines))
    }

    fn remove_directives(self) -> Self {
        let lines = self.lines.filter(|l| {
            !l.starts_with("\t.")
                || l.starts_with("\t.quad")
                || l.starts_with("\t.string")
                || l.starts_with("\t.ascii") | l.starts_with("\t.asciz")
        });

        Self::new(Box::new(lines))
    }

    fn remove_ltags(self) -> Self {
        let lines = self.lines.filter(|l| {
            !(l.starts_with(".LF")
                || l.starts_with(".Lfunc")
                || l.starts_with(".LCFI")
                || l.starts_with(".LEH")
                || l.starts_with(".LHOT")
                || l.starts_with(".LCOLD")
                || l.starts_with(".LLSDA")
                || l.contains("endbr64"))
        });

        Self::new(Box::new(lines))
    }
}

fn main() {
    // Get an interator to stdin
    let input_lines = io::stdin().lines().map_while(Result::ok);

    let input_file = AssemblyFile::new(Box::new(input_lines))
        .add_function_separator()
        .remove_last_section()
        .remove_directives()
        .remove_ltags();

    // Hot output loop
    let stdout = io::stdout().lock();
    let mut stdout = BufWriter::new(stdout);

    for li in input_file.lines {
        stdout.write(li.as_bytes()).expect("stdin broken!");
        stdout.write("\n".as_bytes()).expect("wut");
    }
}

And then I compile it and modify the Makefile.

# Makefile
# <...>

CHOPPER := ./rspper/target/release/rspper

# <...>

out/func.asm: src/func.cpp
    @ mkdir -p out              # Create directory
    ${CC} -S ${CFLAGS} $< -o $@ # Compile to assembly
    cat $@ | c++filt | ${CHOPPER} | tee $@ >/dev/null   # Make it pretty

And finally, the generated good looking assembly output!

sum_manual(std::array<int, 10ul> const&):
    lea rdx, 40[rdi]
    xor eax, eax
.L2:
    add eax, DWORD PTR [rdi]
    add rdi, 4
    cmp rdi, rdx
    jne .L2
    ret

sum_range(std::array<int, 10ul> const&):
    lea rdx, 40[rdi]
    xor eax, eax
.L6:
    add eax, DWORD PTR [rdi]
    add rdi, 4
    cmp rdi, rdx
    jne .L6
    ret

As you can see, using range-based loops generates the exact same assembly!

Getting Rusty

The process isn't that different. We just need some more compilation flags. Edit your Makefile to have this.

#<...>

all: out/func.asm out/rust.asm

out/rust.asm: src/lib.rs
    @ mkdir -p out
    rustc $< -O --crate-type=lib --emit=asm -C llvm-args=-x86-asm-syntax=intel -o $@
    cat $@ | c++filt | ${CHOPPER} | tee $@ >/dev/null

With this, we are basically doing the same as before. It is important to mark functions as pub, because if not they're optimized away by the compiler.

Ending Notes

This has not been profoundly tested, and I may be missing to include/exclude some assembly lines. Also, this makes use of c++filt for C++ and Rust. There is this project, rustfilt, which does the same but for Rust.

I hope you found this usefull, thank you for reading!

Forem: beto-bit

A Function Pointer Wrapper in C++

How are threads created in Linux x86_64

Introduction

Hello, World! (in assembly)

Strace Adventures

Stracing multithrd.c

The clone3 syscall

glibc's clone function

So I basically did that by myself!

How to make your own Godbolt (for C++ and Rust)

Introduction

Generating Assembly

Filtering

Getting Rusty

Ending Notes

Stracing `multithrd.c`

The `clone3` syscall

`glibc`'s `clone` function