Forem: Ben Mezger (seds)

openpgp

Ben Mezger (seds) — Mon, 24 Jul 2023 07:54:40 +0000

$2a$11$gPkdE5FvuGbANwGCajAcZem3IvLCCAN4h/gbZKhUQn9XrCCYzb7ma

Automating and testing dotfiles with Ansible and Github's workflow

Ben Mezger (seds) — Sun, 09 Aug 2020 17:47:23 +0000

I run multiple Archlinux machines at home and an OSX machine for work, so I need to keep my system configuration in sync. I have a lot of applications tinkered for my workflow — I don’t want to switch computers and have to reconfigure something every time I change in another machine. stow or a bash script could allow us to manage files, but they are limited when a specific non-scripting configuration requires different settings for a specific machine.
For example, my Alacritty requires different settings for OSX and Linux.

I ~~solved~~ this issue by keeping multiple branches for multiple machines, but that sucked when I updated a configuration and I needed to sync with my main branch (Linux). Handling merge conflicts or cherry-picking commits was tiring. Eventually, after trying out stow for years and bash scripts, I came up with a neat workflow

Chezmoi to the rescue

Chezmoi solved the branching issue as it allows me to handle multiple configurations for different machines, by using templates of non-scripting configurations for inserting context before copying the file. This solves the headaches with merge conflicts and having to maintain multiple branches. I now need one branch for keeping my files and let Chezmoi handle the configuration context and changes when required.

Handling installation with Ansible

Yet, I still have the dependency problem when reinstalling my operating system, I almost always remember the applications I use, but not always all their non-required dependencies, also, as I keep multiple OSes, some program differ. Keeping a bash script in my dotfiles could solve this, but that would require some time to maintain an installation script for each OS.

I choose Ansible to handle my dependency installation. I keep 4 Ansible roles, osx for anything related to osx installation, archlinux for arch related stuff, linux for common Linux commands and common, shared between osx and archlinux (it takes care of cloning external repositories, applying Chezmoi, etc).

This allows me to easily install my configuration in a new machine and update any new dependency I added to my toolkit.

Continuous Integration

I mentioned before that I am working more from OSX due to my job, so I tend to miss Archlinux package updates/removals, and I always end up having to tinker my Archlinux Ansible roles with the new packages or configuration when they change, again, tiring. I decided to install my dotfiles on a CI, using Github Workflow.
My current Github workflow runs 2 jobs, one for Archlinux (running on Ubuntu with Docker) and another with MacOS. Both runs Ansible, installs all dependencies, does some system checks, and finally caches the result.

This allows me to keep up with Archlinux/OSX updates a bit faster and making sure my Ansible is fully functional — if one of the CI fail, something happened with the dependencies or my configuration.

Going down the hole

With this dotfile structure, I can easily write test scripts for asserting if files were correctly copied, packages installed, etc. For example, I could write a Python script which asserts if files are correctly in place, I could then set this script to run after Ansible did it’s job.

import pathlib
import os
import platform

HOME = pathlib.Path(os.getenv("HOME", ".")).absolute()

def osx_verify_copied_files(home: str = HOME,
                            required=(".zshrc", ".zshenv")):
    for f in required:
        print(f"Checking if {HOME.joinpath(f)} exists")
        assert HOME.joinpath(f).exists() == True, "{f} does not exist"

def osx_verify_hostname(hostname):
    print(f"Verifying if hostname '{hostname}' is set")
    assert platform.node() == hostname, "Hostname does not match"

osx_verify_copied_files()
osx_verify_hostname("benmezger-ckl.local")

Checking if /Users/benmezger/.zshrc exists
Checking if /Users/benmezger/.zshenv exists
Verifying if hostname 'benmezger-ckl.local' is set

Emacs org mode

I live in Emacs, this blog is written in Org mode and Hugo, my snippets are stored in an org file and my code is written in Emacs. To ease my life, I decided to keep a COMMAND.org file with general commands I might need when tinkering with my dotfiles. Org mode supports literate programming, so keeping a COMMAND.org file allows me to execute the commands in-buffer. I simply C-c C-c in the snippet and let org mode make it happen. This is nice when I am tinkering my dotfiles and I need to apply changes with Chezmoi, for example.

Conclusion

System configuration is important for a stable workflow, as we don’t want to change much when switching machines, Ansible allows us to keep multiple installations up to date and Chezmoi allows handling these configurations file properly. Keeping your dotfiles in a CI sounds overwhelming, however, it does guarantee your installation scripts are fully functional against multiple OSes and you will know when something bad happened.

benmezger / dotfiles

My dotfiles

README

Installing with Ansible

Requirements

Ansible

Install Ansible galaxy requirements with ansible-galaxy install -r requirements.yml

OSX

For MacOS, run ansible-playbook -i inventory osx.yml

Linux

For Linux, run ansible-playbook -i inventory archlinux.yml

Installing with Chezmoi

Requirements

Chezmoi

Installing

Run chezmoi init https://github.com/benmezger/dotfiles.git (see reference for more options) to checkout the repository and any submodules. Finally, run chezmoi apply to move changes to your home directory.

When installing without Ansible, chezmoi wont install any external dependencies, so make sure you have all the required dependencies before runnign chezmoi apply.

Commands

For commands related to this repository, see COMMANDS

Dependencies

Fonts
- Inconsolata-DZ
- Hack
ZSH
- Antibody
- FZF
Emacs
- Doom Emacs
System utilities
- Wakatime
- Emacs
- Chezmoi
- Ripgrep
- pyenv
- Hugo
OSX
- Homebrew
- Mas
Linux
- i3wm
- Aura (Archlinux)

Commands

TODOS

View on GitHub

Attaching Jira issues to commit

Ben Mezger (seds) — Sun, 24 May 2020 06:02:01 +0000

TL;DR: Dynamically attach Jira attributes to commit body using git-hook.
Check the project’s README.

The place I work at requires Jira story ID and task ID attached to the commit body. Initially, I was attaching the ID to the commit body manually, by checking either my previous commit or opening up the Jira board, however, after working some hours I was easily forgetting to attach the IDs to the commit and getting annoying having to either reword them and perhaps having to lookup Jira again.

This was tedious and frustrating, so I wrote a Git hook using pre-commit to handle and install the hook. Our workflow requires the task ID to be attached to the branch same, like so: SKYR-123_branch-description, so Jira is capable of logging commits related to task branches¹.
This makes it easy for the hook to know which task are you working on before writing to the commit body. As it checks whether you are in a task branch or any other branch. The Jira ID branch is configurable by specifying a regex for the Jira ID, like so: SKYR-\d+.

Git provides a pre-commit-msg hook, which prepares the default commit message before prompting the user for the commit description/body. To allow extensibility, the hook handles custom Git template with Jinja, so each project may have a custom commit template.

For example, the following template will write the task ID, story ID (if any) and task description.

Task description: {{ summary }}
Jira task ID: {{ key }}
{% if parent__key %}Jira story ID: {{ parent__key }}{% endif %}

The fields are related to Jira’s REST fields. Inner fields such as parent.key should replace the dot (.) with a double underscore (__).

I named the project GJira , as of Git-Jira.

benmezger / gjira

Git hook for adding Jira issues and stories to commit body

GJira

GJira fetches a Jira issue based on the current branch name and appends to the commit body.

GJira allows dynamic branches to be set per project and commit template by using dynamic Jira attributes.

Why?

This came as a requirement from projects I work where makes heavy use of Jira Jira has support for Smart commits which we use in all projects where I work. This allows us to dynamically move cards around depending on their status, and link commits and branches to them.

It's a neat feature for developers and projects managers, as it removes the overhead from developers by having to move cards around manually on each push and gives the project manager an insight of the current development workload.

Requirements

Python >=3
pre-commit

Setup

Git commit template

GJira requires a commit template file. GJira supports Jinja2, which allows customizable templates based on Jira context. For example:

…

View on GitHub

Original post: https://seds.nl/posts/attaching-jira-issues-to-commits/

https://www.atlassian.com/blog/bitbucket/integration-tips-jira-software-bitbucket-server ↩

Patching requests HTTP hooks with custom arguments

Ben Mezger (seds) — Thu, 07 May 2020 22:42:31 +0000

Original post: https://seds.nl/posts/http-hooks-with-custom-arguments/

I am working on a project where we have lots of functions integrated with external APIs. Functions where HTTP requests get dispatched, we log the current caller’s name, headers, and data (if any) in case we need to debug anything. For example:

import logging
import requests
LOGGER = logging.getLogger("external")

def create_external_services():
    ...
    LOGGER.info(f"[func] | Request {data}")
    response = requests.post("..")
    LOGGER.info(f"[func] | Response {response.status_code} {response.text}")
    ...

def delete_external_services():
    LOGGER.info(f"[func] | Request {data}")
    response = requests.delete("..")
    LOGGER.info(f"[func] | Response {response.status_code} {response.text}")

So far so good, but it gets ugly when you have a bunch of functions logging HTTP one or multiple requests and responses.

Python’s requests has a hook system that allows us to manipulate portions of the request process or signal event handling. However, the hook is unable to receive a custom argument. requests requires hooks to have the current argument definition of: def hook(response, *args, **kwargs), however, you are unable to pass custom kwargs to the hook as requests raises a TypeError if any kwarg is not recognized.

The way I solved this issue was by first creating a hook patch decorator.

from typing import Callable
import functools
import logging

def patch_http(
    logger: logging.Logger = None,
    level: int = logging.INFO,
    log_hook: Callable = log_hook,
) -> Callable:
    if logger is None:
        logger = logging.getLogger("http.client")

    def decorate_http(func):
        @functools.wraps(func)
        def log_wrapper(*args, **kwargs):
            # fake func and logger attribute to log_hook
            log_hook.func = func
            log_hook.logger = logger
            log_hook.level = level
            return func(*args, **kwargs)

        return log_wrapper

    return decorate_http

This decorator decorates functions in which requests is used. It allows us to use a custom logger, log level, and log_hook function if required. log_wrapper creates 3 dummy attributes: func which holds the address of the caller, the logger, and the log level.
The log_hook requires some hacking, as we might want to use it without the need of a decorated function.

def log_hook(req, *args, **kwargs):
    if not hasattr(log_hook, "func"):
        log_hook.func = None
    if not hasattr(log_hook, "logger"):
        setattr(log_hook, "logger", logging.getLogger("http.client"))
    if not hasattr(log_hook, "level"):
        setattr(log_hook, "level", logging.INFO)

    log_hook.logger.log(
        log_hook.level,
        "[{}] | Request | Payload: {}".format(
            log_hook.func.__name__ if callable(log_hook.func) else "",
            req.request.data if hasattr(req.request, "data") else {},
        ),
    )

    log_hook.logger.log(
        log_hook.level,
        "[{}] | Response status {} | Response {}".format(
            log_hook.func.__name__ if callable(log_hook.func) else "",
            req.status_code,
            req.content,
        ),
    )
    return req

The first lines are what allows the use of the hook regardless of the decorator, with the downside of not having a function caller named. The actual call to logging is done by using the attribute previously created by our patch_http decorator, however, if no decorator is used, it defaults to a predefined http.client logger.

Now the actual change to our code:

import logging
import requests
LOGGER = logging.getLogger("external")

@patch_http(logger=LOGGER)
def create_external_services():
    response = requests.post("..", hooks={"response": log_hook})

@patch_http(logger=LOGGER)
def delete_external_services():
    response = requests.post("..", hooks={"response": log_hook})

I am still not entirely convinced if using a decorator for patching the hook is the cleanest way, however, it allows us to modify the logging messages for all requests in one place without having to duplicate code or easily add custom logic to all requests.

For example, say we want to log only if a 404 HTTP status code gets returned in create_external_services. We could modify our decorator to create an expected_statuses and check the response status code in log_hook before logging.

from typing import Callable, Tuple
import functools
import logging

def patch_http(
    logger: logging.Logger = None,
    level: int = logging.INFO,
    log_hook: Callable = log_hook,
    expected_statuses: Tuple[int] = (200, 201)
) -> Callable:
    if logger is None:
        logger = logging.getLogger("http.client")

    def decorate_http(func):
        @functools.wraps(func)
        def log_wrapper(*args, **kwargs):
            log_hook.expected_statuses = expected_statuses
            log_hook.func = func
            log_hook.logger = logger
            log_hook.level = level
            return func(*args, **kwargs)

        return log_wrapper

    return decorate_http

def log_hook(req, *args, **kwargs):
    ...
    if not hasattr(log_hook, "expected_statuses"):
        setattr(log_hook, "expected_statuses", (200,))

    if req.status_code in log_hook.expected_statuses:
        log_hook.logger.log(
            log_hook.level,
            "[{}] | Request | Payload: {}".format(
                log_hook.func.__name__ if callable(log_hook.func) else "",
                req.request.data if hasattr(req.request, "data") else {},
             ),
        )
        ....
    return req

Keeping up with my goals by having daily/weekly goals

Ben Mezger (seds) — Mon, 27 Jan 2020 22:42:16 +0000

I tended to procrastinate a lot, especially when I am unhappy with work, relationships or with the task I am currently working on. This has led me to some serious procrastination issues, where I ended up not doing anything I had to, nor anything I've wanted to do.

By procrastinating, I ended up procrastinating on goals I wished to accomplish, like reading more books, improving my career or anything related to my life goals.

I started using some daily/weekly "templates", which has helped me overcome my procrastination during hours I should not be procrastinating. Although procrastinating isn't necessarily bad, these tips I try to apply during my day/week routine, so I don't slip on hours I should not.

Preparing my week

Every Sunday evening, I set up an hour to establish my 3 weekly goals I want to accomplish during the coming week. There may be smaller goals, meaning they take little effort to accomplish but requires some days, like finishing a book. However, there may be a bigger goal you want to accomplish that might take a month to do so, such as delivering a project you are working on. The latter should be divided into subgoals, such as implementing the payment system in week 1.

For example:

# Jan 27 - 31
* Implement the payment system for project Y
* Finish book XYZ
* Meditate before sleep

That's it. It's simple, straight to the point and doesn't feel or look scary or anything you would want to run away from. Keep in mind that these goals are goals for improving yourself and making you feel better, and not designed for work-related, although you could adapt a version alike.

Preparing my day

Every day in the evening, I take a few minutes to think of 3 things I want to accomplish tomorrow. I think of future goals I would like to accomplish, that I could very well be starting tomorrow. Such as improving my algorithm thinking, or reading some paper related to my field.

# Jan 27
* Read 25 pages of book XYZ
* Swimming classes
* Solve 3-4 work code-related tasks

Keeping it up

Keeping this up is easy to forget, so you have to get write them down, I write down my daily goals in my todo app, which makes it easier for me to keep track and read it during the day. My work uses Google Calendar for meetings and events, so I sync todo list with Google calendar, so I only have to check my goals and todos in one place.

I keep my weekly goals written down in a file. I do that because I might add some notes related to that week so I can check later.

Conclusions

By making sure I had all these goals set during my week, I've been able to confront my goals and accomplish them every day and every week, without feeling the guilt of procrastinating when having goals set or feeling time pressure. Three goals daily and weekly allows me to have time for other things which may not necessarily be a goal.

Finding exposed .git repositories

Ben Mezger (seds) — Thu, 23 Jan 2020 14:08:27 +0000

Developers use git to version control their source code. We all do, in fact, this blog is currently being versioned by git. However, we not only use git to version control, but also to deploy applications. Usually we push the new code to a remote server, where the server takes care of testing the code and then deploying the application. There are different ways of deploying an application, but this is one of them.

Some developers/sys-admins simply clone the repository to their server, then simply point their web-server to that directory. Most of the VCS keep a hidden directory at the root of the project. git keeps a .git directory at the root of the repository, where all the information of that project is stored, such as logs, versions, tags, configs, previous revisions and so on.

Not only some developers/sys-admins clone the repository in their server, but they also point their web-server to that directory. Not only that, they also sometimes change the source to set some configurations files with sensitive informations (email/password, and so on).

If the web-server is pointing to the git repository and has directory listing enabled, we could could download the .git using wget/curl recursively, then simply checkout a master and voilà.

For example, say https://seds.nl/.git exists, containing all the objects of this blog, we could simply run wget --mirror -I .git https://seds.nl/.git to download the repository.

How to find exposed git repositories

We can simply try out every URL we know by adding /.git at the end of the TLD.

Just kidding.

An easy way of finding websites which currently expose .git is using Google D0rks. If you are not familiar with Google D0rks, it’s basically a few operators Google offers you to filter out a few queries. Here is a list of a few of them. The one we need is the intext: operator.

intext:"Index of /.git"

This query makes use of the intext operator. It allows us to ask Google to find all the pages that have a specific word in the body somewhere forcing inclusion on the page (source)

Holy sh1t. That’s ~89,900 results from Google. Can you imagine how much sensitive information there must be?

How to fix this

Well, first of all, find a better way to getting your source code to a remote server and simply pointing your web-server to that directory. If you don’t feel like finding a better way, or just want to keep things simple, here is what you need to do.

Nginx

Add the following telling Nginx to deny any request to a .git directory:

location ~ /.git/ {
    deny all;
}

Apache

Add the following telling Apache to deny any request to a .git directory:

<DirectoryMatch "^/.*/\.git/">
    Order deny,allow
    Deny from all
</Directory>

The only question that remains is: Is there anyway you could extract a .git from a web-server that has directory listing disabled? I haven’t looked much into it, but I wonder if there is anyway we could use git against itself.