Forem: Andreas Augustin

Git - GitHub actions-template-sync

Andreas Augustin — Sun, 11 Dec 2022 21:12:22 +0000

GitHub actions-template-sync

abstract

Sometimes we want to synchronize some git repositories. In general this is not a huge task using commands like

git checkout -b "${NEW_BRANCH}"
git pull "${SOURCE_REPO}" --allow-unrelated-histories

in the target repository.
Often there are many repositories you want to synchronize. E.q. you have created your repositories using a GitHub template repository.
Click the links to get some more information about how to create and use template repositories.
When you are making changes within the template repository often like to propagate those changes into the repositories created from the template.
To automate this task I created a Github action acions-template-sync
Instead of looping through all repositories and execute the commands listed above you can configure the actions-template-sync and receive automatically Pull requests within the target repositories.

Remarks

It is possible to use this GitHub action even if the source repository is not a template repository.
If you are interested in the source code, please checkout the source code repository. Contributions of any kind are welcome.

Usage

The usage is designed to be simple. Just put a actions_templae_sync.yaml definition file into your .github/workflows/ folder.
Please click the following link if you want to get further information about GitHub Actions.
The content of the file should look like the following

name: actions-template-sync

on:
  # cronjob trigger At 00:00 on day-of-month 1. https://crontab.guru/every-month
  schedule:
  - cron:  "0 0 1 * *"
  # manual trigger
  workflow_dispatch:

jobs:
  repo-sync:
    runs-on: ubuntu-latest

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v3
      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@<replace_with_latest_version>
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: <replace_with_source_repo_path>
          upstream_branch: main # defaults to main

Please replace the following variables

Variable	replace with
`<replace_with_latest_version>`	replace with the latest release version
`<replace_with_source_repo_path>`	replace with the path of the source repository (without the hostname)

This is a base configuration used for public github.com repositories. Also private and github enterprise servers are supported.
Please have a look into the available configuration paraemters

Authentication and authorization

Currently with a public repository as a source you do not need to do any changes and the example works out of the box.
For a private source repository currently follwoing AUTHN/AUTHZ methods are supported

SSH

SSH is supported. Here you can find an example for the setup.

SSH keygen

First you need to create an ssh key. Just use the ssh-keygen command. Important do set a password.

Private Key

Add the private key as a secret into the target repository, e.q. with the name SOURCE_REPO_SSH_PRIVATE_KEY.

click settings within the target repo

select Secrets -> Actions

click New repository secret

name the secret (e.q. SOURCE_REPO_SSH_PRIVATE_KEY) and paste the private ssh key

Public key

Add the public key as a deployment key into the source repository.

click settings within the source repository

click deploy keys

click add deploy key

Give it a meaningful name. No need to add write permissions. Paste here your public ssh key

Prepare the action

Now you need to prepare the action. Edit/create the github action config the following way

As a best practice it is recommended that also the source_repo_path is used in a secret.
Create a secret like you have done before.

jobs:
  repo-sync:
    runs-on: ubuntu-latest

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v3
      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@<replace_with_latest_version>
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: ${{ secrets.SOURCE_REPO_PATH }} # <owner/repo>, should be within secrets
          source_repo_ssh_private_key: ${{ secrets.SOURCE_REPO_SSH_PRIVATE_KEY }} # contains the private ssh key of the private repository

GIT - Prevent accidentally pushing credentials

Andreas Augustin — Sat, 27 Aug 2022 20:22:00 +0000

abstract

Sometimes by accident people accidentally push credentials or other sensitive information into a git repository. For example AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID.
Obviously bad people are able to find those secrets in the git repository or history using them to e.q. start cryptominers in those accounts. That leads to super high costs and bills for the poor person who accidentally pushed those secrets.
There is a tool which can prevent you doing such mistakes: git-secrets

Installation

Please check git secrets installation for your OS. Here I show you the installation for Linux.
Just type the following commands

git clone git@github.com:awslabs/git-secrets.git
cd git-secrets
[sudo] make install
make test

Remarks

For users with docker knowledge: have prepared a docker image where git-secret is already installed.
- docker run -v <local_git_repo>:/home/git-secrets/ andyaugustin/git-secrets:main git-secrets
For uninstallation on Linux just remove the copied files again

rm /usr/local/bin/git-secrets
rm /usr/local/share/man/man1/git-secrets.1

Fun part

Now lets check the power of git-secrets.
Please keep in mind that you need to enable the tool for every repository. It will install git hooks into your local repository.

For the turorial we will init a git repo first and bootstrap git-secrets.

mkdir git-secrets-example
cd git-secrets-example
git init
echo "# git-secrets-example" >> README.md
git add .
git commit [-S] -m "doc(): initial commit :star:"
git-secrets --install

As stated in the output we got 3 new files added into our local git repository. Those will prevent us to accidentally commit secrets to the git database. Lets check one of those files

$ cat .git/hooks/pre-commit
#!/usr/bin/env bash
git secrets --pre_commit_hook -- "$@"

This hook will be run every time in the local git repo before the commit is added to the database. This means if the mentioned command will have an error, the commit won't be added to the local database.

Now lets understand some more details. For that we will add some fake AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY into our markdown file and lets try to commit it.
Those keys are really fake, you can try 😈

$ git secrets --register-aws
OK
$ echo "This is fake aws key id AKIAIOSFODNN7EXAMPLA" >> README.md
$ git add .
$ git commit -S -m "doc(): accidentally add keys :alien:"
README.md:4:This is fake aws key id AKIAIOSFODNN7EXAMPLA

[ERROR] Matched one or more prohibited patterns

Possible mitigations:
- Mark false positives as allowed using: git config --add secrets.allowed ...
- Mark false positives as allowed by adding regular expressions to .gitallowed at repository's root directory
- List your configured patterns: git config --get-all secrets.patterns
- List your configured allowed patterns: git config --get-all secrets.allowed
- List your configured allowed patterns in .gitallowed at repository's root directory
- Use --no-verify if this is a one-time false positive

As you can see it is not possible to commit the changes to the git history because there is a AWS KEY in the change.
In some cases you want to commit that key. Maybe because it is an example for a tutorial or anything else.
This is quite easy to establish. Just add the key to a file called .gitallowed.

echo "AKIAIOSFODNN7EXAMPLA" >> .gitallowed
git add .
git commit [-S] -m "doc(): now we are able to establish the commit :star:"

GIT - how and why to sign commits

Andreas Augustin — Fri, 26 Aug 2022 22:35:00 +0000

Git - how and why sign commits

abstract

You should always sign your git commits. Why?

Why to sign commits

The git commits are super easy referenced to a user. Anyone all around the world is able to push commits with another name. The reference is done in the commit message with the user.email.

You can try yourself. Just create a new repository in a folder of your choice.

Let's create locally a git repository.

mkdir sign_commits
cd sign_commits
git init

First lets check our current global settings

$ git config --global user.email
jane.doe@world.universe

and compare them with our local git repo settings

git config user.email

Those are the same. Now lets change the local git repo settings

git config user.email 'john.doe@example.dev'

Remark You can also use git config <--global> --edit to edit all configuration values in editor

Lets now check the current settings and compare them to the global settings.

git config user.email
git config --global user.email

You can see that those differ. This does not give us any value for our current context,
but I wanted to make sure that we do not touch our global settings.

Now lets create a commit and check the history.

echo "# git sign commits" >> README.md
git add .
git commit -m "doc(): add some super nice docs"
git log

Now lets change the user name and the mail and lets do another commit.

git config user.email 'fake.me@fake.me'
git config user.name 'fake me'
echo "if the account exists e.q. on github.com the commit will be assigned to that person" >> README.md
git add .
git commit -m "doc(): add some super nice docs"
git log

You can see that the second commit is assigned to fake me with mail address fake.me@fake.me.
If you are using github.com as your git provider and push the commit and also the mail address fake.me@fake.me is registered to an existing user, the commit will be assigned to that user.

As you can see it is super easy to make commits in names of other persons.
Like an example? Here is a prank of a fake Linus Torvalds stating that linux is deleted.

As you can imagine there are not only pranks. This is a security issue. Imagine you are working in a team on a open source project on github. A teammate (who is a fake) is opening a pull request. You know that the original team mate is a great coder and you don't check in detail the changes and merge them into your main branch. This is obviously an attack vector. How to prevent? start signing your commits today

How to sign commits

It is possible to sign git commits with GPG.
First lets revert our fake changes.

git config --user.name 'john doe'
git config --user.email 'john.doe@example.dev'

Now we need to create a gpg key

gpg --gen-key

Follow the Dialog nd save with O.
Now lets grab the Key-id and add it to the git config

$ gpg --list-key $(git config user.email)

$ git config [--global] user.signingkey <key_id>

Now you are able to sign your commits with the -S flag or you add it to the git settings to make it default behaviour.

git config [--global] commit.gpgsign true

Now lets check what has changed. We first create a signed commit and check the signature.

echo "now the commits are signed" >> README.md
git add .
git commit -S -m "doc(): now with signed commit :star:"
git log --show-signatur

You should now also add the key to your git provider settings so that the git provider will verify the signature.

GIT - multi user

Andreas Augustin — Mon, 22 Aug 2022 20:03:00 +0000

Abstract

When we work on different projects, sometimes we have different git providers (e.q. GitHub, GitLab, Gitea,...).
Also it is possible that we need to use different users for our contribution.
This is obiously possible using git with https.
Because ssh is considered more secure in context of git you need to manage several ssh keys.

It is possible to specify the key you need. For example

GIT_SSH_COMMAND='ssh -i private_key_file -o IdentitiesOnly=yes' git clone user@host:repo.git

As you can see this is not super convenient.
There is an easier way using an ssh config file.

Theory

You are able to configure ssh with a file ~/.ssh/config.
Remark If interested you are able to find all available configration parameters here
You are able to set the identity related to the host with the following parameters

Host <host>:<port>
  HostName <host>
  Port <port>
  IdentityFile <path_to_id_file>

Host: add a name for the host
- HostName: The domain, e.q. github.com, gitlab.com, ...
- IdentityFile: absolute path to the specific private ssh cert
- Port [OPTIONAL]: just use if you have your own git provider service running

example

setup

Get somehow 2 users in 2 different git providers (e.q. GitHub, GitLab,...).
Now lets create 2 different (public/private) ssh key pairs.

ssh-keygen -t rsa -N "" -C "<first_user_mail>" -f '~/.ssh/id_rsa_first'
ssh-keygen -t rsa -N "" -C "<second_user_mail>" -f '~/.ssh/id_rsa_second'

To be able to ssh into the git provider, you need to add the private keys to your git provider settings.

cat ~/.ssh/id_rsa_first.pub
cat ~/.ssh/id_rsa_second.pub

This step is dependent on your git provider.

Now we want to configure git ssh to use those files related to the git provider.

touch ~/.ssh/config

add the following content to the file.
The hostname is dependent to the git provider.

# user 1
Host <first_git_provider>
  HostName <hostname_1>
  IdentityFile ~/.ssh/id_rsa_first

# user 2
Host <second_git_provider>
  HostName <hostname_2>
  IdentityFile ~/.ssh/id_rsa_second

to ~/.ssh/config

test

Now we are set to test the settings.
Lets first create 2 repositories. One in each git provider.

Now lets clone the repositories. cd in a directory of your choice.

git clone <ssh_clone_url_1>
git clone <ssh_clone_url_2>

We want to clone the repositories.
Yay it works 🚀

As you can see when that the right user is auto selected 🚀

GIT - share secrets

Andreas Augustin — Sun, 21 Aug 2022 20:36:00 +0000

Abstract

In some cases you want to securely share secrets with other teammates. Furthermore you want to have a history of those secrets.
For this purpose Git crypt helps you to handle secrets within your git repository.

git-crypt enables transparent encryption and decryption of files in a git repository. Files which you choose to protect are encrypted when committed, and decrypted when checked out. git-crypt lets you freely share a repository containing a mix of public and private content.

Some features

No accidentaly push secrets in clear text
Possible to Share credentials
Put credentials into version control

Installation

First we need to install git-crypt

We need make for the installation.
Please clone the repo and use the following commands.
Please cd in a temporary directory first.

git clone git@github.com:AGWA/git-crypt.git
cd git-crypt
make
make install

Remarks

for details please check install instructions to install git-crpyt.
For users with docker knowledge: have prepared a docker image where git-secret is already installed.
- docker run -v <local_git_repo>:/home/git-secrets/ andyaugustin/git-secrets:main git-crypt

Setup

GPG

We need a key-pair (maybe in reality it is already created for your mail adress)
Use the mail adress which is added to your git user

First we want to check those settings

$ git config --global user.email
john.doe@dummy.fake
$ git config --global user.name
John Doe

keep those entries in your mind or write them down :evil_imp:

Now we want to generate the gpg key.
Type in the name and mail you just received.

gpg --gen-key

git-crypt

We need to have a git repository available. Please create a repository with name git-crypt-test in your favourite Git provider (e.q. GitHub, GitLab, Gitea, ...).
Please clone the repository and cd into it.

git clone <use_your_repo_url>/git-crypt-test
cd git-crypt-test

Now we need to init git-crypt

git-crypt init

Now we want to specify files we want to monitor and handle with git-crypt
That is easy. Therefore we just need to add a .gitattributes file with the files we want to encrypt.

echo "secretfile filter=git-crypt diff=git-crypt\n*.key filter=git-crypt diff=git-crypt\nsecretdir/** filter=git-crypt diff=git-crypt" >> .gitattributes

The content of the .gitattributes file should look now

secretfile filter=git-crypt diff=git-crypt
*.key filter=git-crypt diff=git-crypt
secretdir/** filter=git-crypt diff=git-crypt

This is like a .gitignore file and has the following content.

handle all files with name secretfile with git-crypt
handle all files with extension *.key with git-crypt
handle all files within directory secretdir/ with git-crypt

Now we add our git user to the secrets. Therefore we need to get the id of our gpg key.

gpg --list-key $(git config --global user.email)

(The id is at pub between / and the date).

Copy it to any text editor.

Now we add the key to the keyring of the local git repository database.
Please replace with the id you copied to the text editor.

git-crypt add-gpg-user <USER_ID>

Now we add a file for encryption.

echo "This file will be encrypted" >> to_encrypt.key

and commit our changes

git add .
git commit -m "add file to encrypt :lock:"

Now lock the file and check it

git-crypt lock

just check the file

cat to_encrypt.key

it is encrypted. For unlocking type

git-crypt unlock

Check the file again

cat to_encrypt.key

The nice thing is that it is not possible to push the unencrypted file to git repo.
Lets test it.

git add .
git commit -m "add encrypted file"
git push

Check the file in your favourite git provider. You can see that it is encrypted.

Add users to git-crypt database

To add a user to git-crypt you need the public gpg file.
Just tell the other users to use the following command

gpg --armor --output public-key.gpg --export <key_mail_address>

Import the key file into your gpg keyring and add trust level ultimate

gpg --import public-key.gpg
# get the id of the imported key
gpg --list-key <key_mail_address>
gpg --edit-key <key_id>
trust
# We need ultimate trust, so choose 5
save

now you are able to add the user as before with

# the user_id is the id of the user in your public key_ring
git-crypt add-gpg-user <USER_ID>

Now the other user is able to decrypt the file with git-crypt in the git repository 🚀

Start a new typescript project with gts and esbuild

Andreas Augustin — Fri, 19 Aug 2022 21:18:00 +0000

Abstract

When starting a new typescript project you need to configure lots of things

npm project settings
typescript transpiler settings
folder structure
style checker and formatter
bundler

To minize the effort gts is a nice tool to help you setting up the project.
Our of deployment/devliver purposes we also want to bundle the code to create nice and small artifacts.
For those purposes we use esbuild.
The whole example can be found here

Prerequisites

For starting this tutorial you need to have nodejs and npm (also npx) installed on your machine.
Some basic knowledge about nodejs, npm and npx is helpful.

Create the project

Please keep in mind that some of the outputs can vary on your system related to the used versions.

So lets start some actions.
First we want to create the npm project.
Therefore please cd into a nice directory of your choice.

Now lets create the project directory and the npm typescript project with the help of gts.

Remark We add the -y flag to accept the default settings. If you want to adjust those, just do not add the flag.

mkdir ts_gts_esbuild
cd ts_gts_esbuild
npx gts init -y

Analyse

Now lets see what we got.

$ tree -a -L 1
.
├── .eslintignore
├── .eslintrc.json
├── node_modules
├── package.json
├── package-lock.json
├── .prettierrc.js
├── src
└── tsconfig.json

2 directories, 6 files

Because we do not want to add build artifacts and dependent libs to our git database we add them to .gitignore

Remark The build/ folder is used per default for the typescript build artefacts.

echo node_modules/ >> .gitignore
echo build/ >> .gitignore

We can see that we received some default settings

npm project settings
typescript transpiler settings
folder structure
style checker and formatter

NPM project settings

$ cat package.json
{
  "name": "",
  "version": "0.0.0",
  "description": "",
  "main": "build/src/index.js",
  "types": "build/src/index.d.ts",
  "files": [
    "build/src"
  ],
  "license": "Apache-2.0",
  "keywords": [],
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "lint": "gts lint",
    "clean": "gts clean",
    "compile": "tsc",
    "fix": "gts fix",
    "prepare": "npm run compile",
    "pretest": "npm run compile",
    "posttest": "npm run lint"
  },
  "devDependencies": {
    "gts": "^3.1.0",
    "typescript": "^4.0.3",
    "@types/node": "^14.11.2"
  }
}

As you can see some devDependencies have been installed. Also we got some preparations for delivering a npm package.
Furthermore we got some npm scripts.

Typescript transpiler settings

npm run compile

This will compile the typescript sources defined in tsconfig.json file and put the built aretefacts into the outDir folder defined in tsconfig.json.

npm run clean

This will cleanup the typescript build artefacts (delete the outDir directory defined in tsconfig.json).

Style checker and formatter

npm run lint

This will run the style checker eslint.

npm run fix

This will fix some of the found lint issues.

bundler

Because we also want to bundle our project we first install esbuild as dev dependency

npm i -D esbuild

Lets test it.

npx esbuild src/index.ts

The output is the bundled file.

You can pass lots of options to the cli interface.
However, using the command-line interface can become unwieldy if you need to pass many options to esbuild. For more sophisticated uses you will likely want to write a build script in JavaScript using esbuild's JavaScript API.

We will do that now.
Lets create a small nodejs binary.

mkdir bin
touch bin/esbuild.mjs

now please copy the following content to the file

#!/usr/bin/env node

import esbuild from 'esbuild';

const build = esbuild.build({
  entryPoints: ["src/index.ts"],
  outdir: "dist",
  bundle: true,
  loader: {".ts": "ts"}
});

await build;

as you can see we will have the outdir dist set. We also need to add this to our .gitignore file.

echo dist/ >> .gitignore

Now lets test it

node bin/esbuild.mjs

As you can see the bundled package is built.

Now we can add a new npm script. Please add a new script within the script part in package.json. Because in addition we want to create the types we also add npx tsc --emitDeclarationOnly --outDir dist/types

...
"scripts": {
  ...
  "posttest": "npm run lint",
  "bundle": "npx tsc --emitDeclarationOnly --outDir dist/types && node bin/esbuild.mjs"
},
...

[option] Package

Now we have the option to create a nice small npm package.
Therefore we need to add a .npmignore file and add some content.

touch .npmignore
echo node_modules/ >> .npmignore
echo build/ >> .npmignore
echo bin/ >> .npmignore
echo src/ >> .npmignore

Also we need to adjust some paths within our package.json

"name": "ts-gts-esbuild",
"version": "0.1.0",
"description": "",
"main": "dist/index.js",
"types": "dist/src/index.d.ts",
...

We can test it now

npm package

This will create a tar archive in your package folder.
You can check it.

Cons

There are lots of pros related to this project setup. But of course there are also some cons.
One of them is that gts is out of the box not properly working with npm workspaces yet gts issue.
But you just need to do the gts init within the repos and adjust some path parameters in some files.

Forem: Andreas Augustin

Git - GitHub actions-template-sync

GitHub actions-template-sync

abstract

Remarks

Usage

Authentication and authorization

SSH

SSH keygen

Private Key

Public key

Prepare the action

GIT - Prevent accidentally pushing credentials

abstract

Installation

Remarks

Fun part

GIT - how and why to sign commits

Git - how and why sign commits

abstract

Why to sign commits

How to sign commits

Further readings

GIT - multi user

Abstract

Theory

example

setup

test

GIT - share secrets

Abstract

Installation

Remarks

Setup

GPG

git-crypt

Add users to git-crypt database

Start a new typescript project with gts and esbuild

Abstract

Prerequisites

Create the project

Analyse

NPM project settings

Typescript transpiler settings

Style checker and formatter

bundler

[option] Package

Cons