Forem: s-heins

A deeper look: Git behind the scenes

s-heins — Fri, 10 Dec 2021 15:18:36 +0000

In the first article, we have looked at how to initialize a git repo locally, add and commit, and set some configuration options.
Next, the second article covered how to work with remotes and branches, how to resolve conflicts, and how to merge and delete branches both from the CLI or from the GitHub UI.

Now that we have gained some overview over the basic functions of git, it is time for an excursion into some basics on how git works behind the scenes. How does git notice that a file has changed so that it can show us this in our status? How does git realize branches? How does git know the order of the commits in each branch? What is HEAD and what is this mysterious detached HEAD state? And most importantly, how do I get out of this state again?

As a resource for this article, I have used the Pro Git book, written by Scott Chacon and Ben Straub which is available for free and may be shared non-commercially.

Because the details of how git works exactly can easily make up a book (as seen above), we will only look deeply enough in this article to set the stage for more advanced git maneuvers such as rebasing, where you replay your work on top of a different commit than the one you originally set out from, and to give some context to the commands we have covered in the first two articles.

How does git know a file has changed?
File status lifecycle and git project sections
- Git project sections
- Git file status lifecycle
- One file can have multiple changes in different states at once
Anatomy of a commit
Back to the start – creating a repository
References
- Branches
- Tags
- HEAD
- Detached HEAD state
Conclusion and command summary

Cover image by Jeremy Francis (with permission)

How does git know a file has changed?

Git does not store the exact differences between files (for example, add the line "house cat" to your file "list-of-animals-to-write-about"), but rather, it stores snapshots. For this example, git has a snapshot of the file before adding that line and then compares it to the current version of the file to figure out if something has changed.

To do this, git computes a SHA-1 hash value based on the contents for each file and uses it for checksumming. SHA-1 is a cryptographic hash function that makes it highly unlikely to create the same hash for two different files, so if the hash is the same, we can assume that it was created based on the same input. If the checksum of a file has changed, the file itself must have changed. If a file was deleted and another was added but they share the same checksum, they must have the same contents – this is how git knows that a file was renamed. If the contents of a file have changed, the name will be the same but its hash will be different.

In case a file has not changed at all in a commit, git will not store the file itself again, just a link to the one it has already saved in a previous snapshot.

File status lifecycle and git project sections

In the first article, we have already added and committed files and file changes. Git has three states for any file it already knows about: modified, staged, and committed. It then also has a fourth state for any files it doesn't know about: untracked.

Let's now look at what different states a file is in, from its creation, adding it, and finally committing it. We will use git status to check which status our file is in.

Git project sections

As a file goes through its lifecycle, it will also travel through different sections of our git project:

Working directory: After adding new-file, the file is present in our directory and untracked, that means that we created a new file git doesn't know about yet. Any existing files we change will be in the modified state.
Staging area: After git add ., the changes are now marked as staged and are in the staging area. This is also called the index.
Repository: After we git commit our changes, they will be added to our local .git repository as committed. We can also push them to add them to our remote repository.

Git file status lifecycle

We can now look at the file states with regards to sections in the project:

Untracked (Working Directory) Files that are not in the staging area and not in your last snapshot (= commit), i.e. files that git doesn't yet know about.
Modified (Working Directory) Files that have been modified since the last commit.
Staged (Staging Area) Untracked or modified files that were added to the Staging area by git adding them.
Committed (Repository) Files that were committed to the repository by running git commit after they were moved to the staging area.

Any committed files will move back to the modified state after we have changed them, and from there, they can move through the lifecycle again. Any unmodified files will still be in the committed state, so safely stored in your git database (so your local .git folder and, if you have set up a remote and pushed your commit to it, also in your upstream repository).

If you have any modified files for which you want to skip the staging area, you can run git commit -a so that git will commit all tracked files without you having to add them first. This does not work for any untracked files, however.

One file can have multiple changes in different states at once

Git uses the file status to communicate to us what the status of our files is. However, a file can also have multiple states – we could do some preliminary changes, add them to the staging area, and then do some more changes. In that case, git will have multiple snapshots of that file, one snapshot per commit where that file changed, another snapshot for the staged version, and yet another one for the working directory version of our file.
Git will then show us that the file has some changes that are to be committed and some other changes that are not yet staged:

Anatomy of a commit

As discussed before, git stores snapshots. Specifically, git stores files as blobs (binary large objects) within git, as well as the checksums of their file contents and header. Then, it builds a tree object (the root project tree) that models the directory structure.

A commit then stores some meta information and a pointer to that root tree.

Commit metadata includes:

author of the commit (i.e. the person that wrote or changed the respective lines)
the committer (the person who committed the work of the author)
the parent of the commit (if it has one) – this is a pointer to a commit hash

In the example further below, we will see when author and committer can differ: For the merge commit we're going to be looking at, I am the author but GitHub committed for me since I executed the merge on the GitHub UI. Another example would be if someone rebases my commits on top of another parent. That means that they will take my work and replay it on top of another parent than the one I used originally. We will focus on such advanced git maneuvers in a subsequent article.

Since every commit contains a reference to its parent (if it has one), we can think of commits as a tree where each commit has between 0 and n parents:

If a commit has zero parents, it is the very first commit.
If it has two (or more), it is a merge commit. Usually, two branches would be merged so that the resulting commit would have two parents but it is also possible (albeit unusual) to merge more than two branches.
If a commit has one parent, it is a "normal" commit, i.e. neither a merge commit nor the very first commit in a repository.

Let's say our collaborators Anna and Wolfgang have been busy and our git tree currently looks like this:

Hint if you have not read the previous articles:
To be able to use the lg command alias, add this line to your ~/.gitconfig file under the [alias] section:
[alias]
lg = !clear && git log --all --graph --pretty='format:%C(auto)%h%d %s  %C(magenta)[%an] (%ad)%C(reset)' --date=format:'%d.%m.%y %H:%M'

In this example, we have merged the branch "articles-on-animals-from-list" into "main" and the commit 2cca46a (that is also labeled as HEAD on the screenshot) is a merge commit. At the very bottom of the screenshot, 98369a7 ("Add a house cat") is the first commit and has no parent, and all other commits in between have only one parent. The commits 0a6ebca and b9eb6d0 have the same parent, but they only have one parent: Our merge commit that we have currently checked out and is the last commit on the main branch.

If you also want to look at the same commit data as me in this article, you can clone my git-encyclopedia-example project on GitHub by running git clone https://github.com/s-heins/git-encyclopedia-example.git (to clone via HTTPS).

To look at commit metadata, we can use git cat-file -p, where -p lets us pretty-print the object's content.
I'm saying "object" because git stores multiple pieces of information as objects – such as blob objects for files, commit objects, and tree objects which allow us to reference other tree objects and blob objects to model a file tree. (Any children tree objects would then represent folders and blob objects would be files). To read more about this, see chapter 10.2 in the pro git book. The fourth type of git objects are annotated tags which we will discuss soon when we talk about references in git. For short, we will refer to annotated tags as just tags in the following text.

Types of git objects:

blob
commit
tree
tag

We can use git cat-file -p to look at our merge commit, 2cca46a for example, where we will see that it has two parents and what their commit hashes are. As mentioned before, we can also see that in this case, author and committer are different because I created the merge commit via the GitHub UI.

When git gave us some information about our commit before, it not only included information on the parent commit hashes, the author, and committer, but it also listed an object hash for the root tree that models the directory structure.

From this tree, we can look at the different sub-trees (subdirectories) and files with the command git ls-tree -rt, where the -r flag will recurse into subdirectories and the -t flag will include trees when recursing.

$ git ls-tree -rt 240e206

100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391    alligator
100644 blob 4b92c463e35f265b9b43f5beebc9e02df815efb2    animals-to-write-about
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391    chimpanzee
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391    crocodile
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391    elephant
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391    giraffe
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391    gorilla
100644 blob 3be11c69355948412925fa5e073d76d58ff3afd2    house-cat.md
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391    tiger.md

In our example, we do not have any directories yet, so git only lists blob objects here.
So let's move all of Wolfgang's articles into a travel directory and look at the tree again.

$ git commit -m 'Move cities articles into travel subdirectory'

[ENC-002_capital-cities-in-europe cafdadd] Move cities articles into travel subdirectory
 4 files changed, 0 insertions(+), 0 deletions(-)
 rename albania-tirana.md => travel/albania-tirana.md (100%)
 rename austria-vienna.md => travel/austria-vienna.md (100%)
 rename belarus-minsk.md => travel/belarus-minsk.md (100%)
 rename germany-berlin.md => travel/germany-berlin.md (100%)

With the commit hash that git tells us in the output, cafdadd, we can run git cat-file -p cafdadd and find out the hash of the root tree, which we can use for the ls-tree command afterwards. To keep the list a bit shorter, we can tell git to abbreviate the object hashes by using the --abbrev=<digits> flag.

$ git ls-tree -rt --abbrev=7 8ec6421

100644 blob e69de29 alligator
100644 blob 4b92c46 animals-to-write-about
100644 blob e69de29 chimpanzee
100644 blob e69de29 crocodile
100644 blob e69de29 elephant
100644 blob e69de29 giraffe
100644 blob e69de29 gorilla
100644 blob 3be11c6 house-cat.md
100644 blob e69de29 tiger.md
040000 tree ea0789f travel
100644 blob e69de29 travel/albania-tirana.md
100644 blob e69de29 travel/austria-vienna.md
100644 blob e69de29 travel/belarus-minsk.md
100644 blob 1038ae6 travel/germany-berlin.md

Since our encyclopedia authors were a bit lazy here and have not actually added contents to these files, a lot of files share the same object hash (e69de29) – they are all empty.

If we do not use the flag to recurse into subtrees, git will show us that the root tree object only contains information on the animal article blob objects and the travel tree object, and not on the travel article blobs. The information about them will be contained in the travel tree, ea0789f, so the hierarchy of these objects looks like this:

.
├── commit cafdadd 
│   └── tree 8ec6421
│       ├── blob e69de29 alligator
│       ├── blob 4b92c46 animals-to-write-about
│       ├── blob e69de29 chimpanzee
│       ├── # … (other articles on animals)
│       └── tree ea0789f travel
│            ├── blob e69de29 albania-tirana.md
│            ├── blob e69de29 austria-vienna.md 
│            ├── blob e69de29 belarus-minsk.md 
│            └── blob 1038ae6 germany-berlin.md

To find out a bit more on what happens to git objects as we add and commit them, see this article.

Back to the start – creating a repository

As mentioned in the first article, git adds a .git folder in your working directory when you run the git init command.

We can now find all our commits in the objects folder within the .git folder. Git saves these objects in folders that carry the first two characters of the object hash. The rest of the characters are used as the file name within that folder. For example, our commit 2cca46a which we looked at with cat-file is really called 2cca46ab8626e867e2994cac12eb97887b0a82a2 in full but it is enough to use the short name because there are no other git objects whose name also starts with 2cca46a. The information for this commit will be contained in a file within the 2c folder inside the objects folder, and this file will be called ca46ab8626e867e2994cac12eb97887b0a82a2.

References

Like in programming, git references point to other things. This could be a commit if we're talking about a branch reference or a tag, or it could be another reference.

Branches

In case you want to work together with someone, you could always send them the name of your latest commit and they could then check out that commit. That would be a lot of hassle, however, since you would have to keep messaging them any time you make a change to tell them the newest commit hash, and they would need to check their messages constantly. At this rate, no one would want to work with us on any projects and the world would be a sad place.

To make things a lot easier, we can refer to a line of work with branches. Behind the scenes, a branch is just a reference to a commit with the added bonus that if you add a child to this commit, the reference gets advanced to this latest commit. The commit the reference points to then carries information about its parent commits but the branch itself is just a pointer to the latest commit, the so-called head. That means that git just infers which commits belong to a branch based on the commit the branch points to and its parent(s) and their parent(s). Git can show us where our branch diverged from main by going backward from the head to its parents and showing us those commits which are not contained in the main branch, i.e. which are not the commit that the main branch points to nor parents of it.

Git saves all information about references in the refs folder. Any references for our remotes are contained within .git/refs/remotes/origin. As we can also see in our git tree visualization in the command line, main points to 75616f3, the ENC-003… reference points to f82574d, and the ENC-002… reference points to 94a47c8.

If we look at just the commits in our branch and compare them to the graph view, we can see that our ENC-002 branch and main share a few commits (orange in the screenshot) and they also have some commits that only belong only to them (green in the screenshot). In the tree view above it might seem like ENC-002 only began with commit b6eb6d0 (Add article on Tirana in Albania) but really, it contains all commits that came before.
When we merge this branch to main then, git will create a merge commit that has the latest branch commit and the latest main commit as parents as we can see in our example in the merge commit 2ca46a (Merge pull request #1…).

HEAD

Before, we mentioned that the latest branch in a commit is also called a head. Git stores information on the currently checked out commit or reference in the HEAD file within the .git folder. In case we currently have a branch checked out, head will refer to this branch, so HEAD will be a reference to our branch reference.
We can also use HEAD within git to refer to our branch. So instead of using git push origin ENC-002_capital-cities-in-europe, we can also run this following command if we have currently checked out our branch ENC-002… and we want to push it to our upstream repo:

git push -u origin HEAD

The -u flag is a shorthand for --set-upstream which tells git to add our local branch to the upstream repository. Afterwards, we specify our remote repository that git should push to. In most cases this will be called origin; in our repo it is also called origin because we told git to call our upstream "origin" when we ran git remote add origin <repository-address>.
Finally, HEAD is our pointer that points to our branch reference which in turn points to our latest commit (if we have currently checked out a branch).
Git will now create a branch on origin which is called the same as our local branch and points to the same commit.

In the background, git stores information on HEAD in the .git/HEAD file. When we run cat .git/HEAD, git tells us that our HEAD is a reference which points to another reference, namely ENC-002….

Detached HEAD state

Most of the time, HEAD will point to a branch reference which points to the latest commit in a branch but there is also such a thing as detached HEAD state where HEAD will point to a commit which is not the head of a branch. You move into detached head state by checking out a commit that no branch reference points to and you can move out of it again by either creating a new branch (whose reference will then point to this commit), or by checking out an existing branch.

In the following example, I checked out a previous commit that was not the head of any branch, added and committed a file and created a new branch to move out of detached head state again.

Conclusion and command summary

In this article, we looked at the different file states (unmodified/committed, untracked, modified, and staged). We saw that each commit contains information on its author, committer and parent commit(s) and how git branches work. That a branch is just a reference to a commit, and that the last commit of a branch is called its head. We also got to know a special reference, HEAD, which points to a branch reference (which points to a commit) if we are currently on a branch, or a commit if we are in detached HEAD state.

Here are some commands to remember:

git commit -am "my commit message" to commit all modified files without first staging them (does not work for untracked files)
git tag -a "my tag message" to create an annotated tag, for example if you're working with releases
git push -u origin HEAD as an alternate way of doing git push -u origin <my-branch-name>

Git blob and tree objects during the status lifecycle

s-heins — Sat, 06 Nov 2021 21:30:55 +0000

In order to find out what git does when we change files, we're going to create this directory and file hierarchy:

.
├── top-level-dir
│   ├── second-level-file.md
│   └── sub-level-dir
│       └── third-level-file.md
└── top-level-file.md

To do so, we can run these commands in an empty git repository:

echo "Lorem ipsum top level file" > top-level-file.md
mkdir top-level-dir
cd top-level-dir
echo "Lorem ipsum second level file" > second-level-file.md
mkdir sub-level-dir
cd sub-level-dir
echo "Lorem ipsum third level file" > third-level-file.md

(Cover image by wynand van niekerk from FreeImages)

Result after `add`

After just running git add ., not commit, the contents of the .git/objects folder now look like this:

$ tree .git/objects

.
├── 1e
│   └── 86cd88d41dabd1342deb658da84a2bc9ab83cd
├── 51
│   └── 4ef3078e9106262d76be15bf23d83e8cd3bbe8
├── de
│   └── 8fcf5858a255ce7a9a60db7a004fa5b6ad80e5
├── info
└── pack

For every file, git has created a SHA-1 hash that is 40 characters long. The first two characters make up the name of the directory in the .git/objects folder and the remaining 38 characters are used for the file name. To refer to an object, we don't always have to use the full 40-character hash; the first seven or so digits are usually sufficient for git to know which object we want since there is usually no other object sharing those first digits. In our example, our object hashes are 1e86cd8…, 514ef30…, and de8fcf5….

By examining the content with git cat-file -p <object hash>, we can find out that 1e86cd8 is the third-level file, 514ef30 is the top-level file, and de8fcf5 is the second-level file.

Insights

git already adds objects in the objects folder even if they are not committed
apparently, it does not add tree objects yet, just the blobs

Git will only add tree objects after we commit our changes.

Result after `commit`

After running git commit, git has created a commit with the shortened hash 343d8dd.
The git objects folder now looks like this (comments after # sign):

$ tree .git/objects

.
├── 1e
│   └── 86cd88d41dabd1342deb658da84a2bc9ab83cd   # third-level-file.md
├── 34
│   └── 3d8dd527a9740dc15f8be4f5a8c71308b96926   # new, commit object
├── 3a
│   └── 2e7f0d5abbe9b0d1bb26e6118f1c0070489e17   # new
├── 4d
│   └── 9f43d4557e71eb6e5f540493ae5b92dfef7a67   # new
├── 51
│   └── 4ef3078e9106262d76be15bf23d83e8cd3bbe8   # top-level-file.md
├── a6
│   └── 9f01451a19e5ba5fce7c8bd1c584b9e2e711ed   # new
├── de
│   └── 8fcf5858a255ce7a9a60db7a004fa5b6ad80e5   # second-level-file.md
├── info
└── pack

Looking at information on the commit object:

$ git cat-file -p 343d8dd

tree 4d9f43d4557e71eb6e5f540493ae5b92dfef7a67
author Sonja Heins <sonja.heins@example.com> 1636191282 +0100
committer Sonja Heins <sonja.heins@example.com> 1636191282 +0100

Add nested dirs and files

We can look at all files and directories contained in the root tree 4d9f43d with the command git ls-tree using the -r flag for recursing into subtrees and the -t flag for showing trees when recursing. I used --abbrev=7 here so git will abbreviate object hashes to seven digits.

$ git ls-tree -rt 4d9f43d --abbrev=7

040000 tree a69f014 top-level-dir
100644 blob de8fcf5 top-level-dir/second-level-file.md
040000 tree 3a2e7f0 top-level-dir/sub-level-dir
100644 blob 1e86cd8 top-level-dir/sub-level-dir/third-level-file.md
100644 blob 514ef30 top-level-file.md

If we only want the files listed and not the tree objects, we can use git ls-tree -r:

$ git ls-tree -r 4d9f43 --abbrev=7

100644 blob de8fcf5 top-level-dir/second-level-file.md
100644 blob 1e86cd8 top-level-dir/sub-level-dir/third-level-file.md
100644 blob 514ef30 top-level-file.md

So, after the first commit, our hierarchy is like this:

.
├── commit 343d8dd 
│   └── tree 4d9f43d
│       ├── blob 514ef30            (top-level-file.md)
│       └── tree a69f014            (top-level-dir)
│            ├── blob de8fcf5       (second-level-file.md)
│            └── tree 3a2e7f0       (sub-level-dir)
│                └── blob 1e86cd8   (third-level-file.md)

Changing a file

$ echo "Adding a line to the second-level file" >> top-level-dir/second-level-file.md

Now we have added a line to the second-level file. Before we run git add, nothing changes in our object directory:

$ tree .git/objects

.
├── 1e
│   └── 86cd88d41dabd1342deb658da84a2bc9ab83cd
├── 34
│   └── 3d8dd527a9740dc15f8be4f5a8c71308b96926
├── 3a
│   └── 2e7f0d5abbe9b0d1bb26e6118f1c0070489e17
├── 4d
│   └── 9f43d4557e71eb6e5f540493ae5b92dfef7a67
├── 51
│   └── 4ef3078e9106262d76be15bf23d83e8cd3bbe8
├── a6
│   └── 9f01451a19e5ba5fce7c8bd1c584b9e2e711ed
├── de
│   └── 8fcf5858a255ce7a9a60db7a004fa5b6ad80e5
├── info
└── pack

9 directories, 7 files

After we run git add ., we have a new directory and file, f9cc8e0:

$ tree .git/objects

.
├── 1e
│   └── 86cd88d41dabd1342deb658da84a2bc9ab83cd
├── 34
│   └── 3d8dd527a9740dc15f8be4f5a8c71308b96926
├── 3a
│   └── 2e7f0d5abbe9b0d1bb26e6118f1c0070489e17
├── 4d
│   └── 9f43d4557e71eb6e5f540493ae5b92dfef7a67
├── 51
│   └── 4ef3078e9106262d76be15bf23d83e8cd3bbe8
├── a6
│   └── 9f01451a19e5ba5fce7c8bd1c584b9e2e711ed
├── de
│   └── 8fcf5858a255ce7a9a60db7a004fa5b6ad80e5
├── f9
│   └── cc8e0b7374e973acbbc00c8742997ccd17d473   # new
├── info
└── pack

10 directories, 8 files

This object contains the new file changes:

$ git cat-file -p f9cc8e0

Lorem ipsum second level file
Adding a line to the second-level file

Now we commit and, in doing so, produce a new commit, 9d2aeae:

$ git commit -m "Add line to second-level-file.md"

[main 9d2aeae] Add line to second-level-file.md
 1 file changed, 1 insertion(+)

Afterwards, our git object directory looks like this (comments after # sign):

.
├── 1e
│   └── 86cd88d41dabd1342deb658da84a2bc9ab83cd
├── 34
│   └── 3d8dd527a9740dc15f8be4f5a8c71308b96926
├── 3a
│   └── 2e7f0d5abbe9b0d1bb26e6118f1c0070489e17
├── 4d
│   └── 9f43d4557e71eb6e5f540493ae5b92dfef7a67
├── 51
│   └── 4ef3078e9106262d76be15bf23d83e8cd3bbe8
├── 9b
│   └── 676fbb2c624d1541263163ff84447242b5997b   # new
├── 9d
│   └── 2aeae6478aa11d828bc4d969dbfc55186593bf   # new (commit)
├── a0
│   └── 4ef2163c702bea83aa59d0fc5cc1547006b22f   # new
├── a6
│   └── 9f01451a19e5ba5fce7c8bd1c584b9e2e711ed
├── de
│   └── 8fcf5858a255ce7a9a60db7a004fa5b6ad80e5
├── f9
│   └── cc8e0b7374e973acbbc00c8742997ccd17d473   # changed second-level-file.md
├── info
└── pack

13 directories, 11 files

In order to change our second-level-file, git needed to change the tree object for the top-level-dir folder as well as the root tree. After we committed our changes, git told us the new commit hash, 9d2aeae.
We can now use this and run git cat-file -p 9d2aeae to find out the hash for the root tree, 9b676fb.
Let's run ls-tree on this to see what it looks like.

$ git ls-tree -rt 9b676fb --abbrev=7

040000 tree a04ef21 top-level-dir
100644 blob f9cc8e0 top-level-dir/second-level-file.md
040000 tree 3a2e7f0 top-level-dir/sub-level-dir
100644 blob 1e86cd8 top-level-dir/sub-level-dir/third-level-file.md
100644 blob 514ef30 top-level-file.md

Now, our hierarchy is like this:

.
├── commit 343d8dd 
│   └── tree 4d9f43d
│       ├── blob 514ef30            (top-level-file.md)
│       └── tree a69f014            (top-level-dir)
│            ├── blob de8fcf5       (second-level-file.md)
│            └── tree 3a2e7f0       (sub-level-dir)
│                └── blob 1e86cd8   (third-level-file.md)
├── commit 9d2aeae 
│   └── tree 9b676fb
│       ├── blob 514ef30            (top-level-file.md)
│       └── tree a04ef21            (top-level-dir)
│            ├── blob f9cc8e0       (second-level-file.md)
│            └── tree 3a2e7f0       (sub-level-dir)
│                └── blob 1e86cd8   (third-level-file.md)

The object hashes for top-level-file (514ef30), sub-level-dir (3a2e7f0), and third-level-file (1e86cd8) have not changed, only the ones that lead us to second-level-file.md have changed, so the hash for the file itself and the ones for the trees that contain it.

Branching and working with remotes in git

s-heins — Fri, 22 Oct 2021 05:50:23 +0000

In this article, we will have a look at how to work with remotes in order to work together with collaborators, how to work with branches, how to resolve conflicts between two sets of changes (aka "Why does git reject my push?"), and finally, how to merge and delete branches using the CLI or the GitHub UI.

(Cover image by fabrizio turco from FreeImages)

Recap

In the Introduction article, we looked at how to set up a local git repository, how to configure git and create aliases, and how to add and commit files on the example of creating an encyclopedia with articles and a list of topics to write about. We have also added some aliases to our gitconfig file at ~/.gitconfig, such as this one that lets us print a neat-looking tree in the command line to show us the git history. I'm adding it here again since we will keep using it throughout this article.

[alias]
lg = !clear && git log --all --graph --pretty='format:%C(auto)%h%d %s  %C(magenta)[%an] (%ad)%C(reset)' --date=format:'%d.%m.%y %H:%M'

Now we will look at how to save our work remotely so that we won't only have access to it locally on our machine but also on other machines and can work on changes together with collaborators. Our encyclopedia is going places and at any rate, we wouldn't want to do it all by ourselves. It's time to give this thing called balance a go and let other people join in on our project.

Working with remotes

All we have for now is our changes on our own machine. Anything that we commit will not be lost as long as our machine is working, but we can't share it with anyone else easily nor will it be safe if our device decides to stop working (as they tend to do in the most inopportune moments).

Creating a remote repository

Firstly, we need to settle on a place to host our repository. This could be GitHub or GitLab, for example.
After creating an account, we can create our repository.
On how to set up a repository, please see these articles in their respective documentation pages: GitHub, GitLab

Pushing our existing changes to the remote

After creating a new repository on GitHub, it already shows us some instructions on how to push an existing git repository:

The first line is adding a so-called origin to our local repository. This is referencing our new repo on github. In this example you can see that I have chosen to use SSH but this can be switched with a button further on top if you want to use HTTP. Using SSH will require you to set up a private and public SSH key pair whereas you will only have to authenticate with your username and password or access key if you use HTTP, so it will be easier if you have never done this before.
Since I had previously connected to GitHub on my machine, I was not prompted for my login credentials here. If you are, enter the credentials you are prompted for and then you are able to push to your remote repository, to your origin.

After pushing, we can now see our changes on the site we have chosen as a host for our repo after refreshing in the browser. It will also show the last commit message, "Add a house cat" in this example, along with our files.

Cloning a remote

For now, we have worked alone on our encyclopedia, or rather, the list of animals we want to write about.
Our git history will look something like this; it's a single line since we have only worked on one branch.

Now our editor tells us to get cracking since so far, we have only made one measly list of animals to write about, it's not quite an encyclopedia. We have already pushed our repo, so we're ready to have collaborators join us in this project!
We have managed to recruit Anna Animallover and Wolfgang Wanderlust to join us. Anna is going to help us complete our list of animals and then write articles on them and Wolfgang will write about cities of interest.

You can find the address for your repository on the site you have chosen to host it – you will need this so that Anna and Wolfgang can clone your repository.\
On GitHub, it looks like below. As before, you can choose either HTTP or SSH which will change the link. In this example, it is https://github.com/s-heins/git-encyclopedia-example.git for HTTP and git@github.com:s-heins/git-encyclopedia-example.git for SSH.

Anna and Wolfgang can then clone our repository by running git clone <repository-address>.

Branching

The first thing Anna wants to do is create a new branch. This way we can keep the main branch for things that have already been reviewed and we can have our in-progress work in the other branches of our project.

Creating a new branch

To create a new branch and switch to it, use git checkout -b my-branch-name. In case you are working with tickets on Jira, for example, prefixing your branch with the ticket name enables your team to get a good overview on what tickets are currently in progress. We could name our branch "ENC-001_add-animal-articles" if our ticket number is "ENC-001", for example.

Pushing a new branch to origin

After Anna has added some articles (git add .) and committed them to her local repository (git commit -m "My message"), she wants to push those changes to the remote.
She can do so by running git push -u origin my-branch-name. With this, her local changes are propagated to our origin (also called upstream), our repository on GitHub. Now it is possible for others to see Anna's changes and add their own.

From our machine, we can now git pull and git will tell us that a new branch has been created. We can check it out by running git checkout articles-on-animals-from-list. This will switch our working branch to Anna's branch and since we just pulled, we will also have her latest committed changes.

After we have created articles for the animals that are still missing from the list, we can add those changes to a commit and push them. This time, git already knows the upstream branch (namely, origin/articles-on-animals-from-list), and thus we can just use git push without any arguments.

After Anna pulls again, she will also have our changes.

By using the same branch and adding our commits, we can safely collaborate. Anna and me can both commit and push our changes as we go along. We just shouldn't both alter the same file at the same time or we will get conflicts that we will need to resolve.

Resolving conflicts

Let's say we have the idea to add more animals to our list of animals to write about but Anna has the same idea. We add a gorilla and a chimpanzee, she adds an alligator and a crocodile. We push first so now the file will have updated without Anna knowing about it.

When Anna tries to push her commit, git will not let her. It rejects her commit and tells her to pull first.

Once Anna pulls, git now tells her she has a conflict. If you use an IDE with version control functionality, you can also resolve any conflicts there and you normally would. Here, we will go the manual way for now because it will give us some insight into what happens in git behind the scenes without part of it being masked by IDE functionality.

As you can see, the contents of the file have a line saying HEAD that contain our changes where we added the gorilla and chimpanzee (the changes that we have already pushed to our branch on the remote), then another line of equals signs as separators, then her changes where she added the alligator and the crocodile, and then a line with the name of her commit, 7aaee11, and her commit message (Add alligator and crocodile to the list).

If we look at the git tree in the console again, we can see that Anna's commit, 7aaee11 and my commit that is pushed to the upstream, ba9e8f3 both have the same parent. We now have to resolve the conflicts so that Anna's changes can be propagated to our repository.

We can now edit the file to remove the three lines git has added to show us the parts of the file where we have conflicts. Since we want to retain all animals, we only need to delete these lines. As you can see on the screenshot, the file only has the list of animals afterwards, and the lines with the <<<<, >>>> and ==== are no longer contained.

After we have resolved the conflicts, we need to add the file so git knows that we can continue reconciling the changes that Anna and I have made.
In this case, I have put a line in my ~/.gitconfig file telling git it should rebase rather than try to merge any conflicts. That means that it will try to make it as if Anna's changes had come after my own. We will go into the difference between merging and rebasing in a later article. For now, it doesn't matter that much which one you pick. If you look at the above screenshot, you will see git tells me to run git rebase --continue once I am done resolving my conflicts. If you chose merge as a pull strategy, it will tell you to run git merge --continue instead. This will only have an impact on how our tree looks afterwards but it will not have any impact on our file changes.

If we now look at the git tree after finishing the rebase, we can see that it looks as if Anna had done her changes after our own.

Merging and deleting branches from the console

Let's say we have added articles on all animals we had on our list for now, have shown them to our editor, and they are happy with them. That means that we can now propagate those changes into our main branch and will no longer need our branch articles-on-animals-from-list.

Firstly, we'll want to git pull on our article branch to make sure we have the latest changes. Then, we check out the main branch and run git merge articles-on-animals-from-list. Afterwards, we push again to make sure those changes are also on our upstream main branch.

Afterwards, we can delete the branch on your machine by running this command:

git branch -D articles-on-animals-from-list

And this command to delete the remote branch:

git push --delete articles-on-animals-from-list

Merging and deleting branches from the GitHub UI

In practice, you will probably never run these commands from the CLI but rather use a version control provider such as GitHub or GitLab. Rather, you will create a request to propagate your changes into the main branch, called pull request (PR) on GitHub, and merge request (MR) on GitLab.

On GitHub, click on the "Pull requests" tab and then on the button saying "Compare & pull request".

On the next dialogue window, click on "Create pull request".
You will see an overview page where you can see how many commits will be added to main and what changes they will introduce.

After it has been reviewed, it can be merged to main by clicking the "Merge pull request" button.

After confirming the merge, you also have the option to delete your old branch via the UI. However, this will only delete the remote branch and you will still have to run git branch -D articles-on-animals-from-list to delete your local branch.

Conclusion and command summary

We have looked at how to push our changes to a remote repository and have created our first branch, kept adding and committing and have resolved our first conflict!

Here are some commands to keep in mind:

git clone <address> to clone a remote repository
git checkout -b my-branch-name to create a new branch and switch to it
git push -u origin my-branch-name to push a branch to a remote
git checkout my-branch-name to switch to an existing branch from the remote
git pull to pull changes from the remote
git rebase --continue or git merge --continue to continue a rebase or merge after adding files
git merge my-branch to merge "my-branch" into the branch you have currently checked out
git branch -D my-branch to delete "my-branch" locally
git push --delete my-branch to delete "my-branch" on the upstream

A practical introduction to git – jumping in with both feet

s-heins — Wed, 13 Oct 2021 20:08:06 +0000

If you're a developer, you have probably heard about the version control system, git. If you haven't used it yourself yet, or not a lot, you might also be a bit scared by all the different words – commits, branches, pushing, cherry-picking, rebasing, … what is all that stuff? And why is version control so hard? What if I do the wrong thing and I end up losing everything?

First up, git doesn't lose anything you have committed. It might be a bit harder to find but don't despair. We'll get you there.
But let's start at the beginning without any looming threat of losing your changes.

This will be the first article in a series and it will tell you how to set up your repository from the command line, how to add changes and save ("commit") them, how to set your name and email address to be added as metadata to those changes, and how to define a git aliases in general and a specific one to get a pretty git tree of your changes in the console.

(Cover image by Nerry Burg from FreeImages).

Why version control?

If you add some files or change their contents, you'll want to have a way to save your progress. The difference between having the file in a version control system such as git rather than just saving it on your hard drive is that you have a history of the file along with additional metadata – you could revert back to an old version if you wanted to, look at exactly which changes were introduced when, by whom, and, if they wrote a meaningful message, why they did those changes.

Your task

Let's say you're tasked with creating an encyclopedia on animals. This is something that will require a lot of work, and re-working articles, so you want to know your articles are safe and sound. You might also want to revert some changes if your editor tells you to get rid of that article on flamingos one day and then changes their mind the next. You decide to start a git repository to commit your changes.

Setup

Initializing your git repository

In your shell, create a new directory and initialize your git repository by running git init:

As you can see, this creates to a hidden folder .git. If we delete this folder again, your directory will no longer be a git repository.

Configure

So that other people know whom to contact regarding the changes you introduced because they might have some questions (or if you want to be able to run into some code and then marvel at the fact that these changes here were done by yourself a year ago and now you can't remember very much about them), you will want to set your name and email in your git config.
You can either do so globally for the current logged-in user by running it with the --global modifier which sets it for all repositories you commit to from this user on your machine, or for just this one repository if you run these commands with the --local flag or leaving out the flag altogether since "local" is the default setting.

git config --global user.name "Evelyn Example"
git config --global user.email evelyn@example.com

Getting started on content

Now that we've got this out of the way, let's get started!
You might have noticed that git has automatically created a branch named "main" for us (or, depending on your settings, this may also be called "master". If you want to change the name of the default branch that gets created on running git init to something other than master, you can run git config --global init.defaultBranch main , or whichever name you prefer instead of main).

Depending on which approach you are using when developing, you may work with additional branches, then creating merge requests that need to be approved so you can move these changes onto the main branch, or alternatively, just work with the main branch.
Let's assume we're the only collaborator on this encyclopedia for now and thus, we can use the main branch to make our changes and will not run into trouble. (If multiple people are working on just one branch, it requires a team that is highly coordinated so those changes keep getting reviewed)

Adding files

At first, we might want to create a list of animals we want to write articles on. We create a file animals-to-write-about and add a line house cat because, hey, everyone seems to be crazy about cats, so our encyclopedia needs to have an article on them.

If we want to know what the current status of our repository is, we can run git status.
Git says our file is an untracked file… but what does that mean?

If git does not yet know about a file, it is untracked. If we want to add it to git, we can run git add <file> or just add all files with git add . (the dot at the end means all files in the current directory and sub-directories). In the next step, we can then git commit these files. Only files that have been committed are safe from being lost, so we want to make sure we always commit any changes that we want to keep.
By adding the -m flag and then a note in quotation marks or single quotes, we can add a commit message. This should always start with an upper case verb in the present tense. For more pointers on how to write good commit messages, have a look at this article by Chris Beams.

Git aliases

What was this git lg command I just used to look at the changes? It was a git alias. This is a handy way to save all your typing energy for that encyclopedia.
You can add your own aliases by editing the ~/.gitconfig file (on Linux / MacOS). My list looks like this:

# This is Git's per-user configuration file.
[user]
    email = s.heins@example.com
    name = S Heins
[credential]
    helper = osxkeychain
[alias]
    s = status
    c = commit -m
    co = checkout
    lg = !clear && git log --all --graph --pretty='format:%C(auto)%h%d %s  %C(magenta)[%an] (%ad)%C(reset)' --date=format:'%d.%m.%y %H:%M'
    p = pull --all -p
[core]
    excludesfile = /Users/heinss/.gitignore_global
[commit]
    template = /Users/heinss/.gitmessage.txt
[pull]
    rebase = true
[init]
    defaultBranch = main

I highly recommend adding the alias under lg for a pretty git tree right in your command line.

Outlook: Working with remotes and branches

In the next article, we will be looking at how to work with remotes in case you want to save your work somewhere other than just your local machine and how to work with branches.

If you work with multiple collaborators, working with branches in conjunction with a version control service such as github or gitlab that lets you create requests to propagate these changes into the main branch enables you to first review the changes before they go into your main branch most easily. Propagating changes from one branch to the other is called merging one branch into the other.
Otherwise, collaborators would commit in the main branch and if something goes seriously wrong, they would have to roll back those changes. Or if a commit needs to be reworked, additional rework commits need to be added, potentially bloating up your tree instead of the collaborator being able to still add them to their commit before merging. There are working modes though that use only one branch; this is called trunk-based development.
However, most teams I have worked in have chosen to work with branching and so-called merge requests, also called pull requests.
More on working with branches in the next post though!

Conclusion and command summary

We created our first local git repository, and added and committed a file to it!

Here are some commands to keep in mind:

git init to initialize a local git repository
git status to show the current status of a repository and the files it contains
git add . to add all files
git commit -m 'My message' to commit those changes with a commit message (i.e. propagate those changes to the branch and thus keep them from being lost)
setting some options in the global git config file at ~/.gitconfig:
- git config --global user.name "Evelyn Example" to set the author name globally (for all git repositories)
- git config --global user.email evelyn@example.com to set the author email address globally
- git config --global init.defaultBranch main to set the default branch name when initializing a new repository to main globally

Forem: s-heins

A deeper look: Git behind the scenes

How does git know a file has changed?

File status lifecycle and git project sections

Git project sections

Git file status lifecycle

One file can have multiple changes in different states at once

Anatomy of a commit

Back to the start – creating a repository

References

Branches

Tags

HEAD

Detached HEAD state

Conclusion and command summary

Git blob and tree objects during the status lifecycle

Result after add

Insights

Result after commit

Changing a file

Branching and working with remotes in git

Recap

Working with remotes

Creating a remote repository

Pushing our existing changes to the remote

Cloning a remote

Branching

Creating a new branch

Pushing a new branch to origin

Resolving conflicts

Merging and deleting branches from the console

Merging and deleting branches from the GitHub UI

Conclusion and command summary

A practical introduction to git – jumping in with both feet

Why version control?

Your task

Setup

Initializing your git repository

Configure

Getting started on content

Adding files

Git aliases

Outlook: Working with remotes and branches

Conclusion and command summary

Result after `add`

Result after `commit`