Forem: kataoka_nopeNoshishi

Testing Multipart Upload Requests in Axum: Three Approaches

kataoka_nopeNoshishi — Mon, 14 Apr 2025 11:38:47 +0000

Multipart uploads are often essential when building APIs for file uploads or complex form submissions. Testing them thoroughly is just as important as implementing them. This article provides an overview of three distinct testing approaches for multipart requests in Axum, taken from my demo repository. I will walk through the structure, sample code, and the benefits and drawbacks of each approach.

Introduction: How Axum Handles Tests

Axum itself demonstrates various testing patterns in its official repository. In that example, we see how to:

Use method like tower::ServiceExt to send requests without starting a real server.
Use request client like reqwest to send requests with HTTP server.

These official examples confirm that Axum offers a flexible testing environment. Inspired by those patterns, this article explores three ways to test file uploads (multipart requests) specifically. My goal is to provide you with multiple strategies, each with varying levels of realism and complexity.

Testing Approaches

It includes:

Oneshot approach (lightweight, no full server).
Axum Test (using specialized test crates).
HTTP Server with Reqwest (fully simulates real requests).

Each approach has its own folder (and code sample) in the examples/ directory.

1. Oneshot Testing

The Oneshot approach uses tower::ServiceExt::oneshot. It allows you to:

Test your Axum service without HTTP server.

Below is a simplified version of the oneshot test (examples/oneshot.rs):

#[cfg(test)]
mod tests {
    use super::*;
    use axum::http::Request;
    use axum::body::Body as AxumBody;
    use tower::ServiceExt; // for oneshot
    use common_multipart_rfc7578::client::multipart::{Form as MultipartForm, Body as MultipartBody};
    use http_body_util::BodyExt;

    #[tokio::test]
    async fn test_with_oneshot() -> anyhow::Result<()> {
        // Prepare Axum app
        let app = create_app();

        // Create a multipart form
        let mut form = MultipartForm::default();
        form.add_text("file", "hoge-able");
        form.add_file("csv", "./dummy/test_upload.csv")?;

        // Create request
        let content_type = form.content_type();
        let body = MultipartBody::from(form);
        let req = Request::builder()
            .method("POST")
            .uri("/upload")
            .header("Content-Type", content_type)
            .body(AxumBody::from_stream(body))?;

        // Send request using oneshot
        let response = app.oneshot(req).await?;
        assert_eq!(response.status(), 200);

        // Check response body
        let response_body = response.into_body().collect().await?.to_bytes().to_vec();
        assert_eq!(
            String::from_utf8(response_body)?,
            "Files uploaded successfully"
        );

        Ok(())
    }
}

In fact, there is no definition that successfully converts multipart structures of crates such as reqwest into http::Body.
Therefore, it is very easy to convert multipart structures to httt::Body if there is a structure that defines multipart structures and can convert them directly to http::Body (implementing into_stream()).

Benefits

Fast: No need to bind to a real port or run a live server.
Isolated: Tests focus on your application logic rather than network overhead or environment variables.
Simple: Minimal setup, straightforward assertion on requests/responses.

2. Axum Test

The Axum Test approach uses the axum-test crate. This library is specifically designed to help test Axum applications:

You can use TestServer with a mock transport, removing the need for a real network connection.
It provides an intuitive API for building requests, handling multipart data, and asserting responses.

Here’s a short look at examples/axum_test_example.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use anyhow::Result;
    use axum_test::multipart::{MultipartForm, Part};
    use axum_test::TestServer;
    use std::fs;

    #[tokio::test]
    async fn test_with_axum_test() -> Result<()> {
        // Prepare Axum app
        let app = create_app();
        let server = TestServer::builder().mock_transport().build(app)?;

        // Create multipart form
        let file = fs::read("./dummy/test_upload.csv")?;
        let part = Part::bytes(file)
            .file_name("test_upload.csv")
            .mime_type("text/csv");
        let form = MultipartForm::new()
            .add_text("file", "hoge-able")
            .add_part("csv", part);

        // Send the request
        let response = server.post("/upload").multipart(form).await;

        // Assert
        response.assert_status_ok();
        response.assert_text("Files uploaded successfully");

        Ok(())
    }
}

Benefits

Axum-Specific: Tailored for Axum, so it feels very natural to write these tests.
Powerful Assertions: Includes methods like assert_text, making response checks more expressive.
No External Server: Uses an internal mock_transport and never binds to a port.

3. HTTP Server with Reqwest

Lastly, there is the full integration test approach. This technique starts an actual Axum server listening on a TCP socket, and then uses Reqwest to send multipart requests:

Closely simulates real-world scenarios.
Helps you catch issues that only appear in a truly networked environment (e.g. timeouts, actual HTTP headers).
Great for end-to-end testing, though slower than the previous methods.

An excerpt from examples/http_server_with_reqwest.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use anyhow::{Ok, Result};
    use reqwest::multipart::{Form, Part};
    use reqwest::Client;
    use std::fs;

    #[tokio::test]
    async fn test_with_http_server() -> Result<()> {
        // Prepare Axum app
        let app = create_app();
        let listener = TcpListener::bind("127.0.0.1:3001").await.unwrap();
        let addr = listener.local_addr().unwrap();

        // Spawn the server
        tokio::spawn(async move {
            axum::serve(listener, app).await.unwrap();
        });

        // Create multipart form
        let file = fs::read("./dummy/test_upload.csv")?;
        let part = Part::bytes(file)
            .file_name("test_upload.csv")
            .mime_str("text/csv")?;
        let form = Form::new()
            .text("file", "hoge-able")
            .part("csv", part);

        // Send request using Reqwest
        let response = Client::new()
            .post(&format!("http://{}/upload", addr))
            .multipart(form)
            .send()
            .await?;

        // Assert
        assert_eq!(response.status(), 200);
        assert_eq!(
            response.text().await?,
            "Files uploaded successfully"
        );

        Ok(())
    }
}

Benefits

Closest to Production: Involves real network connections, actual HTTP protocols, etc.
E2E Scenarios: Useful for testing external dependencies or any networking layers.
Catch Subtleties: May discover issues (like CORS, TLS, or routing) that other tests miss.

Conclusion

Testing file uploads in Axum does not have to be overly complicated. Whether you’re optimizing for speed, code simplicity, or realism, these three approaches give you flexibility:

Oneshot: Quick unit-like tests for your Axum routes.
Axum Test: A specialized library for Axum, which makes for concise tests.
HTTP Server: A complete integration approach using a live server and a Reqwest client.

Adopting the right approach—or mixing them together—ensures you have robust coverage. You can keep your tests fast while still ensuring your application behaves correctly when actual users upload files.

Happy testing!

Create blob object (Git)

kataoka_nopeNoshishi — Sat, 25 Mar 2023 08:23:04 +0000

Hello Dev community!

I'm noshishi, a apprentice engineer in Tokyo.
This article is about understanding Git from the inside by creating a simple program that add and commit.

I would like to continue my previous article on developing Git using Rust. Let's start by implementing the code to create a blob object.

The repository I actually created is My original git nss. The quality of the code isn't quite there yet and many parts are still incomplete, but you can do a straight line of local development!

Blob Object

"Blob" is an object that corresponds to file data.

// env method
use std::env;
// Pathbuf sturuct
use std::path::PathBuf;
// File sturuct
use std::fs::File;
// Trait for File sturuct
use std::io::prelude::*; 
// sha1 calculate crate
use sha1::{Digest, Sha1};

fn main() -> std::io::Result<()> {
    let filename = "first.txt";

    // Gathering the file content
    let mut path = env::current_dir()?; // get cwd
    path.push(PathBuf::from(filename)); // to absolute path

    let mut f = File::open(path)?; // open file
    let mut content = String::new(); // create buffer
    f.read_to_string(&mut content)?; // write buffer

    // objectは `header`+`\0`+`content`
    let blob_content = format!("blob {}\0{}", content.len(), content); // Note: In Rust, len() function returns the byte count of a string, not the character count.

    // Content to be stored
    println!("blob content: {}", blob_content);

    // Calculate hash value
    let blob_hash = Sha1::digest(blob_content.as_bytes());
    println!("blob hash: {:x}", blob_hash);

    Ok(())
}

Before executing, we are using an external crate called sha1, so we list the dependency in Cargo.toml as follows.

[dependencies]
sha-1 = "0.9.1"

So let's create a new file first.txt and run it.

% echo Hello World! >> first.txt
% echo This is my original git project! >> first.txt
% cargo run
blob content: blob 45Hello World!
This is my original git project!
blob hash: b4aa0076e9b36b2aed8ca8a21ccdd210c905660a

Does this match the Blob object created by Git? You can use the git hash-object command to verify that it really does.

% git hash-object first.txt
b4aa0076e9b36b2aed8ca8a21ccdd210c905660a

Excellent! They matched properly!

Finally, we implement the process of writing the Object to the repository.

// omitting...

use std::fs; //add

// Compression crate
use flate2::Compression; //add
use flate2::write::ZlibEncoder; //add

fn main() {
    // omitting ...

    // Compressing the stored　content in the object
    let mut encoder = ZlibEncoder::new(Vec::new(), Compression::default());
    encoder.write_all(&blob_content.as_bytes())?; // Write by bytes
    let compressed = encoder.finish()?;

    // hash to string
    let hash = format!("{:x}", blob_hash);
    // Up to 2 characters of the hash value is the directory path and 38 characters is the file path
    let (dir, file) = hash.split_at(2);

    // Specify the save location by absolute path
    let mut current_path = env::current_dir()?; // path/to/ngit
    current_path.push(".git/objects"); // path/to/ngit/.git/objects
    current_path.push(dir); // path/to/ngit/.git/objects/b4

    let object_dir = current_path.clone(); // Not moving

    // Create a directory to store in `.git/obejects/`
    fs::create_dir_all(object_dir)?;

    // file path is 38 characters of the hash value
    current_path.push(file); // path/to/ngit/.git/objects/b4/aa0076e9b36b2aed8ca8a21ccdd210c905660a
    let object_path = current_path;
    // File contents are compressed contents
    let mut f = File::create(object_path)?;
    f.write_all(&compressed)?;
    f.flush()?;
}

Since we used a new external crate, we will also add it to the toml file.

[dependencies]
flate2 = "1.0.25" # add
sha-1 = "0.9.1"

Now let's run it.

% cargo run
blob content: blob 45Hello World!
This is my original git project!
blob hash: b4aa0076e9b36b2aed8ca8a21ccdd210c905660a

# Objects are properly stored.
% ls .git/objects/b4
aa0076e9b36b2aed8ca8a21ccdd210c905660a

# look at the contents with `git-cat-file`
% git cat-file -p aa0076e9b36b2aed8ca8a21ccdd210c905660a
Hello World!
This is my original git project!

We can easily create a blob object. In the next article, we will create a function to create this blob! See the next one if you like!

Thanks for reading to the end!

Make original Git by Rust! (Analyze section)

kataoka_nopeNoshishi — Sun, 12 Feb 2023 08:18:13 +0000

Hello Dev community!

I'm noshishi, a apprentice engineer in Tokyo.
This article is about understanding Git from the inside by creating a simple program that add and commit.

But it's a very long story, so I'll post the development section separately!

Foreword

The starting point is 'If I could understand git, I could make it?!!'

I took this opportunity to try out a new programming language, so I decided to try Rust this time. The repository I actually created is My original git nss. The quality of the code isn't quite there yet and many parts are still incomplete, but you can do a straight line of local development!

If you give me a star, I'll be happy to fly, and of course I'll be waiting for your contributions! Feel free to touch this repository any way you like!

Please forgive us for not being able to explain some of the details in this article alone. Also, we use Rust for development, but Python for the stage where we uncover Git's internals!

Git Inside

First, we will unpack how Git handles data, based on the official documentation.
The Git command system is very complex.
But, Git data structure is very simple!

Where is repository

A repository is the directory under the control of Git, and the folder .git in the directory created by init or clone is the actual state of the repository.

Let's put an empty folder called project under Git's control.

$ pwd
/home/noshishi/project
$ ls -a
# nothing yet
$ git init
Initialized empty Git repository in /home/noshishi/project/.git/
$ ls -a
.git

This .git directory consists of the following.

.git
├── HEAD
├── (index)  // Not created by `init`!
├── config
*
├── objects/
└── refs/
    ├── heads/
    └── tags/

(info)

The path types of Git repositories are difficult to understand at first glance. We have added / to the directory path so that you can refer to it. Also, we have omitted parts that are not explained in this article.

Object

Git manage versions by file data called objects.

Objects are stored in .git/objects.

Types

Objects has four types, blob、tree、commit、tag.

The contents of each and the corresponding data will be as follows.

blob ... File data
tree ... Directory data
commit ... Metadata to manage the tree of the repository
tag ... Metadata for a specific commit * Not explained at this article.

Image with first.txt in the project repository

Structure

The Object is FILE DATA, so it has a file name (path) and the data stored in it, just like a normal file.

File name (path)
The file name (path) is 40-character string. This is a hash (sha-12) of object data.

Actually, the first two are the directory path and the remaining 38 are the file path.

Data
Object data is compressed by zlib1. The decompressed data consists of two parts: header and content. The two elements are then separated by \0 (null byte).

header is a combination of the object type and the size of content.

content contains the corresponding data in an easy-to-handle format, as indicated by the type. (Later we will see the details).

How to Create blob Object

Index (staging area)

The actual index used when you add is a file .git/index.

Structure

The index stores data of files marked by add with meta information. The stored data contains the latest file data at the time of add.

It is important to note that all data recorded in the index is in file data units.
I will describe meta information in detail later, but the storage format is exactly defined as shown in index-format.

Hmm.... feel sleepy....

Wait!

Let's actually analyze the object and the index!

Analyze Object

Before starting the analysis work, create all of the blob, tree, and commit.
Just add the files in project and commit.

Createing the following two files...

first.txt

Hello World!
This is first.txt.

second.py

def second():
    print("This is second.py")

next, add and commit.

git add -A
git commit -m 'initial'

Then the contents of .git/objects are now as follows.

.git/
└── objects/
    ├── 48/
    |   └── c972ae2bb5652ada48573daf6d27c74db5a13f
    ├── af/
    |   └── 22102d62f1c8e6df5217b4cba99907580b51af
    ├── da/
    |   └── f3f26f3fa03da346999c3e02d5268cb9abc5c5
    └── f7/
        └── f18b17881d80bb87f281c2881f9a4663cfcf84

**From now on, hash values in the text will omit the number of characters. 3*

The corresponding data and hash values for each are summarized below.

hash value	Object	correspond data
`f7f18b1`	`blob`	first.txt
`af22102`	`blob`	second.py
`daf3f26`	`tree`	project direcrtory
`48c972a`	`commit`	commit version 1

*The analysis work will be conducted interactively using Python, an interpreted language.

blob

blob is an object corresponding to file data.
The image looks like this.

Data

First, let's look at f7f18b1, which corresponds to first.txt.

...Oops, I failed.

% python
>>> with open('.git/objects/f7/f18b17881d80bb87f281c2881f9a4663cfcf84', 'r') as f:
...     contnet = f.read()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 3: invalid continuation byte

Since the content is compressed, attempting to read the content as-is as a string 4 will fail.
Therefore, we read the content as binary.

>>> with open('.git/objects/f7/f18b17881d80bb87f281c2881f9a4663cfcf84', 'rb') as f: # read binary!
...     contnet = f.read()
>>> content
b'x\x01K\xca\xc9OR06d\xf0H\xcd\xc9\xc9W\x08\xcf/\xcaIQ\xe4\n\xc9\xc8,V\x00\xa2\xb4\xcc\xa2\xe2\x12\xbd\x92\x8a\x12=\x00\xfa-\r\x03'

Then I read successfully and the byte string.

Now, decompress the content with zlib, as described in the official documentation.

>>> import zlib
>>> decompressed = zlib.decompress(content)
>>> decompressed
b'blob 31\x00Hello World!\nThis is first.txt.'
>>> decompressed.split(b'\0')
[b'blob 31', b'Hello World!\nThis is first.txt.']

We found that a blob consists of the following elements

header ... blob 31
Null byte ... \x00　※hex notation
content ... Hello World!\nThis is first.txt.

File name

We should check whether the hash value of the object is indeed correct.

The file name of the object should be the value obtained by hashing decompressed with the hash function sha1, so check it.

>>> import hashlib
>>> blob = b'blob 31\x00Hello World!\nThis is first.txt.'
>>> sha1 = hashlib.sha1(blob).hexdigest()
>>> sha1
'f7f18b17881d80bb87f281c2881f9a4663cfcf84'

Great, exact match!

How about another file

Let's also look at af22102, which corresponds to the other second.py.

>>> with open('.git/objects/af/22102d62f1c8e6df5217b4cba99907580b51af', 'rb') as f:
...     contnet = f.read()
>>> decompressed = zlib.decompress(content)
>>> decompressed
b'blob 44\x00def second():\n    print("This is second.py")'

>>> blob = b'blob 44\x00def second():\n    print("This is second.py")'
>>> sha1 = hashlib.sha1(test).hexdigest()
>>> sha1
'af22102d62f1c8e6df5217b4cba99907580b51af'

It can be summarized as follows

header ... blob 44
Null byte ... \x00
content ... def second():\n print("This is second.py")

And the sha1 values (hash values) derived from the data also matched.

Supplemental
The blob itself does not hold the filename of the corresponding file data.

Instead of blob, the object that manages its name is tree.

Tree

tree is an object corresponding to directory data.
The image looks like this.

We will analyze it in the same way as for blob.

>>> with open('.git/objects/da/f3f26f3fa03da346999c3e02d5268cb9abc5c5', 'rb') as f:
...     content = f.read()
>>> decompressed = zlib.decompress(content)
>>> decompressed
b'tree 74\x00100644 first.txt\x00\xf7\xf1\x8b\x17\x88\x1d\x80\xbb\x87\xf2\x81\xc2\x88\x1f\x9aFc\xcf\xcf\x84100644 second.py\x00\xaf"\x10-b\xf1\xc8\xe6\xdfR\x17\xb4\xcb\xa9\x99\x07X\x0bQ\xaf'
>>> decompressed.split(b'\0')
[b'tree 74',
 b'100644 first.txt',
 b'\xf7\xf1\x8b\x17\x88\x1d\x80\xbb\x87\xf2\x81\xc2\x88\x1f\x9aFc\xcf\xcf\x84100644 second.py',
 b'\xaf"\x10-b\xf1\xc8\xe6\xdfR\x17\xb4\xcb\xa9\x99\x07X\x0bQ\xaf']

The tree has multiple contents, so we seem a bit complicated.

The tree contnet is composed of repeating mode5, path and hash, which are meta information about the data in the directory,

If you simply separate them with \0, the hash value of the previous data and the meta information of the next file data are attached to each other.

This is because the meta information and the hash value are separated by \0.

First, we will check the data stored in the first one.
Looking at the split, like first.txt is stored, right?

>>> temp = decompressed.split(b'\0')
>>> temp[1]
b'100644 first.txt'
>>> temp[2]
b'\xf7\xf1\x8b\x17\x88\x1d\x80\xbb\x87\xf2\x81\xc2\x88\x1f\x9aFc\xcf\xcf\x84100644 second.py'

In order to split temp[2] well, let's take it out by 20 bytes.
Array access of byte strings can be byte.

>>> temp[2][0:20]
b'\xf7\xf1\x8b\x17\x88\x1d\x80\xbb\x87\xf2\x81\xc2\x88\x1f\x9aFc\xcf\xcf\x84'
>>> temp[2][0:20].hex()
'f7f18b17881d80bb87f281c2881f9a4663cfcf84'
>>> temp[2][20:]
b'100644 second.py'

Repeating the same process revealed the following.

header ... tree 74
Null byte ... \x00
content1 ... 100644 first.txt\x00f7f18b1...
content2 ... 100644 second.py\x00af22102...

The management of tree hashes is described in (Digression) deciphering Tree bytes!

Supplemental
A tree may contain not only a blob but also a tree.
That is, if there is a directory within a directory.
This is because tree, like blob, does not keep the directory name of itself and the corresponding data.

Commit

commit contains the tree of the repository directory with meta information.
The image looks like this.

Let's analyze!

>>> with open('.git/objects/48/c972ae2bb5652ada48573daf6d27c74db5a13f', 'rb') as f:
...     content = f.read()
>>> decompressed = zlib.decompress(content)
>>> decompressed
b'commit 188\x00tree daf3f26f3fa03da346999c3e02d5268cb9abc5c5\nauthor nopeNoshishi <nope@noshishi.jp> 1674995860 +0900\ncommitter nopeNoshishi <nope@noshishi.jp> 1674995860 +0900\n\ninitial\n'
>>> decompressed.split(b'\0')
[b'commit 188',
 b'tree daf3f26f3fa03da346999c3e02d5268cb9abc5c5\nauthor nopeNoshishi <nope@noshishi.jp> 1674995860 +0900\ncommitter nopeNoshishi <nope@noshishi.jp> 1674995860 +0900\n\ninitial\n']

# a little bit more
>>> header, content = decompressed.split(b'\0')
>>> header
b'commit 188'
>>> content
b'tree daf3f26f3fa03da346999c3e02d5268cb9abc5c5\nauthor nopeNoshishi <nope@noshishi.jp> 1674995860 +0900\ncommitter nopeNoshishi <nope@noshishi.jp> 1674995860 +0900\n\ninitial\n'
>>> content.split(b'\n')
[b'tree daf3f26f3fa03da346999c3e02d5268cb9abc5c5',
 b'author nopeNoshishi <nope@noshishi.jp> 1674995860 +0900', 
 b'committer nopeNoshishi <nope@noshishi.jp> 1674995860 +0900', 
 b'', 
 b'initial',
 b'']

The stored data are as follows.

header ... commit 188
Null byte ... \x00
tree ... tree daf3f26f3fa03da346999c3e02d5268cb9abc5c5
author ... author nopeNoshishi <nope@noshishi.jp> 167...
committer ... committer nopeNoshishi <nope@noshishi.jp> 167...
message ... initial

You can see that it contains the tree hash value that you saw in the tree chapter earlier, information about the repository owner and the person who made the commit, and the message.

I will go ahead with the commit and analyze it again.
Edit first.txt as follows and add and commit again.

first.txt(version2)

Hello World!
This is first.txt.
Version2

git add first.txt
git commit -m 'second'

Then the contents of .git/objects are now as follows.

.git/
└── objects/
    ├── 3f/
    |   └── f934272  # new tree .. project repo version 2
    ├── 37/
    |   └── 349c9b0  # new commit .. "second"
    ├── 48/
    |   └── c972ae2  # old commit .. "initial"
    ├── af/
    |   └── 22102d6  # old blob .. second.py version 1
    ├── c8/
    |   └── 843b4db  # new blob .. first.txt version 2
    ├── da/
    |   └── f3f26f3  # old tree .. project repo version 1
    └── f7/
        └── f18b178  # new blob .. first.txt version 1

See the new commit...

>>> with open('.git/objects/37/349c9b05c73281008e7b6b7453b595bb034a52', 'rb') as f:
...     content = f.read()
... 
>>> decompressed = zlib.decompress(content)
>>> decompressed
b'commit 235\x00tree 3ff9342727caf81397740327aa406c1cc6d4408e\nparent 48c972ae2bb5652ada48573daf6d27c74db5a13f\nauthor nopeNoshishi <nope@noshishi.jp> 1675174139 +0900\ncommitter nopeNoshishi <nope@noshishi.jp> 1675174139 +0900\n\nsecond\n'

The stored data are as follows.

header ... commit 188
Null byte ... \x00
tree ... tree daf3f26f3fa03da346999c3e02d5268cb9abc5c5
parent ... parent 48c972ae2bb5652ada48573daf6d27c74db5a13f
author ... author nopeNoshishi <nope@noshishi.jp> 167...
committer ... committer nopeNoshishi <nope@noshishi.jp> 167...
message ... second

The new commit stored the hash value of the previous version of commit.

Supplemental

The difference against blob or tree is that commit does not store the actual data in the repository. But it has meta data starting from tree.

Key-Value Store

Some of you may have an idea of what I'm talking about.

If you unravel a commit, you can get a tree, and if you unravel a tree, you can get a blob.

The version flow shows the history because commit knows the hash value of the previous commit.
This image shows the history of the current commit.

So Git manages file versions from the starting point, which is the hash value of the object.

(Info)
Officially, Git is called Address (hash) File System.
The hash function itself is an invertible transformation, so the original data cannot be restored from the hash value, but as long as the hash value depends on the contents of the object to begin with, it may be called a value-value store.

Summary

In a world without version control systems like Git, what do you do when you want to keep your current files and work on something new with the same files?
Perhaps one way you might think of doing this is to copy the file and put it in another folder.
In fact, this seemingly weird management method is the closest form of version control that supports Git.

(Info)
Git is a storage system that makes clever use of the OS file system.

Analize Index

The index (staging area) is veiled, but like the object, the design is very simple.
(On the other hand, it is a bit quirky to analyze. The dismantling of the index sucked up dozens of hours...

I'm going to analyze .git/index, which has been committed for the second time.

Specification

In order to analyze, we need to understand the design specification of index.

Referring to Index format in the official document, we found the following specifications.

Index Format
Header
    - 4 bytes   Index header                * DIRC
    - 4 bytes   Index version   　　　　     * basic version 2
    - 32 bits   number of entries in index  * Entries are the meta information for each file.

エントリー
    - 32 bits   create file time
    - 32 bits   create file time at nano
    - 32 bits   modify file time
    - 32 bits   modify file time at nano
    - 32 bits   device id
    - 32 bits   inode
    - 32 bits   Permission (mode)
    - 32 bits   user id
    - 32 bits   group id
    - 32 bits   file size
    - 160 bits  `blob` hash value
    - 16 bits   filename size               * Number of bytes in filename string
    - ?  bytes  filename                    * Variable depending on file name
    - 1-8 bytes padding                     * Variable depending on entry

... The same thing continues by number of entries ....

Index

Now that we have the specifications, we will read them again in python.

The index is uncompressed, but reads in binary format as well as the object because all meta information is stored in bytes.

>>> with open('.git/index', 'rb') as f:
...     index = f.read()
>>> index
b'DIRC\x00\x00\x00\x02\x00\x00\x00\x02c\xd9 \xf4\x05\xeb\x80\xb2c\xd9 \xf4\x05\xeb\x80\xb2\x01\x00\x00\x06\x00\xb8\'\x07\x00\x00\x81\xa4\x00\x00\x01\xf5\x00\x00\x00\x14\x00\x00\x00(\xc8\x84;M\xb8\x06\xe5\xd6Z\x12\xefV\xbfK\xeeQ\xe7\x15\'\x93\x00\tfirst.txt\x00c\xd6hv\x17\xa5\x05nc\xd6hv\x17\xa5\x05n\x01\x00\x00\x06\x00\xb8\'\x14\x00\x00\x81\xa4\x00\x00\x01\xf5\x00\x00\x00\x14\x00\x00\x00,\xaf"\x10-b\xf1\xc8\xe6\xdfR\x17\xb4\xcb\xa9\x99\x07X\x0bQ\xaf\x00\tsecond.py\x00TREE\x00\x00\x00\x19\x002 0\n?\xf94\'\'\xca\xf8\x13\x97t\x03\'\xaa@l\x1c\xc6\xd4@\x8e\xf2\xe4\xd7:\x95\xc1?\x18\xd3\xe9\x7f\x8fp\x9c$N\xc9dX\xa4'

It looks readable in places.
You can see the original DIRC, first.txt and second.py!

Since 32bits is 4bytes, it can be easily pulled out.

>>> index[0:4]
b'DIRC' # Index header -> DIRC
>>> index[4:8]
b'\x00\x00\x00\x02' # Index version => 2
>>> index[8:12]
b'\x00\x00\x00\x02' # number of entries => 2

The index manages metadata per file, so you will have two entries, first.txt and second.py.

For the purpose of this article, I will just take a quick look at the meta information from the next creation time to the group ID, which is not very important except for the mode.

>>> index[12:16]
b'c\xd9 \xf4' # ctime
>>> index[16:20]
b'\x05\xeb\x80\xb2' # ctime nano
>>> index[21:24]
b'\xd9 \xf4' # mtime
>>> index[24:28]
b'\x05\xeb\x80\xb2'  # mtime nano
>>> index[28:32]
b'\x01\x00\x00\x06' # dev id
>>> index[32:36]
b"\x00\xb8'\x07" # inode
>>> index[36:40]
b'\x00\x00\x81\xa4' # mode
>>> index[41:44]
b'\x00\x01\xf5' # user id
>>> index[44:48]
b'\x00\x00\x00\x14' # gorup id

Here are the key points to look at.
First is the file size.

# file size
>>> index[48:52]
b'\x00\x00\x00('
>>> index[48:52][0]
0
>>> index[48:52][1]
0
>>> index[48:52][2]
0
>>> index[48:52][3]
40

The file size of the next file to come is found to be 40bytes.

Next is the hash value.

# hash
>>> index[52:72]
b"\xc8\x84;M\xb8\x06\xe5\xd6Z\x12\xefV\xbfK\xeeQ\xe7\x15'\x93"
>>> index[52:72].hex()
'c8843b4db806e5d65a12ef56bf4bee51e7152793'

We see the hash value matches the one in version 2 first.txt!

And the size of the filename.

# filename size
>>> index[72:74]
b'\x00\t'
>>> index[72:74][0]
0
>>> index[72:74][1]
9

This size (in bytes) is very important, without it, you will have to search for the next file name by your feeling.

Now that we know the filename is 9 bytes, we can...

>>> index[74:83]
b'first.txt'

We can extract the file name without missing anything.

Finally, padding depends on the number of bytes used to represent the entry.
The calculation method is to find X bytes such that the bytes up to the padding plus the X bytes to be padded is a multiple of 8.

Expressed as a formula, X (padding), y (filename size), a (remainder)

In this case, from creation time to file size, 62 bytes, and the file name is 9 bytes.

We found the bytes of padding was 1 byte.

>>> index[83:84]
b'\x00'
>>> index[83:85]
b'\x00c' # There's one that isn't a null bite, and it's from the second bite!
>>> index[83:86]
b'\x00c\xd6'

The bytes of padding up to the next entry creation time was correctly matched.

Summary

Actually, when you add, tree is not created.
You commit, then tree will be generated from index.

index has important role to link added file data to blobs and manage which versions of files are committed.

You may have heared git dealed a snapshot, not difference.
In other words, when indexes have not been updated, file data will always remain unless explicitly excluded.
And that means that everything you commit can be restored through the index.

(Info)
index is an important entity that holds the key to whether or not a file is subject to version control in Git.

Background of Command

Now that we know how Git handles data, let's take a quick look at how the commands behave.

The command has many options, so more complex behavior can be achieved, but I only describe a basic role.

add

add is responsible for adding, deleting, and updating the target file data to the index.
When added, git creates a blob of the instantaneous(latest) file data.

The plumbing commands that make this happen are hash-object and update-index.
※In Plumbing commands chapter, I describe the detail.

commit

Git create a tree corresponding to the repository directory based on the index created, and then create a commit.
After the commit is successfully created, change the hash value of the commit that the HEAD and branch point to.

The plumbing commands that accomplish this are write-tree, commit-tree, and update-ref.

Digression

Deciphering Tree

We'll look into the byte in a bit.

What is the maximum value of a number that can be represented by a single (unsigned) byte?
2^8 - 1 = 255. This corresponds to the maximum number of hexadecimal digits that can be represented by two hexadecimal digits.

>>> temp[2][0]
247　 # = `\xf7`

I used the hex() function quickly above, but if you look at it one byte at a time...

>>> hash = ''
>>> for hex in temp[2][0:20]:
...     hash += format(hex, 'x')
>>> hash
'f7f18b17881d80bb87f281c2881f9a4663cfcf84'

I can get the hash value of the blob corresponding to first.txt as a string!

hash are 40 characters, but each character is a value calculated in hexadecimal. So the trick is that one byte can represent two characters .

commit stores the hash value as a string, but for some reason the tree stores the hash value directly as bytes, not as a string.

There was some discussion on stackoverflow as to why.

https://stackoverflow.com/questions/42009133/how-to-inflate-a-git-tree-object

HEAD and Branch

The Branch is responsible for marking specific commit objects.
It is stored under .git/refs/heads/.
You can easily see the contents with the Linux command cat.

Since we were working on the master branch earlier, we can look at .git/refs/heads/master and see ...

% cat .git/refs/heads/master
37349c9b05c73281008e7b6b7453b595bb034a52

The hash value of the last committed commit object was stored.

The HEAD indicates which commit object you are basing your file edits on.
HEAD can point directly to a commit object, but it basically goes through branch.
.git/HEAD is what it is.

The data is stored as follows.

% cat .git/HEAD
ref: refs/heads/master

It contained the path about where the master branch is stored.

If you want to point directly to a commit (detached head), use checkout to move HEAD.

% git checkout 37349c9b05c73281008e7b6b7453b595bb034a52
% cat .git/HEAD
ref: 37349c9b05c73281008e7b6b7453b595bb034a52

Plumbing commands

To further manipulate Git at a low level, there is a command for every single action.
(These are god-like commands created by Mr. Linus for ordinary people like me.)

`cat-file`

This command allows you to see the contents of an object.
We worked hard earlier to analyze the object, but this single command is the solution.

# See object type
% git cat-file -t af22102d62f1c8e6df5217b4cba99907580b51af # second.py
blob

# Output object content
% git cat-file -p af22102d62f1c8e6df5217b4cba99907580b51af # second.py
def second():
    print("This is second.py")

`hash-object`

You can hash file data, etc. or store them in .git/objects.

Let's create third.rs.

struct Third {
    message: String   
}

# calculate hash value
% git hash-object
4aa58eed341d5134f73f2e9378b4895e216a5cd5

# Create blob object
% git hash-object -w
4aa58eed341d5134f73f2e9378b4895e216a5cd5
% ls .git/objects/4a
a58eed341d5134f73f2e9378b4895e216a5cd5

`update-index`

This command adds the target file to the index.
Note, however, that no object is created.

`ls-files`

This command provides a concise view of the contents of the index.

# see the latest index
% git ls-files
first.txt
second.py

# add index third.rs cache
% git update-index --add third.rs 
% git ls-files
first.txt
second.py
third.rs
% git ls-files -s
100644 c8843b4db806e5d65a12ef56bf4bee51e7152793 0       first.txt
100644 af22102d62f1c8e6df5217b4cba99907580b51af 0       second.py
100644 4aa58eed341d5134f73f2e9378b4895e216a5cd5 0       third.rs

`write-tree`

We create a tree based on the contents of the index.
All directories, not just repository directory.

% git write-tree
109e41a859caa3e3b87e8f59744b0b1845efe275
% ls .git/objects/10 
9e41a859caa3e3b87e8f59744b0b1845efe275

`commit-tree`

We create the commit with the hash of the (repository directory) tree.

# Enter the hash value of the parent `commit` and the 
# hash value of the `tree` you just created
% git commit-tree -p 37349c9b05c73281008e7b6b7453b595bb034a52 -m 'third commit' 109e41a859caa3e3b87e8f59744b0b1845efe275
ddb3c0d94d860ff657e2cdb82f5513f7db2924f1
% ls .git/objects/dd 
b3c0d94d860ff657e2cdb82f5513f7db2924f1　#　object is created

`update-ref`

We can't just commit-tree and follow the history, because no one will see the commits you made.
This is because no one can see the commits we have made.

# Because the git log follows the history sequentially 
# from the commit pointed to by HEAD, the commit you
# just created is not yet referenced.
% git log
commit 37349c9b05c73281008e7b6b7453b595bb034a52 (HEAD -> master)
Author: nopeNoshishi <nope@noshishi.jp>
Date:   Tue Jan 31 23:08:59 2023 +0900

    second

commit 48c972ae2bb5652ada48573daf6d27c74db5a13f
Author: nopeNoshishi <nope@noshishi.jp>
Date:   Sun Jan 29 21:37:40 2023 +0900

    initial

# Change the branch's references.
% git update-ref refs/heads/master ddb3c0d 37349c9 # new-hash old-hash
% git log
commit ddb3c0d94d860ff657e2cdb82f5513f7db2924f1 (HEAD -> master)
Author: nopeNoshishi <nope@noshishi.jp>
Date:   Thu Feb 2 21:17:24 2023 +0900

    third commit

commit 37349c9b05c73281008e7b6b7453b595bb034a52
Author: nopeNoshishi <nope@noshishi.jp>
Date:   Tue Jan 31 23:08:59 2023 +0900

    second

In creating Git, it is difficult to suddenly create something as sophisticated as add or commit.
Therefore, while implementing the plumbing command , we will create add and commit in the development section to bypass the functionality of this command.

Finally

Thank you for reading all the way to the end!!!
This is still a rough explanation, but I hope it contributes to your understanding.
If you may ok, please star my repository!

Reference Site

Officail Documentation

What you need

Listed here are the key elements in making git.

Binary

Byte

Bitwise operation

n-decimal system and character strings

String

Compression algorithms

Hash function

File system

Annotation

zlib

1: This is a free software to compress data losslessly. The main compression algorithm called Deflate is very interesting.Official Site back to article

sha1

2: One of the very famous SHA-based hash functions, characterized by the generation of a 60-bit (20-byte) hash value. Incidentally, the probability of a collision of sha1 hash values is said to be astronomical.The Reality of SHA1 back to article

hash number

3: When you specify a hash value directly in a Git command, you may only use 7 characters. As mentioned in [^2](#ano-2), this means that even with a small input hash value, we can find a specific object because there are almost no hash collisions. It is similar to the feeling of pressing tab in shell to receive input assistance. back to article

compressed string

4: Compressed data is stored in a form that does not correspond to a character code. Therefore, it cannot be read as a specific character code. back to article

mode

5: The mode (permission) can of course also be expressed in binary. And since there are few combinations, certain combinations can be expressed in computation. back to article

Understanding Git through images

kataoka_nopeNoshishi — Thu, 05 Jan 2023 15:22:20 +0000

Hello Dev community!

I am a newbie, still a few months into my career as a developer in Japan. I was inspired by Nico Riedmann's Learn git concepts, not commands, and I have summarized git in my own way. Of course, I supplemented it with reading the official documentation as well.
Understanding git from its system structure makes git more fun. I have recently become so addicted to git that I am in the process of creating my own git system.

Recently, I wrote how to make software like git!
Make original git

What is Git?
- Manage versions and Distribute work
- Using Git means
- Understanding by image
Start new work
- Repositories
- Copy the repository and start working
- (Supplemental) Working Directory
- Change and Add files
- Adapt to remote repositories
- View Differences
- (Aside) One step called staging area
- Summary
Branch
- Create new branch
- Work in Branches
- (Aside)Git-Flow and GitHub-Flow
- Summary
Merge
- Fast Forward
- No Fast Forward
- Deal with Conflicts
- Delete unnecessary branches
- (aside) What is the branch?
- Summary
Rebase
- Move the branch
- Deal with rebase conflicts
Keep local repositories up-to-date
- Branch and Repository
- Check the latest status
- Update to the latest status
- Deal with pull conflicts
- (Aside) Identity of pull requests
Useful Functions
- Correct the commit
- Delete the commit
- Evacuate the work
- Bring the commit
- Mastering HEAD
End
- Source code management without Git
- Where is the remote repository
- Pointer
- To further understand Git
Reference

What is Git?

Manage versions and Distribute work

Git is a type of source code management system called a distributed version control system.
Git is a tool to facilitate development work by recording and tracking the changelog (version) of files, comparing past and current files, and clarifying changes.
The system also allows multiple developers to edit files at once, so the work can be distributed.

Using Git means

First, make a copy of the file or other files in a storage location that can be shared by everyone (from now on referred to as "remote repository") on your computer (from now on referred to as "local repository"), and then add or edit new code or files.
Then, the files will be updated by registering them from the local repository to the remote repository.

Understading by image

When dealing with Git, it is important to follow "how to work" from "what" to "what".
If you only operate commands, you may not understand what is happening and use the wrong command.

(info)

When manipulating Git, try to imagine what is happening before and after the operation.

Start new work

Repositories

A repository in Git is a storage for files, which can be remote or local.

Remote Repository is a repository where the source code is placed on a server on the Internet and can be shared by everyone.
Local repository is a repository where the source code is located on your computer and only you can make changes.

Copy the repository and start working

First, prepare your own development environment.
All you need only to do is decide in which directory you will work.
For example, your home directory is fine, or any directory you normally use.

Next, copy and bring the files from the remote repository.
This is called clone.

The remote repository called project contains only first.txt, and this is the image when you clone the remote repository.

(info)

Of course, you may create a local repository first and then reflect the remote repository.
This is called initialize and allows you to convert a directory you are already working on into a repository.

(Supplemental) Working Directory

A working directory is not any special directory, but a directory where you always work on your computer.
It's easier to understand if you think of it as a directory where you can connect to the target directory that Git manages (in this case, project) with a Git staging area or local repository.

Change and Add file

Changes to the source code are made through the working directory, the staging area.
Actually, in the working directory, we work.

Let's create a new file called second.txt.

Next, move the modified file to the staging area.
This is called add.

It is a feature of Git that there is a cushion before changes are reflected in the local repository.
I will explain why this cushion exists in more detail later.

Then, we registere the content in the staging area to the local repository.
This is called commit.

By the way, we can comment when you commit.
In this case, we added a file, so write git commit -m 'add second.txt'.

(info)

When you commit, a commit object is created in the repository.
A simple explanation of a commit object is the data that has the updater's information and the modified file.
(All data is saved, not just the differences, but the entire state of the file at that moment (snapshot).
Please refer to Git Objects for more information about Git objects.

Adapt to remote repositories

Then, the work is done!
The last step is to reflect the changes in the local repository to the remote repository.
This is called push.

It may be easier to understand if you think of it as a commit to a remote repository.

View Differences

Changes between the same file are called diff.
We can see the changing points in the file.

I won't go into the details of the commands, but here are three that I use frequently.
git diff --stage to see the changes from the original working directory before you add.
git diff --stage to see changes to the working directory after add.
git diff <commit> <commit> to compare commits.

(Aside) One step called staging area

As development work grows, we often make many changes in one working directory.
What happens if you put all the changes in a local repository at once?
In this case, when parsing the commits, you may not know where a feature was implemented.

In Git, it is recommended to do one commit per feature.
This is why there is a staging area where you can subdivide the commit unit into smaller units.

The concept of Git is to stage only what is needed, and then proceed with the work or commit ahead of time to promote efficient development that can be traced back through the history of each implementation.

Summary

The basic workflow is to clone once and then add, commit, and push for each working.

(info)

clone: Make a copy from the remote repository to your development environment (local repository and working directory).
add: Add files from the working directory to the staging area and prepare them for commit.
commit: Register the file from the staging area to the local repository. At this time, a commit object is created.
push: Register changes from the local repository to the remote repository.

Branch

We create a branch to change and add files in multiple branches.
The files saved in the main branch are in ongoing use.
The reason for the separate branches is to work without affecting the currently running source code.

Create new branch

Let's create the branch called develop!
We can create a branch with git branch <new branch> or git checkout -b <new branch>.
The former just create a branch, the latter create a branch and moves you to that branch.
(Branches are maintained in the repository.)

The key point when generating branches is which branch to derive from.
We can specify the source as git checkout -b <new branch> <from branch>.
If we don't, the branch you are currently working on becomes the <from branch>.

(info)

A branch is actually a pointer to the commit (strictly speaking, a hash of commit objects).
Generating a new branch means that the new branch indicate to the commit that the from　branch pointed to as well.

Work in Branches

Moving the branch is called checking out.
The pointer to the branch you are currently working on is called HEAD.
So, moving from the main branch to the develop branch means changing the HEAD.

Now both branches point to the commit named Atr3ul.
You just added second.txt by committing in the main branch, so you are ahead of the commit f27baz.
From here, let's say you change second.txt in the develop branch and make a new commit.

Then, as shown in the figure, the develop branch created a commit called m9sgle and pointed to that commit.

The current HEAD position (working branch position), what stage the file has been worked on, or the status of who is working on it is called status.

(info)

If you are familiar with object-oriented, you may understand the reason for the arrow on the commit.
It represents the relationship between a "parent" commit and a "child" commit.
The assumption is that parent←-child, that is, how much the child (commit) born from the parent (commit) has grown (changed).

(Aside)Git-Flow and GitHub-Flow

The way branches to manage will vary on development team.
On the other hand, like programming naming conventions, there is a general model for how to grow branches in Git.
Here are two simple ones. I think it's enough to know that there is such a thing.

The "Git Flow" is a fairly complex and intricate structure.
I think it's a model of how Git should be used.

Definition of each branch.

master: Branch to release a product. No working on this branch.

development: Branch to develope a product. When ready to release, merge to release. No working on this branch.

feature: Branch for adding features, merged into development when ready for release.

hotfix: For urgent post-release work (critical bug fixes, etc.), branch off from master, merge into master, and merge into develop.

release: For preparation of product release. Branch from develop with features and bug fixes to be released.
When ready for release, merge to master and merge to develop.

The "GitHub Flow" is a somewhat simplified model of the Git Flow.

As you can see, it consists of only master and feature.
The important difference is the cushion of pull requests (explained in the pull below), which allows integration between branches.

Summary

Basically, since there is no work on main (master), we create a branch for each work unit we want to do and create a new commit.

(info)

branch: New pointer to the commit
checkout: Move HEAD to change the branch to work on.

Merge

integrating the branches is called merge.
Basically, we merge into the main or develop branch.
Be careful not to mistake the subject of which branch is merging (absorbing) which branch.
We will always move (HEAD) to the branch from which you are deriving, and then do the integration from the branch from which you are deriving.

I am currently working on the feature branch and have created the following third.txt.

third.txt



Hello, World! I'm noshishi, from Japan.
I like dancing on house music.

Then We add and finished up to commit.

Fast Forward

When the feature branch points to the commit that can be traced back to the develop branch, the develop branch is in a fast-forward state.

First, move to develop with checkout.

In this case, the develop branch has not progressed at all, so to merge the feature branch will simply move the commit forward.
In this case, the develop and feature branches share the same commit.

No Fast Forward

What if the develop branch has progressed to a new commit by commit or merge?
This is called a no fast-forward situation.

In the develop branch, you have made changes to first.txt and have finished commit.
So the develop branch and the feature branch are completely split.

If you try to merge a feature branch from a develop branch, Git will check your changelog against each other.
If there are no conflicting edits, a merge commit is created immediately.
This is called an automatic merge.

Deal with Conflicts

In the no fast-forward state, the differences in work content. is called conflict.
In this case, we must manually fix the conflict content and commit.

In the develop branch, we created the following third.txt and committed.

third.txt



Hello, World! I'm nope, from USA.
I like dancing on house music.

In the develop branch, I'm nope, from USA.
In the feature branch, I'm noshishi, from Japan.
The content of the first line is in conflict.

If you do a merge at this time, a conflict will occur.
Git will ask you to commit after resolving the conflict.

(The branch we work on is the develop branch)

If you look at third.txt as instructed, you will see the following additions

third.txt　（after conflict）



<<<<<<<< HEAD
Hello, World! I'm noshishi, from Japan.
=======
Hello, World! I'm nope, from USA.
>>>>>>>> feature
I like dancing on house music.

The upper HEAD, separated by =======, represents the contents of the develop branch.
The lower side represents the feature branch.

You first considered which one to adopt, and decided to adopt the changes made in the feature branch this time.
The only operation then is to edit third.txt by hand (delete unnecessary parts).

third.txt　(After editing)



Hello, World! I'm noshishi, from Japan.
I like dancing on house music.

And the next thing you do is add and commit.
The conflict is resolved and a new merge commit is created.

Conflicts are feared by beginners, but once you learn this, you will no longer be afraid.

(info)

If you merge and resolve the conflict, why not merge again?
When you merge once, the develop branch enters the merge state, and if there are no conflicts, the new files are automatically added and commit.
So it is not a special commit after conflict is resolved.
That's why it's called merge commit.

Delete unnecessary branches

The merged branch is basically useless, so we will delete it.
If we leave a branch alone, you can move from the branch you want to delete to another branch and git branch -d <branch>.
You may think the commits on that branch are deleted.
In fact, the commits are carried over to the merged branch.
You can use git log to see all the commits you've made on the branch and the commits on the merged branch.

(Aside) What is the branch

We said that a branch is a pointer to a commit, but it also holds another important data.
It is all the commits that have been made on that branch.

A branch is a collection of commits, and it has a pointer to the latest commit in that collection. (Strictly speaking, the commit can trace back to previous commits.)

The following diagram illustrates this.

So we can think of branches on a horizontal axis like Git Flow.
By the way, if you draw the above diagram with branches on the horizontal axis, it looks like this.

Summary

fast-forward merge

no fast-forward merge

no fast-forward merge with conflict

(info)

merge: To integrate (absorb) a working branch (such as feature) into a specific branch (such as main or develop) and create a new commit.

Rebase

Rebase is the process of merging branches by changing the commit from which the branch is derived.
It is similar to merge, except that the branch you are working on is the destination branch.

Suppose you are working on the develop and feature branches.

Move the branch

You may think to reflect the current commit on develop branch into feature branch.
You need to move feature branch from the gp55sw commit to the 3x7oit commit.

This can be moved at once from the feature branch by doing a git rebase develop.

This process is more like re-growing the feature branch from the latest commit on the develop branch than doing a merge.
The difference is that you move the entire commit and make a new commit.

One reason for such a move is that it is fast-forward and easy to merge at any time.
The other reason is that the commits are aligned so that the commit history can be easily traced and the order in which files are updated is consistent.

Deal with rebase conflicts

Of course there is also a conflict in rebase.
You added fourth.txt in the feature branch, but you didn't change fourth.txt in the develop branch.
There is conflict.

However, if the following changes are covered by each other, conflict will occur.

You can just deal with it the same way you would with merge.
However, After you have checked the diff and finished editing the file, you should finish your work with git rebase --continue.
You don't have to commit, it will commit automatically.

(info)

rebase: Move the commit from which the derived branch to a new commit.

Keep local repositories up-to-date

After some local work, you may be faced with a situation where the remote repository has been updated by another developer.
In this case, you can use pull to re-install the information from the remote repository back into the local repository.

Branch and Repository

Branches are stored in each repository.
This is the branch where the actual work is done.

On the other hand, the local repository has the copied branches of the remote repository.
This is called a "remote tracking branch".
It is a branch with a name that is tied to the remote branch in remotes/<remote branch>.

This is only monitoring the remote repository.

Check the latest status

Suppose you have a situation where the develop branch in the remote repository is one step ahead of the remote tracking branch.

Reflecting the latest status of a branch in a remote repository on a remote tracking branch is called fetch.

Update to the latest status

If you want to have it reflected in your local branch, you can do a pull.
When you pull, the local remote tracking branch is updated first.
Then merge to the local branch.

This time, there was a commit that went one branch ahead of the develop branch, so you created a new commit by merge into the local develop branch.

Deal with pull conflicts

When a remote repository commit conflict with a local repository commit, you face the conflict between the remote tracking branch and the local branch when you pull.
In the following case, the remotes/develop and develop branches are in conflict.

Since push is fetch and merge, you can solve in the same way as conflict in merge.
This time, develop merges remotes/develop, so the working branch is develop.
Open the folder that caused the problem and commit when you have fixed it.

(Aside) Identity of pull requests

Basically, the relationship between remote and local is pull from the remote repository to the local repository and push from the local repository to the remote repository.
However, GitHub and other services have a mechanism to send a request before merge from a branch in a remote repository to a branch such as main.
This is because if a developer pushes to the main branch and updates the remote repository, no one can check it and a major failure may occur.
Pull request is to insert a process where a higher level developer reviews the code once.

(info)

pull: fetch + merge. pull is to reflect the state of the remote repository in the local repository.

Useful Functions

Correct the commit

To commit to correct a previous commit is called revert.
For example, suppose you added second.txt to your local repository with m9sgLe.

When you revert, the commit is revoked and second.txt is no longer in the local repository.

The merit of revert is that it allows you to leave commit.
Distinguish this from reset, which will be introduced later.

Delete the commit

To undo the current latest commit and work on it again is called reset.

The --soft option allows you to go back to the stage immediately after add.
The --mixed option allows you to go back to the stage where you were working in the working directory.
The --hard <commit> option removes all commits up to the commit point you are returning to and moves head to the specified commit.

Since reset completely deletes the commit, it is recommended that you do not use it unless you have a good reason, especially for the '--hard' option.

If you want to get your commits back, you can use git reflog to see the commits you have deleted.

Evacuate the work

Since you can't move to another branch if there are change files, you have to choose between going to commit or discarding your changes.
This is where stash comes in handy.
You can temporarily evacuate files in the working directory or staging area.

When you want to move to another branch, stash and when you return, use stash pop to retrieve the evacuated files and resume work.

Bring the commit

Bringing any commit to the current branch to create a commit is called cherry-pick.
It is a very nice feature.

This is used when you want to bring back only features previously implemented in a feature branch and use them for work in the current develop branch, for example.

Mastering HEAD

I explained that HEAD is a pointer to the branch you are currently working on.
I also explained that a branch is a pointer to a commit.

See the figure below.

HEAD points to the develop branch, and the develop branch points to the commit eaPk76.
So, HEAD in this situation refers to the commit eaPk76.

Have you often seen Git documentation or articles that use HEAD after a command?
For example, git revert HEAD.
This is a command that can be achieved because you can replace HEAD with commit.

End

Source code management without Git

Mercurial has the same history as Git.
Mercurial has a very simple command line interface (CLI) that sacrifices the flexibility of Git.
Recently, based on Mercurial, Meta released a new source code management system called Sapling as open source.
I would like to try it again and write about my impressions.

Where is the remote repository

A hosting service is a service that rents a server for a remote repository.
Typical examples are GitHub, Bitbucket, and Aws Code Commit for private use.
Git and Git Hub are completely different.
By the way, as mentioned above, we can use our own servers for remote repositories.

Pointer

If you have been exposed to programming that deals directly with memory, such as the C programming language, you will somehow know what a "pointer" is.
On the other hand, for a beginning programmer, it seems very vague.

I said that commit objects are stored in the repository.
If there are many commit objects in the repository, how can you select the one you want?

We need a label (address) to locate a particular commit object.

The "pointer" is a valuable data that indicates us to the label so that we don't forget it.

The label, by the way, is converted into a mysterious string through a hash function.
If you are curious, please refer to How does Git compute file hashes?.

To further understand Git

There are many things I failed to mention in this article.

The core of Git is a simple key-value type data store
Details of the Git object that is the value
How to relate with each objects.

I hope to fully explore this someday.

Forem: kataoka_nopeNoshishi

Testing Multipart Upload Requests in Axum: Three Approaches

Introduction: How Axum Handles Tests

Testing Approaches

1. Oneshot Testing

Benefits

2. Axum Test

Benefits

3. HTTP Server with Reqwest

Benefits

Conclusion

Create blob object (Git)

Hello Dev community!

Blob Object

Make original Git by Rust! (Analyze section)

Hello Dev community!

Foreword

TOC

Git Inside

Where is repository

Object

Types

Structure

Index (staging area)

Structure

Analyze Object

blob

Data

File name

How about another file

Tree

Commit

Key-Value Store

Summary

Analize Index

Specification

Index

Summary

Background of Command

add

commit

Digression

Deciphering Tree

HEAD and Branch

Plumbing commands

cat-file

hash-object

update-index

ls-files

write-tree

commit-tree

update-ref

Finally

Reference Site

What you need

Annotation

zlib

sha1

hash number

compressed string

mode

Understanding Git through images

Hello Dev community!

What is Git?

Manage versions and Distribute work

Using Git means

Understading by image

Start new work

Repositories

Copy the repository and start working

(Supplemental) Working Directory

Change and Add file

Adapt to remote repositories

View Differences

(Aside) One step called staging area

Summary

Branch

Create new branch

Work in Branches

(Aside)Git-Flow and GitHub-Flow

`cat-file`

`hash-object`

`update-index`

`ls-files`

`write-tree`

`commit-tree`

`update-ref`