Forem: Guilherme Niches

Clone your Github repositories using shell script

Guilherme Niches — Mon, 08 May 2023 22:11:00 +0000

It's a common scenario we lost our local projects because of problems in our machine. Our Laptop/Desktop can break down and we lost a lot of data. It's a painful situation, recovery not only all data, photos, software's, etc. But, recovery all our repositories too.

To our happiness, all repositories are storage/save in cloud through GitHub if we use Github as hosting service.

However, if we work simultaneously on various projects could be a little boring clone all repositories one-by-one.

Automate tasks is a great way to make a better use of our time. And shell script is a good resource to achieve this type of automation.

In this article we will create a shell script that automatically clone all repositories of your GitHub account.

Shell Script

Shell script is a script language used in command line environment. It allow us automate terminal commands in Unix based systems like Linux and MacOS.

Let's start

This article assumes that you have a basic knowledge about terminal commands, so is not the focus here explain what instructions like touch, mkdir, etc do.

First of all, we need to create a new script file.


 clone-github-repositories.sh

Note that the ".sh" is the extension of shell script files, without it the interpreter cannot understand that this file is a shell script.

After, we need open this file in a text editor of your preference (vi, nano, etc).


 clone-github-repositories.sh

In the editor we need write in the first line something very important to our script: indicate the interpreter of the script. In my case, I will use bash, but, exists others like sh, ksh, etc.

So, the first line of our script will be the following:


 #!/bin/bash

We will create a directory where we'll clone all our repositories. I will call this directory "projects", but you can name it like you want.

So the next line of our script will be the this command:


 mkdir projects

Like I said, we will clone our repositories inside projects directory, so we need enter to the projects directory, the following line of script will be:


 cd projects

Okay, so until now we have some simple instructions in our script, we need have access to all repositories from our GitHub account, we could go to our GitHub profile and copy the URL repositories one-by-one, but, this is not an agile way, right? Here, enter in scene the GitHub REST API.

The GitHub REST API allow us make integrations, automation and much more.

Our next line in script will be a request to GitHub API using curl:


 -s https://api.github.com/users/{your-github-username}/repos

Replace {your-github-username} by your GitHub username.

If you run this command solo in terminal the return will be a JSON content with a array of objects containing all properties of your repositories. Below an example of the JSON that return from the above command.

Between the many properties we have two especially important: ssh_url and clone_url.

This properties contains the URL's that we can use to clone the repositories. If you use ssh feature to clone your projects, you can use the "ssh_url", else, the "clone_url" will work the same way. In my case, I use SSH to clone my repositories.

Now, we have to work with some resources to manipulate content of files, because we don't need use all information that return from GitHub API.

Here, "grep" is our best friend. grep will search inside a file by a string/pattern that we inform.

We will refactor our script line to the following:


 curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url

Remember that if you don't use SSH you must change the "ssh_url" by "clone_url".

The output of this instruction will be something like this (if you run this directly in terminal):

To clone a project we need only the URL, so we need filter this grep return.

We will use the command "cut" with a delimiter to remove the unnecessary sections from the lines.

Refactoring our instruction, now the command will be this:


 curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url | cut -d'"' -f 4

-d is a flag for field delimiter and we use the double quotes (") as delimiter.

-f is a flag for fields that select only the field (or range of fields) informed. And 4 is the specified index of field line that we want extract.

If you run this instruction in terminal the output is like this:

Now, we have the exact URL to clone our repositories.

So, we will refactor the line to save the output list in a variable.


 repos=$(curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url | cut -d'"' -f 4)

repos is the name of variable and the "$" symbol indicate to the shell that this is a variable, we have a list of URL's in this variable.

We will need apply some programming logic to do the task, iterable the repos list and for each repository of the list we will clone the repository.


 for repo in $repos
do
  git clone $repo
done

The full script is below:



#!/bin/bash

mkdir projects
cd projects

repos=$(curl -s https://api.github.com/users/gniches/repos | grep ssh_url | cut -d'"' -f 4)

for repo in $repos 
do
    git clone $repo
done

It's done. Now we need run the script to test if our script is working as expected.


 bash clone-github-repositories.sh

If everything is ok, now you have a directory called projects with all your repositories inside.

Concluding

In this article we learned how to create a shell script to clone all our repositories from GitHub account using GitHub API.

Shell script is particularly useful to automate process and tasks, like monitoring servers, create periodic backups of databases, send e-mails and much more.

Reviews, suggestions and/or contributions to this article are encouraged and welcome.

References

GitHub REST API Documentation

How to manage multiple GitHub accounts on one computer with SSH keys

Guilherme Niches — Fri, 10 Mar 2023 16:31:32 +0000

In our journey through the world of programming it is very common to have to create more than one account on Github.

Either to have a personal account, which we use for our own projects, and a professional account linked to the company where we work. Either to have separate personal accounts for different purposes.

The question is, how can we use two or more different Github accounts on a single computer?

One of the possible answers is: using SSH keys.

SSH Keys

What is it?

SSH (Secure Shell) or Key Pair is a tool of remote access with focus on security, it's the means of identification between your computer and an SSH server using public encryption key authentication method.

One of it's advantages is that it does not transmit information such as login and password, so if your request is intercepted by malicious agents, they will not have access to your credentials.

Note: It's not the purpose of this article to explain in depth how the SSH keys works and it's types of encryption. However, at the end of the article I will put some links that help us to understand the concept in more depth.

Note 2: This article will focus on Linux/Mac commands, if you use Windows you can install Git Bash to emulate a Unix Shell. Git Bash is a build-in package of Git for Windows. Or, in some modern Windows environments is possible to use Windows Subsystem for Linux (WSL) to reproduce a Linux system inside Windows. After installing Git Bash or WSL you can follow the below steps to reach the same result that a Linux/Mac user.

Hands On

Okay, so now we know a little more about SSH keys and why use it let's start to configure our SSH to management two or more Github account.

We will reproduce the most common scenario: When you have a personal GitHub account for your personal projects and a company/work account.

I will suppose that you have the basic knowledge of terminal commands such as navigation between directories, create and edit files, etc.

ssh directory

The ssh directory is usually located in the main directory of
user: /home/yourUser

Is a occult directory, so, if you want list the directory, run the ls command with the -a flag to list all files and directories including the occult ones.

Open your terminal and type the command below to access ssh directory.

cd ~/.ssh

Create SSH Keys

Now, we will create our first SSH key for our personal account. In terminal run the following command:

ssh-keygen -t rsa -C "your-personal@email" -f "github-personal"

Replace "your-personal@email" by your personal e-mail account. You can replace the "github-personal" too, but the name you put here will be used later, so take this in mind in the next steps.

Repeat the same command before, changing the "your-personal@email" to your work e-mail account, to create a new SSH key to your professional account.

ssh-keygen -t rsa -C "your-work@email" -f "github-work"

Add the SSH Keys to your SSH-agent

After create our two SSH keys we will add this keys to the SSH-agent. Type in terminal the following commands to add our personal and work accounts.

ssh-add ~/.ssh/github-personal

ssh-add ~/.ssh/github-work

As said before, the above commands will be slightly differents if you set another value to the "github-personal" or "github-work" in the creation command.

To test if the SSH keys were correctly added to the SSH agent, run the below command to list your active keys by fingerprint:

ssh-add -l

The output should be similar to this:

3072 SHA256:a lot of random characters your-personal@email (RSA)

3072 SHA256:a lot of random characters your-work@email (RSA)

Add to Github account

Here we have two options.

1) We can set our two SSH keys to our personal account, in this way, you can access your personal account by the two SSH keys (personal and work keys). The same is valid if you set the two keys to the work account.

2) Or, we can set just the personal key to personal account and the work key to the work account, in this way, your personal key will not have access to the your work account and vice versa.

I personally, prefer to keep the things separated. So I will follow the second option.

The next step will be copy your public key and link to your Github account. In Mac, type the below command to copy the public ssh key.

pbcopy < ~/.ssh/github-work.pub

pbcopy < ~/.ssh/github-personal.pub

In Linux, you need use some tool like xclip or just open the created files with any text editor or cat command and copy all the content of files.

The content of this files should be similar to this:

ssh-rsa [...long list of random characters]= your-work@email

Now, you can go to Github webpage, signup in your account and access settings in profile menu.

In settings page, go to SSH and GPG Keys in navigation menu.

Here, will be listed all SSH keys that you already have associated to your account.

Click in the "New SSH key" green button and you will be redirect to the page below.

In this page we will paste in the Key field the SSH public key that we copy in the last step. Set a Title to this key and click in the green button "Add SSH key". Back in the previous page, the new created SSH key should be listed.

Create Config file

In terminal, create a new file called config.

touch ~/.ssh/config

In Mac, type this command to open the config file created:

open -e ~/.ssh/config

In Linux, you can use vim, nano or other text editor that you like.

nano config

Inside config file paste the following:

# Personal GitHub account
Host github.com-personal
 HostName github.com
 User git
 IdentityFile ~/.ssh/github-personal

# Work GitHub account
Host github.com-work
 HostName github.com
 User git
 IdentityFile ~/.ssh/github-work

Save the file and exit.

Make default account global

Set one of the accounts as default. In terminal, type:

git config --global user.name "personal"

git config --global user.email "your-personal@email"

Clone repository with the secondary account

I will clone a repository from my professional account.

Go to your Github profile, open a repository, click "Code" button, select SSH tab and copy the SSH link. (Image below)

Here we got a point of attention: the default SSH link provided by Github has this format:

git@github.com:repo-owner-user-name/the-repo-name.git

What we will do is a little change in this format adding the Host as we defined in the config file of our SSH, the SSH link will be similar to this:

git@github.com-work:repo-owner-user-name/the-repo-name.git

So, to clone the repository the complete command will be:

git clone git@github.com-work:repo-owner-user-name/the-repo-name.git my-repo

"my-repo" is just the name of the directory where the repository will be clone inside, you can put the name you want, or putting nothing to clone the repository inside the directory where you are.

Go to my-repo directory, type:

cd my-repo

If you run: git config user.name in terminal, you will receive as answer the default global value configured, in my case, was the personal. The same is valid to git config user.email.

So, for this repository that was cloned from my work account I want to use my professional credentials. To do this, just type in terminal, inside the directory of the repository:

git config user.name "work"
git config user.email "your-work@email"

This commands will be set your professional credentials to this specific repository from work and you will be able to perform git actions to this repository using your work account.

If you set your global credentials to the personal account and clone a personal repository, you don't need specify the user.name and user.email to this repository, because the default credential is your personal account and the git actions that you perform in this repository will be associated to your personal account. The same is valid if you set your work account as global and clone a work repository.

Concluding

We have seen how to create, configure and manage multiple Github accounts on one machine using SSH keys, this is very useful when we need different accounts for distinct purposes.

Reviews, suggestions and/or contributions to this article are encouraged and welcome.

References

What is SSH?

SSH Keys Concepts

Git SSH Keys

Start and Setup a NodeJs Typescript Project

Guilherme Niches — Mon, 06 Feb 2023 12:18:49 +0000

Start a new project usually involves a serie of commands and not always we remember all this commands and the main dependencies that we normally use.

This article show us one of many ways to start and setup a new NodeJs Typescript project.

Before this, some fast definitions of the tools we will use:

NodeJs

"NodeJs is an open-source, cross-platform JavaScript runtime environment."

In other words, NodeJs allow us use Javascript in the server-side, without the need of a web browser.

Typescript

Typescript is a superset of JavaScript, this basically means that the Typescript adds static typing on top of JavaScript.

Note: This article assumes that you already have NodeJS and some package manager installed on your machine. If you doesn't, check this links first:

NodeJs/NPM

Yarn

Commands to start a NodeJS TS Project

TS/Node dependencies

yarn add typescript ts-node ts-node-dev @types/node

npm i typescript ts-node ts-node-dev @types/node

tsconfig.json

Create tsconfig.json (Visit https://aka.ms/tsconfig.json to read more about this file) file in the root directory of your project with the following structure:

{
  "compilerOptions": {
    /*Basic Options*/
    "target": "es2018",   /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019', 'ES2020', 'ES2021', or 'ESNEXT'. */
    "module": "commonjs", /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', 'es2020', or 'ESNext'. */   
    "allowJs": true,      /* Allow javascript files to be compiled. */    
    "outDir": "./dist",   /* Redirect output structure to the directory. */
    "rootDir": "./src",   /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */  

    /* Strict Type-Checking Options */
    "strict": true,       /* Enable all strict type-checking options. */    
    "esModuleInterop": true,   /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */    

    /* Advanced Options */
    "skipLibCheck": true,  /* Skip type checking of declaration files. */
    "forceConsistentCasingInFileNames": true   /* Disallow inconsistently-cased references to the same file. */
  }
}

According to this configuration you should write your code in src/ folder and should have an entry point file called server.ts inside.

Our project structure should be similar to this:

.
├── src
     ├── server.ts

Inside the file server.ts we could put a simple log just to got a return when we run the server.

console.log("Server is Up");

package.json

Add dev, dev:debug and build scripts to your package.json file,

Dev: Uses ts-node-dev to start a dev server that automatically transpiles your .ts files to .js whenever you edit something. (More info at https://www.npmjs.com/package/ts-node-dev)
Build: Uses tsc to transpile your whole application to .js, is used when you want to deploy to production. (More info at https://www.typescriptlang.org/docs/handbook/compiler-options.html)
Dev:debug: Uses ts-node-dev --inspect to start a debug function at port defined after '=' in inspect flag.

"scripts": {
    "dev": "ts-node-dev --respawn src/server.ts",
    "dev:debug": "ts-node-dev --inspect=0.0.0.0:9229 src/server.ts",
    "build": "tsc",
    "start": "node dist/server.js"
  }

Run

npm run dev

yarn dev

Dev:debug

npm run dev:debug

yarn dev:debug

Build

npm run build

yarn build

Optional

These steps above are enough to start and run a NodeJS Typescript project, it's a 'clean' install. But, normally, we want start a project with some dev dependencies.

The following steps are optional and will install some dependencies commonly used with NodeJS.

Express => NodeJs framework to handling with routes (http://expressjs.com/)

npm i express --save

yarn add express -D

Sequelize or TypeORM => ORMs (https://sequelize.org/) || (https://typeorm.io/#/)

npm i sequelize --save

npm i typeorm --save

yarn add sequelize -D

yarn add typeorm -D

Some other dependencies often used with NodeJS/Typescript project:

Eslint => Identify code patterns in disagreement with pre-established rules (https://eslint.org/)
PostgreSql or MongoDB => Relational Database and NoSql Database (MongoDb)

Extra

NPM Package

If you are writing a npm package specify main and types path and add --declaration to your build script in order to generate types suggestions to who is using your package.

package.json

"main" : "dist/index.js",
"types" : "dist/index.d.ts",
"scripts": {
    "build": "tsc --declaration"
  }

Concluding

This is one of many ways to start and setup a new NodeJs Typescript project, you can do the same in different ways even faster. But, I like to do in this way because is more "step-by-step" form.

Reviews, suggestions and/or contributions to this article are encouraged and welcome.

References

NodeJS
Typescript

Utility Types, in Typescript.

Guilherme Niches — Tue, 20 Dec 2022 00:17:32 +0000

In our day routine as software developer we usually needs to manipulate data and types for multiple purposes: to update a entry in database, to mapping a DTO (Data Transfer Object), to structure our return to the client and much more.

In Typescript, we have some resources that help us to handling with this Object Type Transformations called Utility Types.

Okay, but what's Object Type Transformations?

"Object Type Transformations refers to generating a modified type from an existing type or interface that represents an object." (Source: Refine.dev)

Cool, now we know about this Typescript resources and about Object Type Transformations. So, let's see some examples?

Note: Some utility Types described here are available only from Typescript 3.5 or upper.

Utility Types of Typescript

Typescript has a great diversity of types that can help with some common type manipulation, in this article we will see some of them.

Partial

Changes all the properties in an object to be optional.

Syntax definition:

type Partial<T> = {[A in keyof T]?: T[A]};

Right, but what does the above example mean?

We know that T is the type passed to Partial, in the below example is IUser.

interface IUser {
  uid: string;
  name: string;
  age: number;
};

So, keyof T in our example will be keyof IUser, in other words, keyof IUser it's equivalent of the keys of our object (in our example: "uid" | "name" | "age").

[A in "uid" | "name" | "age"]?

Hence, A must be any of the property keys of the IUser type.

['uid']? = IUser['uid']
['name']? = IUser['name']
['age']? = IUser['age']

All keys with the question mark (?) are to show that the property key is optional. Then, it's like the Partial generate other IUser interface (called PartialIUser in our example below) with all optional keys.

interface PartialIUser {
  uid?: string;
  name?: string;
  age?: number;
};

Usecase Example

In the below example, the function updateUser would accept any user fields, ie, the User could be partially updated. And we can now use this function to update each field individually without needing the other ones to be set.

Omit

Removes keys from an object type.

Syntax definition:

type Omit<T, K extends keyof any> = Pick<T, Exclude<keyof T, K>>;

Maybe you noted, the Omit type is based on using other two utility types: Pick and Exclude (we are talk about Pick later).

Omit definition is equal to:

 Pick<T, Exclude<keyof T, K>>

Using our IUser interface from previous example, again, T is our IUser and K is a string literal of keys that are removed from T, ie, our IUser.

Let's see a example.

Usecase Example

In the following example, we use Omit to remove the key "age" from our IUser interface and set the new type IUserWithoutAge to a new variable. In this variable, the property "age" is not available and can't be used.

Note that it's possible omit multiple keys from an object, passing a union of string literals in K:

type IUserWithoutAgeAndName = Omit<IUser, "age" | "name">;

Pick

"Keeps" only the specified keys.

Syntax definition:

type Pick<T, K extends keyof T> = {
    [P in K]: T[P];
};

Pick will generate a new object. So, it's not possible to have a primitive type (string or number, for example) as the new type.

Breaking down the syntax definition and using our IUser interface as example we have the following:

T[P] is a way to access the type of a property. So, if we want access the "age" property from our IUser interface we can do like this: IUser['age'].

Again, keyof T are the keys of our type, in this example IUser interface and the keys "uid", "name" and "age".

And we have [P in K] which means that property P needs be one of the keys extracted from K. Then, P could be either "uid", "name" or "age".

Usecase Example

In the example above, we picked the key "age", so, the variable that has the type IUserWithOnlyAge has only the key "age" to be setted, the others properties "uid" and "name" hasn't available.

Readonly

Turn all properties readonly.

Syntax definition:

type Readonly<T> = {
    readonly [P in keyof T]: T[P];
};

Understanding the syntax definition:

Like the previous examples, T[P] is a way to access the type of a property and keyof T are the keys of our type.
The readonly prefix is a keyword of typescript that makes a property as read-only in the class, type or interface.

So, basically, for each property of our type we are turning this property readonly.

Usecase Example

The above example show us that try reassign the variable newUser with has IUserReadonly type will cause a compilation error.

Concluding

In this article we just scratch the surface of utility types of Typescript. Has a lot more utility types that can be very useful for different usecases and you can discovery it in Typescript documentation.

Utility types help us to handling with transform and manipulate data and can improve our code turning it more clean, should be used wisely.

Let your comment if you already used utility types.

Reviews, suggestions and/or contributions to this article are encouraged and welcome.

References

Typescript Documentation

Types Transformations With Typescript

A Detailed Guide on Typescript Guide Pick Type

Function with dynamically typed parameters, in Typescript.

Guilherme Niches — Tue, 08 Nov 2022 14:52:00 +0000

In Typescript, how to create a function that returns the type that itself receives?

If we don't explicitly set the parameter the function returns an 'any' type.
If we set it, the function returns the fixed type in the parameter.

So, how to handle it? Let's see.

In the below example, we declare a function that receives a parameter and returns the parameter itself. This parameter was not explicitly set, so when we call the function, the returned value has an 'any' type.

But, wait... We passed a number as a parameter to our function and this function just returns this parameter, so, the value should be a number, right? Well, unfortunately, the Typescript "doesn't understand" it in this way.

To handle with this, we can use Generics of Typescript.

Generics in Typescript

Okay, so we will use Generics, but what is Generics?

The Typescript documentation explains Generics as:

"(...) being able to create a component that can work on multiple types rather than a single one."

Refactoring the code, our function now received the type variable Type, a kind of variable that works on types rather than values. This Type allow us to capture the type that the user inform. And in this example, we declare this Type as the type of the return of the function.

So, if we pass a number to our function it will return a number, if we pass a string it will return a string, if we pass an object with some attributes (like as example above), it will return the object with the typed values. The same goes for array and other types.

Concluding

In this way, our function will apply types regardless of what we pass as a parameter. This allows us to create functions that can be used in different contexts, turning the function more reusable.

It is worth mentioning that, despite this feature of Typescript, it is always interesting to try to be declarative in your typing. So, this is a feature to use with prudence.