Forem: t-o-d

Split CSV by number in shell only.

t-o-d — Wed, 27 Jan 2021 08:13:53 +0000

Sometimes I have to deal with large CSV files.
In this case, dividing and foldering in advance will make it easier to handle.
This section describes how to split files and classify folders according to the number of lines using only Shell.

Result

The following is the directory structure before execution.

.
├── sample.csv
├── main.sh

The following is the description of main.sh
- sample.csv has 100 rows of data
- ※Error handling is omitted.

#!/bin/sh
set -e

# file path
[ ! -e "$1" ] && exit 1 || datafile="$1"
# File extension deletion
filename="${datafile%.*}"
# Get number of lines
row=$(grep -c '' $datafile)
# Obtaining the number of splits
sep="$2"
# Number of directories created
dir_cnt=$(awk -v row="$row" -v sep="$sep" 'BEGIN {
    i=row/sep
    printf("%d\n",i+=i<0?0:0.999)
    }
    '
)
# Folder creation
seq -f "${filename}_%01.0f" 1 ${dir_cnt} |
xargs mkdir -p
# File division
split -l ${sep} -a 2 $datafile "${filename}_"
# File movement
count=1
for i in `find . -type f -name "${filename}_*" | sort`
do
    mv $i "${filename}_${count}/${i//_*/_${count}}.csv"
    let count++
done

Run as follows.

sh main.sh sample.csv 25

After executing, check that the directory structure is as follows.

.
├── main.sh
├── sample.csv
├── sample_1
│   ├── sample_1.csv
├── sample_2
│   ├── sample_2.csv
├── sample_3
│   ├── sample3.csv
├── sample_4
│   ├── sample4.csv

Supplement

Number of created directories

Rounding up
- In the case of a decimal number such as 100/15, the directory is not created normally.
- Round up to an integer with printf.

File splitting and moving

Extension is added by mv.
- Additional extension (--additional-suffix) in split is not the default on Mac etc.

Link

Prepare a set of Rust tools that can be used with zsh in zinit

t-o-d — Tue, 19 Jan 2021 02:01:22 +0000

I was using oh-my-zsh as a plugin manager to extend zsh, which has become the standard shell for Macs, but changed to a lightweight tool called zinit in terms of operation.
At the same time, it turned out that I could use the Rust command tools that I usually use (ripgrep and exa).
In this article, we will describe how to set up zinit and configure the tools.

Environment

Mac OS X 10.15.6
ZSH 5.7.1

Procedure

Install zinit

Install zinit (a lightweight and fast zsh plugin management tool) in your environment with the following command.

sh -c "$(curl -fsSL https://raw.githubusercontent.com/zdharma/zinit/master/doc/install.sh)"

After entering the command, the following information will be automatically entered in ~/.zshrc.

### Added by Zinit's installer

# ...............

source "$HOME/.zinit/bin/zinit.zsh"
autoload -Uz _zinit
### End of Zinit's installer chunk

Update manually with the following command.

zinit self-update

Selecting a tool

This time, the Rust command tools managed by zinit are as follows.
- ripgrep
  - Fast grep alternative tool
- exa
  - Fast ls alternative tool
- bat
  - Fast cat alternative tool
- fd
  - Fast find alternative tool

Setting up zshrc

After completing the tool selection, describe the following plug-in contents in .zshrc

### Added by Zinit's installer
# ....
### End of Zinit's installer chunk

# ripgrep
zinit ice as"program" from"gh-r" mv"ripgrep* -> rg" pick"rg/rg"
zinit light BurntSushi/ripgrep

# exa
zinit ice as"program" from"gh-r" mv"exa* -> exa"
zinit light ogham/exa

# bat
zinit ice as"program" from"gh-r" mv"bat* -> bat" pick"bat/bat"
zinit light sharkdp/bat

# fd
zinit ice as"program" from"gh-r" mv"fd* -> fd" pick"fd/fd"
zinit light sharkdp/fd

After saving, load it with the following command

source ~/.zshrc

The tool will start downloading, and after it finishes, enter the various commands.
- ripgrep : rg word
- bat : bat filename
Once confirmed, you are done.

Link

【Mac】Regular Expressions for exact matching as words with grep and sed

t-o-d — Mon, 18 Jan 2021 11:40:27 +0000

When we use grep or sed in our business, we often use word matching to search and replace.
In this case, it is not uncommon that the exact match is not found.
In this article, we will record the regular expression notation for doing so with a high probability of an exact match.

Result

A description of the results is given first.
- ※It is designed to work with BSD-style grep and sed on Mac.

# Data Example
$ cat data

10,30,50,100,150,250,300,500

# grep only 50 of the data examples
$ cat data | grep -o '[[:<:]]50[[:>:]]'

50

# Replace only 50 of the example data with "match".
$ cat data.txt | sed 's;[[:<:]]50[[:>:]];match;g'

10,30,match,100,150,250,300,500

Description

The above example shows the process of search and replace for arbitrary data.
The method used here is to enclose the word like [[:<:]]word[[:>:]].

grep

In the first place, grep extracts not only words but also lines.
To extract only words, use option and use grep -o word.
However, in the above data example, there is a partial match as shown below, and four are extracted.

$ cat data | grep -o 50

50
50
50
50

From these things, we use the [[:<:]]word[[:>:]] which has the following meanings.
- Since it matches the beginning and end of a word, it does not recognize the character strings before and after.

sed

The same is true for sed. If you replace it with the string as it is, it will be replaced 4 times.

$ cat data.txt | sed 's;50;match;g'

10,30,match,100,1match,2match,300,match0

From these things, it is necessary to recognize it as a word like [[:<:]]word[[:>:]] like grep.

Conclusion

From the above, I was reminded of further study of regular expressions, conscious of data duplication.

Link

https://dev.mysql.com/doc/refman/5.6/en/regexp.html

Mac hardware and system information retrieval one-liner

t-o-d — Sat, 31 Oct 2020 12:07:47 +0000

When I use Mac, there are times when I want to get system information and so on by command.
Also, you may want to use that information in shellscript.
In this article, we will write commands to obtain hardware and system information from your Mac.

Result

Here are the results.
Enter each command based on the information you want to retrieve.

Model Name

# model name
# output : MacBook Pro**

info=$(system_profiler SPHardwareDataType | grep 'Model Identifier'); echo "${info#*: }"

Processor

# processor
# output : Dual-Core Intel **
info=$(system_profiler SPHardwareDataType | grep 'Processor Name'); echo "${info#*: }"

Core

# core
# output : 2
info=$(system_profiler SPHardwareDataType | grep 'Cores'); echo "${info#*: }"

Serial Number

# serial number
# output : *****
info=$(system_profiler SPHardwareDataType | grep 'Serial Number'); echo "${info#*: }"

UUID

# UUID
# output : *****-*****-****
info=$(system_profiler SPHardwareDataType | grep 'UUID'); echo "${info#*: }"

Memory

# memory information
# output : ** GB
info=$(system_profiler SPHardwareDataType | grep 'Memory'); echo "${info#*: }"

User

# user information
# output : **** *** (*******)
info=$(system_profiler SPSoftwareDataType | grep 'User Name'); echo "${info#*: }"

Supplement

For ease of use, the fetched content only outputs the results.
Use shall be in the following format.
- info=$(system_profiler data-type-name | grep 'field-name'); echo "${info#*: }"
The above data-type-name and field-names are confirmed by the following command.
- data-type-name : system_profiler -listDataTypes
- field-name : system_profiler data-type-name
Other information that can be obtained with system_profiler can be found here.

Link

https://ss64.com/osx/system_profiler.html

Tips for writing npm-scripts in npm-run-all for different purposes.

t-o-d — Wed, 07 Oct 2020 00:07:04 +0000

The npm-scripts are used in package.json for task processing in web development.
We also use npm-run-all, which is useful for serialization and parallelization of multiple processes.
Therefore, we'll use it to illustrate a small technique to make npm-scripts flexible and clear.

Result

A description of the results is given first.
The following package.json is a task to mainly build and watch.

{
  "scripts": {
    "build"      : "run-s build:{sass,ts}",
    "build:sass" : "sass input.scss output.css",
    "build:ts"   : "tsc main.ts",
    "watch"      : "run-s watch:{sass,ts}",
    "watch:sass" : "sass --watch input.scss output.css",
    "watch:ts"   : "tsc -w main.ts"
  }
}

Description

The method used here is to specify what to do with the process in braces.

Specification using braces

There are three ways to execute a task classified by group in a colon (:) in a single session.
- build-basic
- build-glob
- build-brace

{
  "scripts": {
    "build-basic" : "run-s build:sass build:ts",
    "build-glob"  : "run-s build:*",
    "build-brace" : "run-s build:{sass,ts}",
    "build:sass"  : "sass input.scss output.css",
    "build:ts"    : "tsc main.ts",
  }
}

From these I use the method specified in braces, such as build-brace.
The benefits are as follows.
- unduplicated
  - There is no duplication of tasks like run-s build:sass build:ts, and if the number of tasks increases in the future, you only need to describe the purpose of the task.
- clarification
  - Compared to an asterisk specification such as run-s build:*, the explicit description makes the process easier to understand.
- ordering
  - The processes are performed in the order of the braces, so even if you reverse build:sass and build:ts in package.json, there is no problem.
    - If you use run-s build:*, the order of processing is the same as the order in package.json, which may cause a problem if you use it in reverse.

Conclusion

The above reminded us of the importance of readability, clarity, and non-overlap in the task as well.

Link

【Vim】Setting to automatically remove white space at the end of a line when starting and saving

t-o-d — Mon, 05 Oct 2020 00:27:20 +0000

Introduction

The following points bothered me when editing in vim and I investigated whether they could be resolved.
- I have to remove it manually or by command every time because "habit or unintentional" whitespace is put at the end of a line.
- Sometimes I forget to delete them.
- In order to prevent deletion, I want to perform automatic deletion not only at the time of saving, but also at the time of starting.
So, we'll use the autocmd function to write an automatic deletion setting.

Result

Open .vimrc (configuration file)

vim ~/.vimrc

Write the following to .vimrc

" auto remove blank
augroup vimrc
autocmd BufRead,BufWrite * if ! &bin | silent! %s/\s\+$//ge | endif
augroup END

Loading settings in the vim

:source ~/.vimrc

As a test, enter a blank space at the end of a line in the file, and if it is automatically deleted when the file is started or saved, it's done.

Link

autocmd

Three main useful techniques to check input forms using only CSS

t-o-d — Fri, 02 Oct 2020 12:46:56 +0000

Technique1 : Blank check for input form

Using the 「placeholder-shown」 attribute

Result

Details

The attribute used in the CSS part of this input form is placeholder-shown.
This indicates when a placeholder is displayed.
In other words, by using 「not」 in the following specification, we can make the button clickable by increasing the transparency of the button when the placeholder is not displayed.

<input placeholder="Please enter a value.">
<button>save</button>

/*
Use opacity to reduce the initial transparency by half.
By using pointer-events, the initial button click is disabled.
*/
button {
    opacity: 0.5;
    pointer-events: none;
}

/*
Using not(:placeholder-shown), you can use
If you don't see the placeholder, you can use
Automatic setting of pointer-events.
Restore the opacity's transparency as well.
*/
input:not(:placeholder-shown) + button {
    opacity: 1;
    pointer-events: auto;
}

Technique2 : Check the format of the input form

Using the 「valid and invalid」 attribute

Result

Details

In the CSS part of this input form, I'm using invalid and valid as attributes.
This indicates if the form matches the type specified in the input tag's type attribute (valid) or not (invalid).
In other words, the following specification will change the background color depending on whether the input value matches the format.

<input type="email" placeholder="Enter email address">
<button>save</button>

/*
Using the invalid attribute, the
The background color if the type attribute format does apply.
*/
input:invalid {
  background-color: #ee6666;
}

/*
Using the valid attribute, the
The background color if the type attribute format does not apply.
*/
input:valid {
  background-color: #95f195;
}

Technique3 : Check the scope of the input form

Using the 「in-range and out-of-range」 attribute

Result

Details

The CSS parts of this input form use 「in-range」 and 「out-of-range」 attributes.
This indicates if the range of values specified in the input tag's min and max attributes is met (in-range) or not (out-of-range).
In other words, the following specification will mean that the background color will be changed depending on whether the input value is beyond the specified numerical range.

<input type="number" min="1" max="20">
<button>save</button>

/*
Using the in-range attribute, the
The background color if it falls within a range of values.
*/
input:in-range {
  background-color: #95f195;
}
/*
Using the out-of-range attribute, you can use the
Background color if the value does not fall within a range of values.
*/
input:out-of-range {
  background-color: #ee6666;
}

Link

css selectors

【Vim】Setting to make renaming a file during editing easier with a shortcut

t-o-d — Wed, 30 Sep 2020 07:25:35 +0000

Introduction

When I want to do file renaming even when editing vim, I was wondering if it is possible because of the following points.
- I don't want to use a long command, but a shortcut using key bindings.
- If I forget to save a command like :w, I want to save it automatically.
- I want to perform a process interactively.
So, I'll write a simple setting for renaming with the Leader.

Result

Open .vimrc (configuration file)

vim ~/.vimrc

Write the following to .vimrc.

" setting leader
" Use 「Space + other keys」.
let mapleader = "\<Space>"

" Calling a renaming function
" Enables the renaming process in the vim with 「space + n」.
map <leader>n :call RenameCurrentFile()<cr>

" Define the renaming function
function! RenameCurrentFile()
  let old = expand('%')
  let new = input('NewFileName: ', old , 'file')
  if new != '' && new != old
    exec ':saveas ' . new
    exec ':silent !rm ' . old
    redraw!
  endif
endfunction

Loading settings in the vim

:source ~/.vimrc

As a test, open the file you want to change with vim, press "spacebar + n" to change it as follows, and if it's OK, it's done.

Link

document

【Bash】Creating a secure and versatile temp directory

t-o-d — Sat, 26 Sep 2020 12:55:28 +0000

Introduction

The code for making tmp directory, which is usually used in shell, has the following points to be improved.
- Because of the direct specification such as mkdir /tmp/path, the names will be duplicated in some cases.
- Because of the direct designation, it is not secure in terms of security.
- The names need to be unique and also add relevance so that they are easy to use and generic within the process.

Result.

First, we describe the following contents in the shell file we have prepared as a result.

#!/bin/bash

basepath=$(basename $0)
timestamp=$(date +%Y%m%d%H%M%S)
tmpd=$(mktemp -dt "$basepath.$timestamp.$$")/
echo $tmpd

# Outputs
# $TMPDIR/index.sh.20200919152709.XXXX.XXXXX/

After describing it, run it multiple times to check for creation and non-overlapping of names in the output path, and you're done.

Using the mktemp command

You can use the mktemp command to create a directory, it creates files and directories of size 0 automatically.
- For more information on how to use mktemp, please refer to Reference

Use of Template Options

When creating with mktemp, use the t(template) option, which is specific to this command.
By doing so, the directory will be created in the path set in the $TMPDIR automatically without specifying the directory name.
- ※To find the configured tmp directory, use echo $TMPDIR.
- ※If it is not set, it is specified directly by -p or stored in /tmp.

Add related items to the name

Add anything relevant to the name to make it easier to use when processing it in the program.
At a minimum, we have added the following.
- Execution path name
- Timestamp (time)

参考

Reference

npm scripts management using nps and nps-utils

t-o-d — Sun, 16 Aug 2020 00:13:24 +0000

When developing with npm, the amount and content of scripts in package.json may increase.
Also, it may not be multi-platform, such as using commands that are not available in some environments.
Therefore, I will describe how to manage scripts separately using two libraries, nps and nps-utils.

preparation

As an example, we are going to change the following package.json scripts.

{
  "scripts": {
    "dev": "NODE_ENV=development nodemon index.js",
    "test": "NODE_ENV=test jest --coverage",
    "lint": "eslint . --cache"
    "build": "NODE_ENV=production run-s build:{clean,js}",
    "build:clean": "rm -rf ./dist",
    "build:js": "esbuild src/index.js --bundle --outfile=dist/out.js"
  },
  "devDependencies": {
　　................,
    "esbuild": "^0.6.12",
    "nodemon": "^2.0.4",
    "jest": "^26.4.0",
    "eslint": "^7.7.0",
    "npm-run-all": "^4.1.5"
  }
}

This package.json has the following concerns
- The rm command and environment variable settings are limited to the execution environment.
- There is a lot of content and volume, and it will become more complex as it increases in the future.

setting

install

Install the following two libraries
- nps : npm scripts management tool
  - By keeping the details of scripts in a separate file, management and maintenance will be easier.
  - It is highly flexible and extensible because it can be written in js and yaml instead of json.
- nps-utils : Packaged tools to enhance nps
  - A number of multi-platform libraries are built into the system, making it complete in one.

npm install --save-dev nps nps-utils

init

Create a configuration file with the following command.
- ※This time, we'll create it in js.

# The default is package-scripts.js
./node_modules/.bin/nps init

# If you create with yaml.
./node_modules/.bin/nps init --type yml

After creation, the structure is as follows.

.
├── package.json
└── package-scripts.js
└── .......

Example of a configuration template

First, make the following contents of package-scripts.jscreated in the initialization process.

// Multiplatform support(Mac,Windows,Linux)
const {
  series,  // Simplification of continuous execution
  rimraf,  // rm command
  crossEnv  // Environment variable setting
} = require('nps-utils');

module.exports = {
  scripts: {
    dev: {
      default: crossEnv('NODE_ENV=development nodemon index.js')
    },
    test: {
      default: crossEnv('NODE_ENV=test jest --coverage')
    },
    lint: {
      default: 'eslint . --cache'
    },
    build: {
      default: crossEnv(
        `NODE_ENV=production ${series.nps(
          'build.clean',
          'build.js',
        )}`,
      ),
      clean: rimraf('dist'),
      js: 'esbuild src/index.ts --bundle --outfile=dist/out.js',
    }
  }
};

The content of the final package.json is as follows.

{
  "scripts": {
    "dev": "nps dev",
    "test": "nps test",
    "lint": "nps lint",
    "build": "nps build"
  },
  "devDependencies": {
　　................,
    "esbuild": "^0.6.12",
    "nodemon": "^2.0.4",
    "jest": "^26.4.0",
    "eslint": "^7.7.0",
    "npm-run-all": "^4.1.5",
    "nps": "^5.10.0",
    "nps-utils": "^1.7.0",
    ..........
  }
}

run

Each script is executed with the following command.

# npm run script name

# dev
npm run dev
# test
npm run test

Link

【NPM】Three useful ways to write a grouping in scripts configuration.

t-o-d — Mon, 10 Aug 2020 09:13:47 +0000

I think you need to write scripts configuration in package.json when you use npm. In this post, I'll write three useful ways to group scripts.

preparation

Install npm-run-all, which is useful when writing multiple commands in scripts.
- It can be written as run-s ******(scripts name).

$ npm i -D npm-run-all

1. Basic Writing

Write all of the scripts.

{
  "scripts": {
    "build": "run-s build:clean build:copy build:js",
    "build:clean": "rimraf ./dist",
    "build:copy": "cpx -C public/** dist",
    "build:js": "esbuild src/index.js --bundle --outfile=dist/out.js"
  }
}

2. wildcard writing

Write with a wildcard using an asterisk.

{
  "scripts": {
    "build": "run-s build:*",
    "build:clean": "rimraf ./dist",
    "build:copy": "cpx -C public/** dist",
    "build:js": "esbuild src/index.js --bundle --outfile=dist/out.js"
  }
}

3. grouping writing (Convenient and recommended)

Write it as you would an array using brackets.
It is recommended that the order and names are clearly marked.

{
  "scripts": {
    "build": "run-s build:{clean,copy,js}",
    "build:clean": "rimraf ./dist",
    "build:copy": "cpx -C public/** dist",
    "build:js": "esbuild src/index.js --bundle --outfile=dist/out.js"
  }
}

That's all.
Thank you for reading.