Forem: Shamil

Processing Text with Linux Shell - Part 2

Shamil — Sun, 29 Jul 2018 19:39:20 +0000

`grep` - searching the haystack made easy

When you need to search a piece of text from a very large source, grep command is the answer. grep accepts strings, regular expressions and it can produce output in various formats.

The most basic syntax to search a piece of text in a file is:

grep "pattern" file_name

If we need to search multiple files for a pattern, we can do the following:

grep "pattern" file1 file2 file3 ...

grep can also be used with standard input.

echo "linux is awesome" | grep linux

If no file or directory is specified, grep reads the STDIN for input.

# Highlight the matched text

One major feature of grep is it's ability to highlight the matched string in its output. There are 3 color options: auto, always and never. To use the color options, we can pass an additional --color flag to instruct which option to use.

grep "pattern" file_name --color=always

As we can already guess, with --color=always option it will always color the output matched string. With --color=auto, grep displays color in the output only if the output is not piped to a command or redirected to any file. The last option, --color=never will turn off the coloring.

If you want to turn on coloring for every grep operation, but feeling lazy to type --color=always with each command (which would be me), you could add GREP_OPTIONS to your environment.

export GREP_OPTIONS='--color=always'

This will turn on coloring for all grep commands.

# Limit what to output

Normally, grep only outputs the line that contains the matched pattern. But there are circumstances where we might want to see the contents of the entire file with the matched pattern highlighted. We can do the following for this:

grep --color 'pattern\|' file_name

This will output all the lines containing the pattern as well as the lines that have an end, i.e the complete contents of the file with only the matched part colored. Of course, we can pass multile patterns to be matched.

grep --color 'pattern_one\|pattern_two\|' file_name

Notice the \ before | . This is because grep only interprets some of the special characters. However we can also write commands without \ .

grep -E --color 'pattern_one|pattern_two|' file_name

The -E option instructs grep to use full set of regular expressions. However, there is a better way of writing such patterns containing regular expressions . We can use egrep, which is an extended version of grep that supports extended regular expressions out of the box. Therefore, we can rewrite the above commands like this.

egrep --color 'pattern_one|pattern_two|' file_name

But what if we neither need the complete line that contains the matching text nor the complete contents, rather just the matching content?

For example, if we are searching for all the email addresses in a file, we do not need the line that contains the email, rather we just need all the emails that appears in the file.

Well, we can use the -o option, which stands for only matching, that limits the output to only the matched text.

egrep -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" file_name

Neat, isn't it?

What if we need to see all the lines except the ones where the given pattern appears?

grep -v "pattern" file_name

The -v option inverts the results and thus the above command will output all the contents except the lines containing the matched pattern.

# Count the appearances

If we need to get a count of how many times a pattern appears in a file, we can use the -c flag. From the above email pattern example, if we need to see how many email addresses are there in a file, we could do the following.

egrep -c "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" file_name

But wait, there is a gotcha.

The -c counts only the number of lines that contains the pattern, not the actual number of matched pattern. So if there are multiple matches in a single line, it will count only once!

We can overcome this with a little tweak.

Remember how we printed only the matching portions with the -o flag above. If pipe wc -l with it,

egrep -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" file_name | wc -l

it will give us the exact number of times a particular pattern appears in a file.

# Extract additional information

If we want to print the line numbers where the searched pattern appears, we can use the -n flag.

grep -n --color "pattern" file_name

The -n flag will also print the name of the file, in case we are searching in multiple files.

If we want to see the offset where the pattern starts, we can use the -b flag.

echo "linux is the answer" | grep -b -o "the"

This is print 9:the as the searched word "the" starts at the 9th position of the line. Note that that -b flag is always used with the -o flag.

# Search within directories

To see which files contain a particular pattern , we can use the -l flag.

grep -l "pattern" file1 file2 file3 ...

This will output only the file names contains the given pattern. If we want to see the name of the files that does not contain the files, we can use -L flag (note the case-sensitive flags). This basically inverts results we get with -l flag.

Well, so far we have only seen how to search in a single file or a set of file which are specified in the command. However, we can also search all the files in a complete directory and it's subdirectories recursively.

grep -r "pattern" .

Here . specifies the current directory. The -r flag implies that the search should be done recursively in all subdirectories. This search is case-sensitive.

If we want to search all occurances of the pattern, we can append -i flag.

grep -ir "pattern" .

This will output all matching patterns irrespective of the case.

# The pattern file

If we have too many patterns, strings etc that we want to search, piping them with | is very tedious. In such scenarios, we can use a pattern file, that contains all the strings/patterns we want to search, each in a new line. Then we can use that pattern file in the grep command.

grep -ir . -f pattern_file

This will print all the matching strings from the pattern file.

ヽ(´▽`)/

(You can find Part 1 of this series here )

Processing Text with Linux Shell - Part 1

Shamil — Fri, 27 Jul 2018 20:50:46 +0000

Into the world of `sed`

If you are using any *nix systems on a daily basis, chances are you are already familiar with, or at least you have heard about the sed command.

sed , short for Stream Editor, is a text transformation tool that comes bundled with every unix system. What makes sed distinguishable from other text editors is the speed at which the text manipulation is performed. sed only makes one pass over the input text, therefore making the processing quite faster.

# Replace those ugly text

sed is a very powerful tool to replace a piece of text with another. The text can be matched using regular expressions.

sed 's/text_to_be_replaced/replacement_text/' file_name

However, this will only print the substitued text in the console, but won't change the same in the file itself. If we want to save the changes to the file, we can use the -i flag.

sed -i 's/text_to_be_replaced/replacement_text/' file_name

This above replaces only the first occurance of the given pattern in each line. So if we want to replace every occurence of the pattern, we can append the g parameter to the end.

sed 's/text_to_be_replaced/replacement_text/g' file_name

Note that the delimiter character / we used in the above commands is not fixed, we can use almost any delimiter character in sed. For example,

sed 's:text_to_be_replaced:replacement_text:g' file_name


sed 's|text_to_be_replaced|replacement_text|g' file_name

Okay, but what if the delimiter character is itself a part of the pattern to be replaced? ¿ⓧ_ⓧﮌ

Well, we can escape that character with a backslash. For example, to replace the word following: with below - , we can do this:

sed 's:following\::below - :' file_name

Notice the use of \: before the delimiter : that separates the pattern and it's replacement.

# Delete that scrap

sed also allows us to delete lines from a file. The d option is used to indicate a delete operation. The generic syntax to delete line is

sed 'Nd' file_name

Here N is the line number that we want to delete. If we want to delete the 10th line from a file, N would be 10.

One most common use of this command is deleting all blank lines in a file.

sed '/^$/d' file_name

The above will delete all the blank lines in the file. The regular expression ^$ marks an empty line and the d option specifies that the line should be deleted.

That's not it. We can also specify a range of lines that should be deleted.

sed 'm,nd' file_name

The above command will delete all the lines starting from mth upto nth.

# Pipelining is important

Now what about pipelining multiple sed commands?

We can pipeline as many sed as we wish and they would be processed in that order. Consider the following example.

echo Linux | sed 's/L/l/' | sed 's/n/N/' | sed 's/l/L/' | sed 's/x/X/'

This will output LiNuX.

Finally let's take a look at how we can use variables within sed command. So far we have used ' ' (single quote) in our commands. However we can aslo use " " when we need to use an expression in our command. Take a look at the following example.

greet=hello

echo hello shamil | sed "s/$greet/hi" file_name

This will replace evaluate the value of $greet and and replace hello with hi.

# Better safe than sorry

When using -i in the sed command, we need to be careful, as it replaces the actual content in the file. (Trust me, I have done this many times)

Therefore, it is a good practice to first use this command without -i flag and check if the replacements are correct. However, if the file contents are too long to be checked like that, you can use the following command to create a backup copy of the same and then modifying the content.

sed -i.bak '12,30d' file_name

This will delete all lines from 12 to 30, but most importantly it will create a file_name.bak in the same directory before modifying the actual file.

Who knows, this might just end up saving your job (◠﹏◠)

(EDIT: See this comment for more info on -i usages)