- Environment
- Command Substitution
- Process Substitution
- Variable Substitution
- Brace Expansion
- Parameter Expansion
- Positional Parameters
- Tilde Expansion
- Arithmetic Substitution
- Word Splitting
- Filename Expansion
- Conclusion
Environment
In order to test all of the bash shell expansions features we need to make sure we are running a recent version of bash. Below is the system information for this article. The tests in this article are running on Ubuntu 19.10 as shown below.
Linux instance-1 5.3.0-1014-gcp #15-Ubuntu SMP Tue Mar 3 04:14:57
UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
The bash version for these tests is bash version 5, which is quite recent. Older versions of bash are missing a bunch of features.
GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
Command Substitution
Command substitution allows the running of one or multiple commands and capturing outputs and actions from those commands and including them in another command all in one line or less lines than running all the commands separately. Command Substitution has two syntaxes; the more popular syntax is the backtick syntax where the command to be executed is enclosed in two backquotes, or backticks. The other syntax which is equally powerful encloses commands in $() syntax and the output can be used from that new expansion. Let’s look at a number of examples of Command Substitution below.
Simple command substitution using $() syntax to run the date command.
Wed Mar 18 01:42:46 UTC 2020
Simple command substitution using backtick syntax to run the date command.
Wed Mar 18 01:43:17 UTC 2020
Using the stdin operator at the beginning of command substitution syntax is a fancy way to read the text of a file into a variable and use it in a command on the shell as below.
$ echo $(< mytext)
hello world
Read a file into a variable to be used in a command using the cat command and Command Substitution.
$ echo $(cat mytext)
hello world
Same as above, read a file and use it in Command Substitution using backticks and cat command.
$ echo `cat mytext`
hello world
Combine embedded Command Substitution with another Command Substitution using both $() and backticks together
$ cat myfile
Wed
Embedded Command Substitution inside another one using two $() syntax operations
$ cat myfile
today is Wed
Use output from a command as arguments into another command, with the backtick syntax. We will get a list of files by running cat which contains one file per line and then pass that into the rm command which will remove each file
$ echo one > myfiles; echo two >> myfiles
$ rm `cat myfiles`
Same as above but with $() syntax, pass command output from cat into rm command to delete files.
$ echo one > myfiles; echo two >> myfiles
$ rm $(cat myfiles)
Store the output from a cat command into a variable and then loop through the variable so you can more clearly see what is happening.
$ echo one > myfiles; echo two >> myfiles
$ MYFILES=$(cat myfiles)
$ for f in $MYFILES; do echo $f; rm $f; done
one
two
Same as above but use backticks syntax to run the cat command and store the output in a variable and then loop through the files int the variable.
$ echo one > myfiles; echo two >> myfiles
$ MYFILES=`cat myfiles`
$ for f in $MYFILES; do echo $f; rm $f; done
one
two
Use the Command Substitution with stdin operator to read a file line by line into a variable and then loop through the variable afterwords
$ echo one > myfiles; echo two >> myfiles
$ MYFILES=$(< myfiles)
$ for f in $MYFILES; do echo $f; rm $f; done
one
two
Process Substitution
Process Substitution is a documented feature of bash; it’s quite cryptic in my opinion. In fact I have not found many good use cases to recommend for it. One example is included here for completeness where we use Process Substitution to get the output of a command and then use it another command. We will print the list of files in reverse order with sort command in this example after fetching files from the ls command.
$ sort -r <(ls *txt)
two.txt
three.txt
one.txt
Variable Substitution
Variable Substitution is what you can consider basic usage of variables and substituting the value of the variable when it is referenced. Its fairly intuitive, a few examples are provided below.
Simple variable assignment and usage where we put a string into variable X and then print it to stdout
$ echo $X
12345
Check if a variable is assigned something or null, in this case it’s assigned so we print it to stdout
$ if [ -z "$X" ]; then echo "X is null"; else echo $X; fi
12345
Check if a variable is assigned something or null, in this case it’s not set so we print “is null” instead of the value.
$ if [ -z "$X" ]; then echo "X is null"; else echo $X; fi
X is null
Brace Expansion
Brace Expansion is a super powerful feature of bash that allows you to write more compact scripts and commands. It has many different features and options described below. Within braces your syntax is interpreted into a more verbose syntax depending when you enter into the curly braces. Let’s look at a number of examples for Brace Expansion.
Each version of the items in the list in braces is executed. So we go from one echo command and print 3 versions of the word below separated by spaces.
a_warehouse m_warehouse p_warehouse
Expressions in the expansion causes execution multiple times. To prove this we use the date and sleep command to validate that the date command is run once for each iteration of the pattern in the Brace Expansion.
a_Sun Mar 22 18:56:45 UTC 2020 m_Sun Mar 22 18:56:46 UTC
2020 p_Sun Mar 22 18:56:47 UTC 2020
Expansions using numbers with .. will cause sequential numbers to be expanded in a numerical sequence
1_warehouse 2_warehouse 3_warehouse 4_warehouse 5_warehouse 6_warehouse 7_warehouse
8_warehouse
Reverse order brace expansion with sequence of numbers
8_warehouse 7_warehouse 6_warehouse 5_warehouse 4_warehouse 3_warehouse 2_warehouse
1_warehouse
Using an optional increment value to specify the numerical increments of brace expansion
1_warehouse 4_warehouse 7_warehouse
Lexicographical brace expansion will iterate through letters in the alphabet in the order of the locale
a_warehouse b_warehouse c_warehouse d_warehouse e_warehouse
Reverse order lexicographical brace expansion
e_warehouse d_warehouse c_warehouse b_warehouse a_warehouse
Lexicographical brace expansion with increment specified will iterate through a list of characters from the begin to the end point but skip characters according the increment value
a_warehouse f_warehouse k_warehouse p_warehouse u_warehouse z_warehouse
Multiplicative brace expansion with 2 brace expansions in one command
a1_warehouse a2_warehouse a3_warehouse a4_warehouse a5_warehouse b1_warehouse
b2_warehouse b3_warehouse b4_warehouse b5_warehouse c1_warehouse c2_warehouse
c3_warehouse c4_warehouse c5_warehouse d1_warehouse d2_warehouse d3_warehouse
d4_warehouse d5_warehouse e1_warehouse e2_warehouse e3_warehouse e4_warehouse
e5_warehouse
Brace expansion to use the same root two times in a command. This creates a foo.tgz tar file from a directory under name foo. Its a handy syntax where you are using it inside another loop and want to assume that the base of the word is used multiple times. This example shows it with tar, but it can also be used with mv and cp as per this example.
$ touch foo/foo{a..e}
$ tar czvf foo{.tgz,}
foo/
foo/foob
foo/fooc
foo/fooa
foo/food
foo/fooe
Parameter Expansion
Parameter expansion is also a nice compact syntax with many capabilities such as: allow scripts to set default values for unset variables or options, string substring operations, search and replace substitutions and other use cases. Examples are below.
Check for null and use the parameter if not null or the default value. In this case X is not null so it will be used
$ echo ${X:-2}
1
Check for null and use the parameter if not null or the default value. In this case X is null so the default value will be used
$ echo ${X:-2}
2
Check if the variable is NULL and set and echo it if it is NULL. X is assigned 2 and printed $X. This can both set the variable and use it in the command with the ${:=} syntax.
$ if [ -z "$X" ]; then echo NULL; fi
NULL
$ echo ${X:=2}
2
$ if [ -z "$X" ]; then echo NULL; else echo $X; fi
2
Substring expansion will substitute from an offset point a certain number of characters in the string
$ echo ${X:0:7}
Hello W
Change the offset to the second character and print 7 characters of substring
$ echo ${X:1:7}
ello Wo
Substring from beginning of string but cut off last 2 characters
$ echo ${X:0:-2}
Hello Wor
Get a string length with this version of parameter expansion
$ echo ${#X}
11
Search and replace within a variable. In this example replace the first lowercase o with uppercase O
$ echo ${X/o/O}
HellO World
Search and replace within a variable but with all matches replaced because of the leading slash in the search pattern.
$ echo ${X//o/O}
HellO WOrld
Patterns starting with #, mean the match must start at the beginning of the string in order to be substituted
$ echo ${X/#H/J}
Jello World
Example where searching for match at beginning of string, but failing because match is later in the string
$ echo ${X/#W/J}
Hello World
Patterns beginning with % will only match at the end of string like in this example.
$ echo ${X/%d/d Today}
Hello World Today
Example for end of string match which fails because the match is in the beginning of string.
$ echo ${X/%H/Today}
Hello World
Use shopt with nocasematch to do case insensitive replacement.
$ X="Hello World"
$ echo ${X/hello/Welcome}
Welcome World
Turn off shopt with nocasematch to do case sensitive replacement.
$ X="Hello World"
$ echo ${X/hello/Welcome}
Hello World
Search for environment variables that match a pattern.
$ MY_B=2
$ MY_C=3
$ echo ${!MY*}
MY_A MY_B MY_C
Get a list of matching variables and then loop through each variable and print its value
$ MY_B=2
$ MY_C=3
$ variables=${!MY*}
$ for i in $variables; do echo $i; echo "${!i}"; done
MY_A
1
MY_B
2
MY_C
3
Make a string all uppercase
$ echo ${X^^}
HELLO WORLD
Make a string all lowercase
$ X="Hello World"
$ echo ${X,,}
hello world
Make first character of a string uppercase
$ X="george washington"
$ echo ${X^}
George washington
Make first character of a string lowercase
$ X=BOB
$ echo ${X,}
bOB
Positional Parameters
Positional Parameters are normally thought of as command line parameters, how to use them are shown with examples below.
Parameter $0 is the script name that is running and then $1, $2, $3 etc are command line parameters passed to a script.
echo $0
echo $1
echo $2
echo $3
$ bash ./script.sh apple banana carrot
./script.sh
apple
banana
carrot
Parameter $* is a single variable with all the command line arguments concatenated.
echo $1
echo $2
echo $*
$ bash ./script.sh apple banana
apple
banana
apple banana
Parameter $# is a number with the quantity of positional parameters passed to a script in this case below there are 2 arguments passed.
echo $1
echo $2
echo $*
echo $#
$ bash ./script.sh apple banana
apple
banana
apple banana
2
Tilde Expansion
Tilde expansion is commonly seen with usernames and home directories, examples are shown below.
Tilde Expansion to get the HOME directory of the current user, using just tilde without the username.
root
$ cd ~/
$ pwd
/root
Refer to a specific user’s home directory, not the current user with Tilde and the username
$ pwd
/home/linuxhint
Arithmetic Substitution
Arithmetic Substitution allows bash to do mathematical operations in the shell or in a script. Examples of common usages are shown below.
Simple Arithmetic Substitution with $ and double parentheses
5
Post increment operator will update the value by one after the current command, note there is an equivalent post decrement not shown here.
$ echo $((X++))
2
$ echo $X
3
Pre increment operator will update the value by one just before the current command, note there is an equivalent pre decrement operator not shown here.
$ echo $((++X))
3
$ echo $X
3
Exponent operator can raise a number to a power exponentially
25
Left bitwise shift; in this case shift the bits of the decimal number 8 to the left which essentially multiplies it by 2
16
Right bitwise shift; in this case shift the bits of the decimal number 8 to the right which essentially divides the number by 2
4
Bitwise AND Operator will compare the numbers bit by bit and and the result will be the bits that are all set.
4
Bitwise OR Operator will compare the numbers bit by bit and the results will be the bits where either of the inputs has the bit set.
13
Arithmetic equality operator will test for truth and return 1 or 0
1
Arithmetic inequality operator will test for non-equality and return 1 or 0
0
The Conditional Operator will test the first argument if true, replace with the second argument and if false replace with the third. In this case 5 equals 4+1 so the first conditional is true and 9 is returned. 5 does not equal 4+2 so in the second echo 7 is returned.
9
$ echo $(( 5==4+2 ? 9 : 7 ))
7
You can use hexadecimal numbers in arithmetic expansions, in this case 0xa is equivalent to 10 and 10+7 = 17.
17
Word Splitting
Using the IFS environment variable to register a delimiter, and using the read and readarray commands we can parse strings into an array of tokens and then count the tokens and operate on them. Examples are shown below.
Use IFS parameter as delimiter, read tokens into an array split by IFS which is set to a space character, and then print out the tokens one by one
$ IFS=' '
$ read -a tokens <<< "$text"
$ echo "There are ${#tokens[*]} words in the text."
There are 2 words in the text.
Hello
World
User readarray without IFS and specify delimiter in the readarray command. Note this is just an example where we split a directory path based on the slash delimiter. In this case the code has included the empty string before the first slash which would need to be adjusted in a real usage, but we are just showing how to call readarray to split a string into tokens in an array with a delimiter.
$ readarray -d / -t tokens <<< "$path"
echo "There are ${#tokens[*]} words in the text."
There are 6 words in the text.
home
linuxhint
usr
local
bin
Filename Expansion
When wanting to refer to a list of files or directories in the filesystem, a bash command or bash script can use Filename Expansion to generate a list of files and directories from simple commands. Examples are shown below.
The * character expands to a wildcard and picks up all matching files with the rest of the wild card string. Here we pick up all files ending in .txt and pass them into the du command for checking disk size.
$ echo "Hello World" > content.txt
$ du *.txt
0 a.txt
0 b.txt
0 c.txt
4 content.txt
The ? character will only match a single character not an infinite number of characters, and therefore in this example will only pickup filenames with a single character followed by .txt.
$ echo "Hello World" > content.txt
$ du ?.txt
0 a.txt
0 b.txt
0 c.txt
Characters in brackets expand to match any of the characters. In this example a.txt and c.txt are picked up by the expansion
$ echo "Hello World" > content.txt
$ du [ac].txt
0 a.txt
0 c.txt
Characters in brackets can be a range of characters and we see here all files from a to c range followed by .txt suffix are picked up
$ echo "Hello World" > content.txt
$ du [a-c].txt
0 a.txt
0 b.txt
0 c.txt
Conclusion
We have covered many types of shell expansions in this article, and I hope the simple examples can serve as a cookbook for what is possible in bash to make you more productive with shell expansions. As further references I recommend reading the full Bash Manual, and also the many good articles on NixCraft website about bash scripting including Shell Expansions. We have other articles that may be of interest to you on LinuxHint including: 30 Bash Script Examples, Bash Lowercase Uppercase Strings, Bash Pattern Matching, and Bash Split String Examples. Also we have a popular free 3 hour course on Bash Programming you can find on YouTube.
from Linux Hint https://ift.tt/33KfSwU
0 Comments