UNIX Shell Scripting

Ken Steube
UCSD Extension
[email protected]

This course will teach you to write Bourne shell Scripts. We will then learn about C shell scripts, and will have a brief introduction to perl as well. Scripting skills have many applications, including:

  • Ability to automate tasks, such as
    • Backups
    • Administration tasks
    • Periodic operations on a database via cron
    • Any repetetive operations on files
  • Increase your general knowledge of UNIX
    • Use of environment
    • Use of UNIX utilities
    • Use of features such as pipes and I/O redirection

Our course syllabus is available on-line.

Some tips on working in SDSC's training lab are available.

I also have a comparison between features of the C shell and Bourne shell.

To connect to the SDSC lab computers from home or work you will need to use ssh instead of telnet or rlogin. Be sure to use an implementation of SSH that uses protocol version 1.0.

There is a free ssh program available from PuTTY.

SSH for UNIX is available from SSH.

For this class you will need a data file called last.out. To get this file, put the mouse over the blue text, press the right mouse button, and select "Save link as".

Table of Contents:

  1. Review of a few Basic UNIX Topics (Page 1)
  2. Storing Frequently Used Commands in Files: Shell Scripts (Page 6)
  3. More on Using UNIX Utilities (Page 9)
  4. Performing Search and Replace in Several Files (Page 11)
  5. Using Command-line Arguments for Flexibility (Page 14)
  6. Using Functions (Page 30)
  7. Miscellaneous (Page 38)
  8. Trapping Signals (Page 43)
  9. Understanding Command Translation (Page 50)
  10. Writing Advanced Loops (Page 59)
  11. Creating Remote Shells (Page 67)
  12. More Miscellaneous (Page 73)
  13. Using Quotes (Page 75)





Section 1: Review of a few Basic UNIX Topics

Variables

  • Topics covered: capturing output of a command
  • Utilities covered: echo, expr

  • Before trying the commands below start up a Bourne shell: 
    sh
  • A variable stores a string 
    name="John Doe"
echo $name
  • The quotes are required in the example above because the string contains a special character (the space)
  • A variable may store a number 
    num=137
  • The shell stores this as a string even though it appears to be a number
  • A few UNIX utilities will convert this string into a number to perform arithmetic 
    expr $num + 3
  • Try defining num as '7m8' and try the expr command again
  • What happens when num is not a valid number?
  • Now you may exit the Bourne shell with 
    exit

Page 1



I/O Redirection

  • Topics covered: specifying the input or capturing the output of a command
  • Utilities covered: wc, sort

  • The wc command counts the number of lines, words, and characters in a file 
    wc /etc/passwd
wc -l /etc/passwd
  • You can save the output of wc (or any other command) with output redirection 
    wc /etc/passwd > wc.file
  • You can specify the input with input redirection 
    wc < /etc/passwd
  • Many UNIX commands allow this: 
    sort /etc/passwd
sort < /etc/passwd
  • You can append lines to the end of an existing file with 
    wc -l /etc/passwd >> wc.file

Page 2



Backquotes


Page 3



Pipes

  • Topics covered: using UNIX pipes
  • Utilities covered: sort, cat, head

  • Pipes are used for post-processing data
  • One UNIX command prints results to the standard output (usually the screen), and another command reads that data and processes it 
    sort /etc/passwd | head -5
  • Notice that this pipe can be simplified 
    cat /etc/passwd | head -5
  • You could accomplish the same thing more efficiently with either of the two commands: 
    head -5 /etc/passwd
head -5 < /etc/passwd

Page 4



awk

  • Topics covered: processing columnar data
  • Utilities covered: awk

  • The awk utility is used for processing columns of data
  • A simple example shows how to extract column 5 (the file size) from the output of ls -l 
    ls -l | awk '{print $5}'
  • A more complicated example shows how to sum the file sizes and print the result at the end of the awk run 
    ls -al | awk '{sum = sum + $5} END {print sum}'

Page 5







Section 2: Storing Frequently Used Commands in Files: Shell Scripts

Shell Scripts

  • Topics covered: storing commands in a file and executing the file, $USER variable (standard Bourne shell variable)
  • Utilities covered: date, cal, last, pipes

  • Store the following in a file named simple.sh and execute it 
    #!/bin/sh
# Generate some useful info for
# use at the start of the day
date
cal
last $USER | head -6
  • Shows current date, calendar, and a six of your previous logins for security check
  • You might run this at the beginning of each day
  • Notice that the commands themselves are not displayed, only the results
  • To display the commands verbatim as they run, execute with 
    sh -v simple.sh
  • To echo the commands after variable translation, execute with 
    sh -x simple.sh
  • With -v or -x (or both) you can easily relate any error message that may appear to the command that generated it
  • When an error occurs in a script, the script continues executing at the next command
  • Verify this by changing 'cal' to 'caal' to force an error, and then run the script again
  • Run the 'caal' script with 'sh -v simple.sh' and 'sh -x simple.sh' and verify the error message comes from cal
  • Now you can re-use commands easily and save some typing and mistakes
  • Other standard variable names include: HOME, PATH, TERM, PAGER, PRINTER

Page 6



Storing Strings in Variables


Page 7



Scripting With sed


Page 8







Section 3: More on Using UNIX Utilities

Performing Arithmetic


Page 9



Translating Characters

  • Topics covered: converting one character to another, translating and saving string stored in a variable
  • Utilities covered: tr

  • Copy the file sdsc.txt to your home directory
  • The utility tr translates characters 
    tr 'a' 'Z' < sdsc.txt
  • This example shows how to translate the contents of a variable and display the result on the screen with tr
  • Store the following in a file named tr1.sh and execute it 
    #!/bin/sh
# Translate the contents of a variable
Cat_name="Piewacket"
echo $Cat_name | tr 'a' 'i'
  • This example shows how to change the contents of a variable
  • Store the following in a file named tr2.sh and execute it 
    #!/bin/sh
# Illustrates how to change the contents of a variable with tr
Cat_name="Piewacket"
echo "Cat_name is $Cat_name"
Cat_name=`echo $Cat_name | tr 'a' 'i'`
echo "Cat_name has changed to $Cat_name"
  • You can also specify ranges of characters.
  • This example converts upper case to lower case 
    tr 'A-Z' 'a-z' < file
  • Now you can change the value of the variable and your script has access to the new value

Page 10







Section 4: Performing Search and Replace in Several Files

Processing Multiple Files


Page 11



Using File Name Wildcards in For Loops


Page 12



Search and Replace in Multiple Files


Page 13







Section 5: Using Command-line Arguments for Flexibility

What's Lacking in the Scripts Above?

  • Topics covered: looping over files specified with wildcards
  • Utilities covered: no new utilities

  • File names are hard-coded
  • To execute for loops on different files, the user has to know how to edit the script
  • Not simple enough for general use by the masses
  • Wouldn't it be useful if we could easily specify different file names for each execution of a script?

Page 14



What are Command-line Arguments?

  • Topics covered: specifying command-line arguments
  • Utilities covered: no new utilities

  • Command-line arguments follow the name of a command 
    ls -l .cshrc /etc
  • The command above has three command-line arguments 
    -l	(an option)
.cshrc	(a file name)
/etc	(a directory name)
  • The argument '-l' will be interpreted as an option of ls 
    wc *.sh
  • The command above has an unknown number of arguments, use 'echo *.sh' to see them
  • Your scripts may also have arguments

Page 15



Accessing Command-line Arguments


Page 16



Looping Over the Command-line Arguments


Page 17



If Blocks

  • Topics covered: testing conditions, executing commands conditionally
  • Utilities covered: test (used by if to evaluate conditions)

  • This will be covered on the whiteboard
  • See Chapter 8 of the book

Page 18



The read Command


Page 19



Command Exit Status


Page 20



Regular Expressions

  • Topics covered: search patterns for editors, grep, sed
  • Utilities covered: no new utilities

  • Zero or more characters: .* 
    grep 'provided.*access' sdsc.txt
sed -e 's/provided.*access/provided access/' sdsc.txt
  • Search for text at beginning of line 
    grep '^the' sdsc.txt
  • Search for text at the end of line 
    grep 'of$' sdsc.txt
  • Asterisk means zero or more the the preceeding character 
    a*     zero or more a's
aa*    one or more a's
aaa*   two or more a's
  • Delete all spaces at the ends of lines 
    sed -e 's/ *$//' sdsc.txt > sdsc.txt.new
  • Turn each line into a shell comment 
    sed -e 's/^/# /' sdsc.txt

Page 21



Greed and Eagerness

  • Attributes of pattern matching
  • Greed: a regular expression will match the largest possible string
  • Execute this command and see how big a string gets replaced by an underscore 
    echo 'Big robot' | sed -e 's/i.*o/_/'
  • Eagerness: a regular expression will find the first match if several are present in the line
  • Execute this command and see whether 'big' or 'bag' is matched by the regular expression 
    echo 'big bag' | sed -e 's/b.g/___/'
  • Contrast with this command (notice the extra 'g') 
    echo 'big bag' | sed -e 's/b.g/___/g'
  • Explain what happens in the next example 
    echo 'black dog' | sed -e 's/a*/_/'
  • Hint: a* matches zero or more a's, and there are many places where zero a's appear
  • Try the example above with the extra 'g' 
    echo 'black dog' | sed -e 's/a*/_/g'

Page 22



Regular Expressions Versus Wildcards

  • Topics covered: clarify double meaning of asterisk in patterns
  • Utilities covered: no new utilities

  • Asterisk used in regular expressions for editors, grep, sed
  • Different meaning in file name wildcards on command line and in find command and case statement (see below) 
    regexp  wildcard  meaning

.*      *         zero or more characters, any type
.       ?         exactly one character, any type
[aCg]   [aCg]     exactly one character, from list: aCg
  • Regexps can be anchored to beginning/ending of line with ^ and $
  • Wildcards automatically anchored to both extremes
  • Can use wildcards un-anchored with asterisks 
    ls *bub*

Page 23



Getting Clever With Regular Expressions


Page 24



The case Statement


Page 25



The while Statement


Page 26



Example With a while Loop

  • Topics covered: Using a while loop to read and process a file
  • Utilities covered: no new utilities

  • Copy the file while2.data to your home directory
  • The example below uses a while loop to read an entire file
  • The while loop exits when the read command returns false exit status (end of file)
  • Store the following in a file named while2.sh and execute it 
    #!/bin/sh
# Illustrates use of a while loop to read a file
cat while2.data |   \
while read line
do
	echo "Found line: $line"
done
  • The entire while loop reads its stdin from the pipe
  • Each read command reads another line from the file coming from cat
  • The entire while loop runs in a subshell because of the pipe
  • Variable values set inside while loop not available after while loop

Page 27



Interpreting Options With getopts Command


Page 28



Example With getopts

  • Topics covered: interpreting options in a script
  • Utilities covered: getopts

  • The second example shows how to use if blocks to take action for each option
  • Store the following in a file named getopts2.sh and execute it 
    #!/bin/sh
#
# Usage:
#
#	getopts2.sh [-P string] [-h] [file1 file2 ...]
#
# Example runs:
#
#	getopts2.sh -h -Pxerox file1 file2
#	getopts2.sh -hPxerox file1 file2
#
# Will print out the options and file names given
#

# Initialize our variables so we don't inherit values
# from the environment
opt_P=''
opt_h=''

# Parse the command-line options
while getopts 'P:h' option
do
	case "$option" in
	"P")	opt_P="$OPTARG"
		;;
	"h")	opt_h="1"
		;;
	?)	echo "getopts2.sh: Bad option specified...quitting"
		exit 1
		;;
	esac
done

shift `expr $OPTIND - 1`

if [ "$opt_P" != "" ]
then
	echo "Option P used with argument '$opt_P'"
fi

if [ "$opt_h" != "" ]
then
	echo "Option h used"
fi

if [ "$*" != "" ]
then
	echo "Remaining command-line:"
	for arg in "$@"
	do
		echo "	$arg"
	done
fi
  • Execute it several times with 
    sh getopts2.sh -h -Pjunky
sh getopts2.sh -hPjunky
sh getopts2.sh -h -Pjunky /etc /tmp
  • Can also implement actions inside case statement if desired

Page 29







Section 6: Using Functions

Functions

  • Sequence of statements that can be called anywhere in script
  • Used for
    • Good organization
    • Create re-usable sequences of commands

Page 30



Define a Function

  • Define a function 
    echo_it () {
  echo "In function echo_it"
}
  • Use it like any other command 
    echo_it
  • Put these four lines in a script and execute it

Page 31



Function Arguments

  • Functions can have command-line arguments 
    echo_it () {
  echo "Argument 1 is $1"
  echo "Argument 2 is $2"
}
echo_it arg1 arg2
  • When you execute the script above, you should see 
    Argument 1 is arg1
Argument 2 is arg2
  • Create a script 'difference.sh' with the following lines: 
    #!/bin/sh
echo_it () {
	echo Function argument 1 is $1
}
echo Script argument 1 is $1
echo_it Barney
  • Execute this script using 
    sh difference.sh Fred
  • Notice that '$1' is echoed twice with different values
  • The function has separate command-line arguments from the script's

Page 32



Example With Functions

  • Use functions to organize script 
    read_inputs () { ... }
compute_results () { ... }
print_results () { ... }
  • Main program very readable 
    read_inputs
compute_results
print_results

Page 33



Functions in Pipes

  • Can use a function in a pipe 
    ls_sorter () {
  sort -n +4
}
ls -al | ls_sorter
  • Function in pipe executed in new shell
  • New variables forgotten when function exits

Page 34



Inherited Variables

  • Variables defined before calling script available to script 
    func_y () {
  echo "A is $A"
  return 7
}
A='bub'
func_y
if [ $? -eq 7 ] ; then ...
  • Try it: is a variable defined inside a function available to the main program?

Page 35



Functions -vs- Scripts

  • Functions are like separate scripts
  • Both functions and scripts can:
  • Use command-line arguments 
    echo First arg is $1
  • Operate in pipes 
    echo "test string" | ls_sorter
  • Return exit status 
    func_y arg1 arg2
if [ $? -ne 0 ] ...

Page 36



Libraries of Functions

  • Common to store definitions of favorite functions in a file
  • Then execute file with 
    . file
  • Period command executes file in current shell
  • Compare to C shell's source command

Page 37







Section 7: Miscellaneous

Here Files

  • Data contained within script 
    cat << END
This script backs up the directory
named as the first command-line argument,
which in your case in $1.
END
  • Terminator string must begin in column one
  • Variables and backquotes translated in data
  • Turn off translation with \END

Page 38



Example With Here File

  • Send e-mail to each of several users 
    for name in $USER
do
  mailx -s 'hi there' $name << EOF
  Hi $name, meet me at the water
  fountain
EOF
done
  • Use <<- to remove initial tabs automatically

Page 39



Set: Shell Options

  • Can change Bourne shell's options at runtime
  • Use set command inside script 
    set -v
set +v
set -xv
  • Toggle verbose mode on and off to reduce amount of debugging output

Page 40



Set: Split a Line

  • Can change Bourne shell's options 
    set -- word1 word2
echo $1, $2
  word1, word2
  • Double dash important!
  • Word1 may begin with a dash, what if word1 is '-x'?
  • Double dash says "even if first word begins with '-', do not treat it as an option to the shell

Page 41



Example With Set

  • Read a line from keyboard
  • Echo words 3 and 5 
    read var
set -- $var
echo $3 $5
  • Best way to split a line into words

Page 42







Section 8: Trapping Signals

What are Signals?

  • Signals are small messages sent to a process
  • Process interrupted to handle signal
  • Possibilities for managing signal:
    • Terminate
    • Ignore
    • Perform a programmer-defined action

Page 43



Common Signals

  • Common signals are
    • SIGINTR sent to foreground process by ^C
    • SIGHUP sent when modem line gets hung up
    • SIGTERM sent by kill -9
  • Signals have numeric equivalents 
    2 SIGINTR
9 SIGTERM

Page 44



Send a Signal

  • Send a signal to a process 
    kill -2 PID
kill -INTR PID

Page 45



Trap Signals

  • Handling Signals 
    trap "echo Interrupted; exit 2" 2
  • Ignoring Signals 
    trap "" 2 3
  • Restoring Default Handler 
    trap 2

Page 46



Where to Find List of Signals

  • See file 
    /usr/include/sys/signal.h

Page 47



User Signals

  • SIGUSR1, SIGUSR2 are for your use
  • Send to a process with 
    kill -USR1 PID
  • Default action is to terminate process

Page 48



Experiment With Signals

  • Script that catches USR1
  • Echo message upon each signal 
    trap 'echo USR1' 16
while : ; do
  date
  sleep 3
done
  • Try it: does signal interrupt sleep?

Page 49







Section 9: Understanding Command Translation

Command Translation

  • Common translations include
    • Splitting at spaces, obey quotes
    • $HOME -> /users/us/freddy
    • `command` -> output of command
    • I/O redirection
    • File name wildcard expansion
  • Combinations of quotes and metacharacters confusing
  • Resolve problems by understanding order of translations

Page 50



Experiment With Translation

  • Try wildcards in echo command 
    echo b*
b budget bzzzzz
  • b* translated by sh before echo runs
  • When echo runs it sees 
    echo b budget bzzzzz
  • Echo command need not understand wildcards!

Page 51



Order of Translations

  • Splits into words at spaces and tabs
  • Divides commands at 
    ; & | && || (...) {...}
  • Echos command if -v
  • Interprets quotes
  • Performs variable substitution

Page 52



Order of Translations (continued)

  • Performs command substitution
  • Implements I/O redirection and removes redirection characters
  • Divides command again according to IFS
  • Expands file name wildcards
  • Echos translated command if -x
  • Executes command

Page 53



Exceptional Case

  • Delayed expansion for variable assignments 
    VAR=b*
echo $VAR
  b  b_file
  • Wildcard re-expanded for each echo

Page 54



Examples With Translation

  • Variables translated before execution
  • Can store command name in variable 
    command="ls"
$command
  file1 file2 dir1 dir2...
  • Variables translated before I/O redirection 
    tempfile="/tmp/scriptname_$$"
ls -al > $tempfile

Page 55



Examples (continued)

  • Delayed expansion of wildcards in variable assignment
  • Output of this echo command changes when directory contents change (* is re-evaluated each time the command is run) 
    x=*
echo $x
  • Can view values stored in variables with 
    set
  • Try it: verify that the wildcard is stored in x without expansion

Page 56



Examples (continued)

  • Wildcards expanded after redirection (assuming file* matches exactly one file): 
    cat < file*
  file*: No such file or directory
  • Command in backquotes expanded fully (and before I/O redirection) 
    cat < `echo file*`
  (contents of file sent to screen)

Page 57



Eval Command

  • Forces an extra evaluation of command 
    eval cat \< file*
  (contents of matching file)
  • Backslash delays translation of < until second translation

Page 58







Section 10: Writing Advanced Loops

While loops

  • Execute statements while a condition is true 
    i=0
while [ $i -lt 10 ]
do
  echo I is $i
  i=`expr $i + 1`
done

Page 59



Until loops

  • Execute statements as long as a condition is false 
    until grep "sort" dbase_log > /dev/null
do
  sleep 10
done
echo "Database has been sorted"
  • Example executes until grep is unsuccessful

Page 60



Redirection of Loops

  • Can redirect output of a loop 
    for f in *.c
do
  wc -l $f
done > loop.out
  • Loop runs in separate shell
  • New variables forgotten after loop
  • Backgrounding OK, too

Page 61



Continue Command

  • Used in for, while, and until loops
  • Skip remaining statements
  • Return to top of loop 
    for name in *
do
  if [ ! -f $name ] ; then
    continue
  fi
  echo "Found file $name"
done
  • Example loops over files, skips directories

Page 62



Break Command

  • Used in for, while, and until loops
  • Skip remaining statements
  • Exit loop 
    for name in *
do
  if [ ! -r $name ] ; then
    echo "Cannot read $name, quitting loop"
    break
  fi
  echo "Found file or directory $name"
done
  • Example loops over files and directories, quits if one is not readable

Page 63



Case Command

  • Execute one of several blocks of commands 
    case "string" in
pattern1)
  commands ;;
pattern2)
  commands ;;
*) # Default case
  commands ;;
esac
  • Patterns specified with file name wildcards 
    quit) ...
qu*)   ...

Page 64



Example With Case

  • Read commands from keyboard and interpret
  • Enter this script 'case.sh' 
    echo Enter a command
while read cmd
do
  case "$cmd" in
    list) ls -al ;;
    freespace) df . ;;
    quit|Quit) break ;;
    *) echo "$cmd: No such command" ;;
  esac
done
echo "All done"
  • When you run it, the script waits for you to type one of: 
    list
freespace
quit
Quit
  • Try it: modify the example so any command beginning with characters "free" runs df

Page 65



Infinite Loops

  • Infinite loop with while 
    while :
do
  ...
done
  • : is no-op, always returns success status
  • Must use break or exit inside loop for it to terminate

Page 66







Section 11: Forking Remote Shells

Remote Shells

  • Rsh command 
    rsh hostname "commands"
  • Runs commands on remote system
  • Must have .rhosts set up
  • Can specify different login name 
    rsh -l name hostname "commands"

Page 67



Examples With rsh

  • Check who's logged on 
    rsh spooky "finger"
  • Run several remote commands 
    rsh spooky "uname -a; time"
  • Executes .cshrc on remote system
  • Be sure to set path in .cshrc instead of .login

Page 68



Access Control with .Rhosts

  • May get "permission denied" error from rsh
  • Fix this with ~/.rhosts on remote system
  • Example: provide for remote shell from spunky to spooky 
    spunky % rlogin spooky
spooky % vi ~/.rhosts
	(insert "spunky login-name")
spooky % chmod 600 ~/.rhosts
spooky % logout
spunky % rsh spooky uname -a
	spooky 5.5 sparc SUNW,Ultra-1
  • May also rlogin without password: security problem!

Page 69



Remote Shell I/O

  • Standard output sent to local host 
    rsh spooky finger > finger.spooky
  • Standard input sent to remote host 
    cat local-file | rsh spooky lpr -

Page 70



Return Status

  • Get return status of rsh 
    rsh mayer "uname -a"
echo $?
  • Returns 0 if rsh managed to connect to remote host
  • Returns 1 otherwise
    • Invalid hostname
    • Permission denied

Page 71



Remote Return Status

  • What about exit status of remote command?
  • Have to determine success or failure from stdout or stderr

Page 72







Section 12: More Miscellaneous

Temporary Files

  • Use unique names to avoid clashes 
    tempfile=$HOME/Weq_$$
command > $tempfile
  • $$ is PID of current shell
  • Avoids conflict with concurrent executions of script
  • Do not use /tmp!

Page 73



Wait Command

  • Wait for termination of background job 
    command &
pid=$!
(other processing)
wait $pid
  • Allows overlap of two or more operations

Page 74







Section 13: Using Quotes

Quotes

  • Provide control of collapsing of spaces and translation of variables
  • Try it: run three examples
  • No quotes (variables translated, spaces collapsed) 
    echo Home:   $HOME
  Home: /users/us/freddy
  • Double quotes (no collapsing) 
    echo "Home:   $HOME"
  Home:   /users/us/freddy
  • Single quotes (no translation or collapsing) 
    echo 'Home:   $HOME'
  Home:   $HOME
  • Try it: single quotes within double quotes 
    echo "Home directory '$HOME' is full..."

Page 75



Metacharacters

  • Characters with special meaning to shell 
    " ' ` $ * [ ] ?
; > < & ( ) \
  • Avoid special meaning with quoting 
    echo 'You have $20'
  • Backslash like single quotes
  • Applies only to next character 
    echo You have \$20

Page 76



Examples With Quotes

  • Bad command line: 
    grep dog.*cat file
  • Shell tries to expand dot.*cat as file name wildcard
  • Use quotes to avoid translation 
    grep 'dog.*cat' file
  • Single quotes OK in this case because we don't need variable translation

Page 77



More Examples With Quotes

  • Read name and search file for name 
    read name
grep "$name" dbase
  • Single quotes not OK because we need variable translation

Page 78



Searching for Metacharacters

  • Bad command line: search for dollar sign 
    grep "Gimme.*$20" file
  • Problem: shell translates variable $20
  • Solution: use single quotes 
    grep 'Gimme.*$20' file

Page 79