17.2 Objective 2: Customize or Write Simple Scripts

You've seen how the use of bash configuration files, aliases, functions, variables, and key bindings can customize and make interaction with your Linux system efficient. The next step in your relationship with the shell is to use its natural programming capability, or scripting language. The scripting language of the original Bourne shell is found throughout a Linux system, and bash is fully compatible with it. This section covers essential bash scripting language concepts as required for Exam 102.

In order to have a full appreciation of shell scripting on Linux, it's important to look at your Linux system as a collection of unique and powerful tools. Each of the commands available on your Linux system, along with those you create yourself, has some special capability. Bringing these capabilities together to solve problems is among the basic philosophies of the Unix world.

17.2.1 Script Files

Just as the configuration files discussed in the last section are plain text files, so are the scripts for your shell. In addition, unlike compiled languages such as C or Pascal, no compilation of a shell program is necessary before it is executed. You can use any editor to create script files, and you'll find that many scripts you write are portable from Linux to other Unix systems.

17.2.1.1 Creating a simple bash script

The simplest scripts are those that simply string together some basic commands and perhaps do something useful with the output. Of course, this can be done with a simple alias or function, but eventually you'll have a requirement that exceeds a one-line request, and a shell script is the natural solution. Aliases and functions have already been used to create a rudimentary new command, lsps. Now let's look at a shell script (Example 17-6) that accomplishes the same thing.

Example 17-6. The lsps Script

# a basic lsps command script for bash
ls -l $1
ps -aux | grep `/bin/basename $1`

As you can see, the commands used in this simple script are identical to those used in the alias and in the function created earlier. To make use of this new file, instruct your currently running bash shell to source it, giving it an option for the $1 positional parameter:

$ source ./lsps /usr/sbin/httpd

If you have /usr/sbin/httpd running, you should receive output similar to that found previously for the alias. By replacing the word source with a single dot, you can create an alternate shorthand notation to tell bash to source a file, as follows:

$ . ./lsps /usr/sbin/httpd

Another way to invoke a script is to start a new invocation of bash and tell that process to source the file. To do this, simply start bash and pass the script name and argument to it:

$ /bin/bash ./lsps /usr/sbin/httpd

This last example gives us the same result; however, it is significantly different from the alias, the function, or the sourcing of the lsps file. In this particular case, a new invocation of bash was started to execute the commands in the script. This is important, because the environment in which the commands are running is distinct from the environment where the user is typing. This is described in more detail later.

The ./ syntax indicates that the file you're referring to is in the current working directory. To avoid specifying ./ for users other than the superuser, put the directory . in the PATH . The PATH of the superuser should not include the current working directory, as a security precaution against Trojan horse-style attacks.

Thus far, a shell script has been created and invoked in a variety of ways, but it hasn't been made into a command. A script really becomes useful when it can be called by name like any other command.

17.2.1.2 Executable files

On a Linux system, programs are said to be executable if they have content that can be run by the processor (native execution) or by another program such as a shell (interpreted execution). However, in order to be eligible for execution when called at the command line, the files must have attributes that indicate to the shell that they are executable. Conspicuously absent is anything in the filename that indicates that the file is executable, such as the file extension of .exe found on MS-DOS and Windows applications. It would be possible to name our example file lsps.exe if desired, or for that matter lsps.sh or lsps.bin. None of these extensions has any meaning to the shell, though, and the extension would become part of the command entered when executing the program. For this reason, most executable Linux programs and scripts don't have filename extensions. To make a file executable, it must have at least one of its executable bits set. To turn our example script from a plain text file to an executable program, that bit must be set using the chmod command:

$ chmod a+x lsps

Once this is done, the script is executable by owner, group members, and everyone else on the system. At this point, running the new command from the bash prompt yields the familiar output:

$ ./lsps /usr/sbin/httpd

When lsps is called by name, the commands in the script are interpreted and executed by the bash shell. However, this isn't ultimately what is desired. In many cases, users will be running some other shell interactively but will still want to program in bash. Programmers also use other scripting languages such as Perl. To have our scripts interpreted correctly, the system must be told which program should interpret the commands in our scripts.

17.2.1.3 She-bang!

There are many kinds of script files found on a Linux system, and each interpreted language comes with a unique and specific command structure. There needs to be a way to tell Linux which interpreter to use. This is accomplished by using a special line at the top of the script naming the appropriate interpreter. Linux examines this line and launches the specified interpreter program, which then reads the rest of the file. The special line must begin with #!, a construct often called "she-bang." For bash, the she-bang line is:

#!/bin/bash

This command explicitly states that the program named bash can be found in the /bin directory and designates bash to be the interpreter for the script. You'll also see other types of lines on script files, including:

#!/bin/sh

The bourne shell.

#!/bin/csh

The C-shell.

#!/bin/tcsh

The enhanced C-shell.

#!/bin/sed

The stream editor.

#!/usr/bin/awk

The awk programming language.

#!/usr/bin/perl

The Perl programming language.

Each of these lines specifies a unique command interpreter use for the script lines that follow.^[4]

^[4] bash is fully backward compatible with sh; sh is just a link to bash on Linux systems.

On the Exam An incorrectly stated she-bang line can cause the wrong interpreter to attempt to execute commands in a script.

17.2.1.4 The shell script's environment

When running a script with #!/bin/bash, a new invocation of bash with its own environment is started to execute the script's commands as the parent shell waits. Exported variables in the parent shell are copied into the child's environment; the child shell executes the appropriate shell configuration files (such as .bash_profile). Because configuration files will be run, additional shell variables may be set and environment variables may be overwritten. If you are depending upon a variable in your shell script, be sure that it is either set by the shell configuration files or exported into the environment for your use, but not both.

Another important concept regarding your shell's environment is one-way inheritance. Although your current shell's environment is passed into a shell script, that environment is not passed back to the original shell when your program terminates. This means that changes made to variables during the execution of your script are not preserved when the script exits. Instead, the values in the parent shell's variables are the same as they were before the script executed. This is a basic Unix construct; inheritance goes from parent process to child process, and not the other way around.

On the Exam It is important to remember how variables are set, how they are inherited, and that they are inherited only from parent process to child process.

17.2.1.5 Location, ownership, and permissions

The ability to run any executable program, including a script, under Linux depends in part upon its location in the filesystem. Either the user must explicitly specify the location of the file to run or it must be located in a directory known by the shell to contain executables. Such directories are listed in the PATH environment variable. For example, the shells on a Linux system (including bash) are located in /bin. This directory is usually in the PATH, because you're likely to run programs that are stored there. When you create shell programs or other utilities of your own, you may want to keep them together and add the location to your own PATH. If you maintain your own bin directory, you might add the following line to your .bash_ profile:

PATH=$PATH:$HOME/bin

This statement modifies your path to include your /home/bin directory. If you add personal scripts and programs to this directory, bash finds them automatically.

Execute permissions (covered in Section 4.5) also affect your ability to run a script. Since a script is just a text file, execute permission must be granted to them before they are considered executable, as shown earlier.

You may wish to limit access to the file from other users using:

$ chmod 700 ~/bin/lsps

This prevents anyone but the owner from making changes to the script.

The issue of file ownership is dovetailed with making a script executable. By default, you own all of the files you create. However, if you are the system administrator, you'll often be working as the superuser and will be creating files with username root as well. It is important to assign the correct ownership and permission to scripts to ensure that they are secured.

17.2.1.6 SUID and GUID rights

On rare occasions, it may become necessary to allow a user to run a program under the name of a different user. This is usually associated with programs run by nonprivileged users who need special privileges to execute correctly. Linux offers two such rights, known as set user ID (SUID) and set group ID (SGID).

When an executable file is granted the SUID right, processes created to execute it are owned by the user who owns the file instead of the user who launched the program. This is a security enhancement in that the delegation of a privileged task or ability does not imply that the superuser password must be widely known. On the other hand, any process whose file is owned by root and which has the SUID set will run as root for everyone. This could represent an opportunity to break the security of a system if the file itself is easy to attack (as a script is). For this reason, Linux systems will ignore SUID and SGID attributes for script files. Setting SUID and SGID attributes is detailed in Section 4.5.

On the Exam Be sure to think through any questions that require you to determine a user's right to execute a file. Consider location, ownership, execute permissions, and SUID/SGID rights together. Also, watch for new scripts that haven't been granted any execute privileges.

17.2.2 Basic bash Scripts

Now that some of the requirements for creating and using executable scripts are established, some of the features that make them so powerful can be introduced. This section contains basic information needed to customize and create new bash scripts.

17.2.2.1 Return values

As shell scripts execute, it is important to confirm that their constituent commands complete successfully. Most commands offer a return value to the shell when they terminate. This value is a simple integer and has meaning specific to the program you're using. Almost all programs return the value when they are successful, and return a nonzero value when a problem is encountered. The value is stored in the special bash variable $?, which can be tested in your scripts to check for successful command execution. This variable is reset for every command executed by the shell, so you must test it immediately after execution of the command you're verifying. As a simple example, try using the cat program on a nonexistent file:

$ cat bogus_file
cat: bogus_file: No such file or directory

Then immediately examine the status variable twice:

$ echo $?
1
$ echo $?
0

The first echo yielded 1 (failure) because the cat program failed to find the file you specified. The second echo yielded 0 (success) because the first echo command succeeded. A good script makes use of these status flags to exit gracefully in case of errors.

If it sounds backward to equate zero with success and nonzero with failure, consider how these results are used in practice:

Error detection

Scripts that check for errors include if-then code to evaluate a command's return status:

command
if (failure_returned) {
  ...error recovery code...
}

In a bash script, failure_returned is simply the $? variable, which contains the result of the command's execution.

Error classification

Since commands can fail for multiple reasons, many return more than one failure code. For example, grep returns 0 if matches are found and 1 if no matches are found; it returns 2 if there is a problem with the search pattern or input files. Scripts may need to respond differently to various error conditions.

On the Exam Make certain you understand the meaning of return values in general and that they are stored in the $? variable.

17.2.2.2 File tests

During the execution of a shell script, specific information about a file -- such as whether it exists, is writable, is a directory or a file, and so on -- may sometimes be required. In bash, the built-in command test performs this function.^[5] test has two general forms:

^[5] There is also a standalone executable version of test available in /usr/bin for non-bash shells.

test expression

In this form, test and an expression are explicitly stated.

[ expression ]

In this form, test isn't mentioned; instead, the expression is enclosed inside brackets.

The expression can be formed to look for such things as empty files, the existence of files, the existence of directories, equality of strings, and others. (See the more complete list with their operators in the next section.)

When used in a script's if or while statement, the brackets ([ and ]) may appear to be grouping the test logically. In reality, [ is simply another form of the test command, which requires the trailing ]. A side effect of this bit of trickery is that the spaces around [ and ] are mandatory, a detail that is sure to get you into trouble eventually. See the later section, "Abbreviated bash command reference," for some of the available tests.

17.2.2.3 Command substitution

Bash offers a handy ability to do command substitution. This feature allows you to replace $(command ) with the result of command, usually in a script. That is, wherever $(command) is found, its output is substituted prior to interpretation by the shell. For example, to set a variable to the number of lines in your .bashrc file, you could use wc -l:

$ RCSIZE=$(wc -l ~/.bashrc)

An older form of command substitution encloses command in backquotes:

$ RCSIZE=`wc -l ~/.bashrc`

The result is the same, except that the backquote syntax allows the backslash character to escape the dollar symbol ($), the backquote (`), and another backslash (\ ). The $(command) syntax avoids this nuance by treating all characters between the parentheses literally.

17.2.2.4 Mailing from scripts

The scripts you write will often be rummaging around your system at night when you're asleep or at least while you're not watching. Since you're too busy to check on every script's progress, a script will sometimes need to send some mail to you or another administrator. This is particularly important when something big goes wrong or when something important depends on the script's outcome. Sending mail is as simple as piping into the mail command:

echo "Backup failure 5" | mail -s "Backup failed" root

The -s option indicates that a quoted subject for the email follows. The recipient could be yourself, root, or if your system is configured correctly, any Internet email address. If you need to send a log file, redirect the input of mail from that file:

mail -s "subject" recipient < logfile

Sending email from scripts is easy and makes tracking status easier than reviewing log files every day. On the downside, having an inbox full of "success" messages can be a nuisance too, so many scripts are written so that mail is sent only in response to an important event, such as a fatal error.

17.2.2.5 Abbreviated bash command reference

This section lists some of the important bash built-in commands used when writing scripts. Please note that not all of the bash commands are listed here; for a complete overview of the bash shell, see Learning the bash Shell by Cameron Newham and Bill Rosenblatt (O'Reilly & Associates).

Syntax

break [n]

Description

Exit from the innermost (most deeply nested) for, while, or until loop or from the n innermost levels of the loop.

Syntax

case string
in
   regex1)
   commands1
   ;;
   regex2)
   commands2
   ;;
   ...
esac

Description

Choose string from among a series of possible regular expressions. If string matches regular expression regex1, perform the subsequent commands1. If string matches regex2, performcommands2. Proceed down the list of regular expressions until one is found. To catch all remaining strings, use *) at the end.

Syntax

continue [n]

Description

Skip remaining commands in a for, while, or until loop, resuming with the next iteration of the loop (or skipping n loops).

Syntax

echo [options] [string]

Description

Write string to standard output, terminated by a newline. If no string is supplied, echo only a newline.

Frequently used options

-e

Enables interpretation of escape characters.

-n

Suppresses the trailing newline in the output.

\a

Sounds an audible alert.

\b

Inserts a backspace.

\c

Suppresses the trailing newline (same as -n).

\f

Form feed.

Syntax

exit [n]

Description

Exit a shell script with status n. The value for n can be (success) or nonzero (failure). If n is not given, the exit status is that of the most recent command.

Example

if ! test -f somefile
then
  echo "Error: Missing file somefile"
  exit 1
fi
for x [in list]
do
   commands
done

Syntax

for x in list
do
   commands
done

Description

Assign each word in list to x in turn and execute commands. If list is omitted, it is assumed that positional parameters from the command line, which are stored in $@, are to be used.

Example

for filename in bigfile*
{
  echo "Compressing $filename"
  gzip $filename
}

Syntax

function name
{
   commands
}

Description

Define function name. Positional parameters ($1, $2, ...) can be used within commands.

Example

# function myfunc
{
  echo "parameter is $1"
}
# myfunc 1
parameter is 1
# myfunc two
parameter is two

Syntax

getopts  string name [args]

Description

Process command-line arguments (or args, if specified) and check for legal options. The getopts command is used in shell script loops and is intended to ensure standard syntax for command-line options. The string contains the option letters to be recognized by getopts when running the script. Valid options are processed in turn and stored in the shell variable name. If an option letter is followed by a colon, the option must be followed by one or more arguments when the command is entered by the user.

Syntax

if expression1
then
    commands1
elif expression2
then
    commands2
else
    commands
fi

Description

The if command is used to define a conditional statement. There are three possible formats for using the if command:

if-then-fi
if-then-else-fi 
if-then-elif-then-...fi

The expressions are made up of tests (or [] commands).

Syntax

kill [options] IDs

Description

Send signals to each specified process or job ID, which you must own unless you are a privileged user. The default signal sent with the kill command is TERM, instructing processes to shut down.

Options

-l

List the signal names.

-s signal or -signal

Specifies the signal number or name.

Syntax

read [options] variable1 [variable2...]

Description

Read one line of standard input, and assign each word to the corresponding variable, with all remaining words assigned to the last variable.

Example

echo -n "Enter last-name, age, height, and weight > "
read lastname everythingelse
echo $lastname
echo $everythingelese

The name entered is placed in variable $lastname; all of the other values, including the spaces between them, are placed in $everythingelse.

Syntax

return [n]

Description

This command is used inside a function definition to exit the function with status n. If n is omitted, the exit status of the previously executed command is returned.

Syntax

shift [n]

Description

Shift positional parameters down n elements. If n is omitted, the default is 1, so $2 becomes $1, $3 becomes $2, and so on.

Syntax

source file [arguments]
. file [arguments]

Description

Read and execute lines in file. The file does not need to be executable but must be in a directory listed in PATH. The "dot" syntax is equivalent to stating source.

Syntax

test expression
[ expression ]

Description

Evaluate the conditional expression and return a status of (true) or 1 (false). The first form explicitly calls out the test command. The second form implies the test command. The spaces around expression are required in the second form. expression is constructed using options.

Frequently used options

-d file

True if file exists and is a directory

-e file

True if file exists

-f file

True if file exists and is a regular file

-L file

True if file exists and is a symbolic link

-n string

True if the length of string is nonzero

-r file

True if file exists and is readable

-s file

True if file exists and has a size greater than zero

-w file

True if file exists and is writable

-x file

True if file exists and is executable

-z string

True if the length of string is zero

file1 -ot file2

True if file1 is older than file2

string1 = string2

True if the strings are equal

string1 != string2

True if the strings are not equal

Example

To determine if a file exists and is readable, use the -r option:

if test -r file
then
   echo "file exists"
fi

Using the [ ] form instead, the same test looks like this:

if [ -r file ]
then
   echo "file exists"
fi

Syntax

until
    test-commands
do
    commands
done

Description

Execute test-commands (usually a test command) and if the exit status is nonzero (that is, the test fails), perform commands ; repeat. Opposite of while.

Syntax

while
    test-commands
do
    commands
done

Description

Execute test-commands (usually a test command) and if the exit status is zero, perform commands; repeat. Opposite of until.

Example

Example 17-7 shows a typical script from a Linux system. This example is /etc/rc.d/init.d/sendmail, which is the script that starts and stops sendmail. This script demonstrates many of the built-in commands referenced in the last section.

Example 17-7. Sample sendmail Startup Script

#!/bin/sh
#
# sendmail     This shell script takes care of starting 
#              and stopping sendmail.
#
# chkconfig: 2345 80 30
# description: Sendmail is a Mail Transport Agent, which
#              is the program that moves mail from one 
#              machine to another.
# processname: sendmail
# config: /etc/sendmail.cf
# pidfile: /var/run/sendmail.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Source sendmail configuration.
if [ -f /etc/sysconfig/sendmail ] ; then
  . /etc/sysconfig/sendmail
else
  DAEMON=yes
  QUEUE=1h
fi

# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0

[ -f /usr/sbin/sendmail ] || exit 0

# See how we were called.
case "$1" in
  start)
  # Start daemons.
  echo -n "Starting sendmail: "
  /usr/bin/newaliases > /dev/null 2>&1
  for i in virtusertable access domaintable mailertable ; do
    if [ -f /etc/mail/$i ] ; then
      makemap hash /etc/mail/$i < /etc/mail/$i
    fi
  done
  daemon /usr/sbin/sendmail $([ "$DAEMON" = yes ] \
    && echo -bd) $([ -n "$QUEUE" ] && echo -q$QUEUE)
  echo
  touch /var/lock/subsys/sendmail
  ;;

  stop)
  # Stop daemons.
  echo -n "Shutting down sendmail: "
  killproc sendmail
  echo
  rm -f /var/lock/subsys/sendmail
  ;;

  restart)
  $0 stop
  $0 start
  ;;

  status)
  status sendmail
  ;;

  *)
  echo "Usage: sendmail {start|stop|restart|status}"
  exit 1
esac

exit 0

On the Exam You should be familiar with a script's general structure, as well as the use of she-bang, test, if statements and their syntax (including the trailing fi ), return values, exit values, and so on.