The IEEE 1003.2 POSIX Shell Standard (Learning the Korn Shell, 2nd Edition)

A.3. The IEEE 1003.2 POSIX Shell Standard

There have been many attempts to standardize Unix. Hardware companies' monolithic attempts at market domination, fragile industry coalitions, marketing failures, and other such efforts are the stuff of history -- and the stuff of frustration.

Only one standardization effort has not been tied to commercial interests: the Portable Operating System Interface, known as POSIX. This effort started in 1981 with the /usr/group (now UniForum) Standards Committee, which produced the /usr/group Standard three years later. The list of contributors grew. Eventually, the effort to create a formal standard moved under the umbrella of the Institute of Electrical and Electronic Engineers (IEEE) and the International Organization for Standardization (ISO).

The first POSIX standard was published in 1988 and revised in 1996. This one, called IEEE Standard 1003.1, covered low-level issues at the system call level. IEEE Standard 1003.2, covering the shell, utility programs, and user interface issues, was ratified in September 1992 after a six-year effort. In September 2001, a joint revision of both standards was approved. The new standard, covering all the material in the two earlier separate documents, is known as IEEE Standard 1003.1-2001.

The POSIX standards were never meant to be rigid and absolute. The committee members certainly weren't about to put guns to the heads of operating system implementers and force them to adhere. Instead, the standards are designed to be flexible enough to allow for both coexistence of similar available software, so that existing code isn't in danger of obsolescence, and the addition of new features, so that vendors have the incentive to innovate. In other words, they are supposed to be the kind of third-party standards that vendors might actually be interested in following.

As a result, most Unix vendors currently comply with both standards. The Korn shell is no exception; it is intended to be 100% POSIX-compliant. It pays to be familiar with what's in the standard if you want to write code that is portable to different systems.

The shell part of the standard describes utilities that must be present on all systems, and others that are optional, depending upon the nature of the system. One such option is the User Portability Utilities option, which defines standards for interactive shell use and interactive utilities like the vi editor. The standard -- on the order of 2000 pages -- is available through the IEEE; for information, contact the IEEE:

IEEE Customer Service
445 Hoes Lane, PO Box 1331
Piscataway, NJ 08855-1331
(800) 678-IEEE (United States and Canada)
(732) 981-0060 (international/local)
(732) 981-9667 (fax)

[email protected]
http://www.standards.ieee.org/catalog/ordering.html

The committee members had two motivating factors to weigh when they designed the shell standard. On the one hand, the design had to accomodate, as much as possible, existing shell code written under various Bourne-derived shells (the Version 7, System V, BSD, and Korn shells). These shells are different in several extremely subtle ways, most of which have to do with the ways certain syntactic elements interact with each other.

It must have been quite difficult and tedious to spell out these differences, let alone to reach compromises among them. Throw in biases of some committee members towards particular shells, and you might understand why it took six years to ratify the first 1003.2 standard and another five years to merge the two standards.

On the other hand, the shell design had to serve as a standard on which to base future shell implementations. This implied goals of simplicity, clarity, and precision -- objectives that seem especially elusive in the context of the above problems.

The designers found one way of ameliorating this dilemma: they decided that the standard should include not only the features included in the shell, but also those explicitly omitted and those included but with unspecified functionality. The latter category allows some of the existing shells' innovations to "sneak through" without becoming part of the standard, while listing omitted features helps programmers determine which features in existing shell scripts won't be portable to future shells.

The POSIX standard is primarily based on the System V Bourne shell. Therefore you should assume that Korn shell features that aren't present in the Bourne shell also aren't included in the POSIX standard.

However, the Korn shell did contribute a few of its features to the POSIX standard, including:

$((...)) syntax for arithmetic expressions.
$(...) syntax for command substitution, except that the $(< filename) shorthand for $(cat filename) isn't supported.
Tilde expansion (originally derived from the C shell).

The following Korn shell features are left "unspecified" in the standard, meaning that their syntax is acceptable but their functionality is not standardized:

The ((...)) syntax for arithmetic conditionals. The arithmetic test operators introduced in Chapter 5 (e.g., -eq, -lt), however, are included.
The [[...]] syntax for conditional tests. The external test or [...] utility should be used instead. The Korn shell's version of test is POSIX-compliant when used with no more than three arguments. (It also complies with four arguments, if the first argument is !.)
The syntax for defining functions that this book uses. The other syntax shown in Chapter 4 (fname() instead of function fname) is supported, with what we described as "POSIX semantics;" see below.
The select control structure.
Signal numbers are only allowed if the numbers for certain key signals (INT, TERM, and a few others) are the same as on the most important historical versions of Unix. In general, shell scripts should use symbolic names for signals.

The POSIX standard supports functions, but the semantics are weaker than the Korn shell's function-style functions: functions do not have local traps or options, and it is not possible to define local variables. (For this reason, ksh93 has two different syntaxes for defining functions, with different semantics.)

Code blocks ({...; }) are supported. For maximum portability, when you want literal curly braces, you should quote them (for reasons too complicated to go into here).

The POSIX standard introduced the following features, which are different from traditional Bourne shell behavior. ksh93 supports them all:

The command lookup order has been changed to allow certain built-in commands to be overridden by functions -- since aliases aren't included in the standard. Built-in commands are divided into two sets by their positions in the command lookup order: some are processed before functions, some after. Specifically, the built-in commands break, : (do nothing), continue, . (dot), eval, exec, exit, export, readonly, return, set, shift, trap, and unset take priority over functions.
A new built-in command, command, allows you to use built-in commands other than the above even if there are functions of the same name.[148]

[148] But note that it's not a special built-in! Design by committee shows through here.

A new keyword, !, takes the logical "not" of a command's exit status: if command returns exit status 0, ! command returns 1; if command returns a non-zero value, ! command returns 0. ! can be used with &&, ||, and parentheses (for nested subshells) to create logical combinations of exit statuses in conditionals.

The command unset -v is used instead of unset (without an option) to remove the definition of a variable. This provides a better syntactic match with unset -f, for unsetting functions.

Finally, because the POSIX standard is meant to promote shell script portability, it explicitly avoids mention of features that only apply to interactive shell use -- including aliases, editing modes, control keys, and so on. The User Portability Utilities option covers these. It also avoids mentioning certain key implementation issues: in particular, there is no requirement that multitasking be used for background jobs, subshells, etc. This was done to allow portability to non-multitasking systems like MS-DOS, so that, for example, the MKS Toolkit (see later in this appendix) can be POSIX compliant.