6.7 Capturing the Output and Error Streams from a Unix Shell Command

Credit: Brent Burley

6.7.1 Problem

You need to run an external process in a Unix-like environment and capture both the output and error streams from the external process.

6.7.2 Solution

The popen2 module lets you capture both streams, but you also need help from fcntl to make the streams nonblocking and thus avoid deadlocks:

import os, popen2, fcntl, FCNTL, select

def makeNonBlocking(fd):
    fl = fcntl.fcntl(fd, FCNTL.F_GETFL)
    try:
      fcntl.fcntl(fd, FCNTL.F_SETFL, fl | FCNTL.O_NDELAY)
    except AttributeError:
      fcntl.fcntl(fd, FCNTL.F_SETFL, fl | FCNTL.FNDELAY)

def getCommandOutput(command):
    child = popen2.Popen3(command, 1) # Capture stdout and stderr from command
    child.tochild.close(  )             # don't need to write to child's stdin
    outfile = child.fromchild
    outfd = outfile.fileno(  )
    errfile = child.childerr
    errfd = errfile.fileno(  )
    makeNonBlocking(outfd)            # Don't deadlock! Make fd's nonblocking.
    makeNonBlocking(errfd)
    outdata = errdata = ''
    outeof = erreof = 0
    while 1:
        ready = select.select([outfd,errfd],[],[]) # Wait for input
        if outfd in ready[0]:
            outchunk = outfile.read(  )
            if outchunk == '': outeof = 1
            outdata = outdata + outchunk
        if errfd in ready[0]:
            errchunk = errfile.read(  )
            if errchunk == '': erreof = 1
            errdata = errdata + errchunk
        if outeof and erreof: break
        select.select([],[],[],.1) # Allow a little time for buffers to fill
    err = child.wait(  )
    if err != 0:
        raise RuntimeError, '%s failed with exit code %d\n%s' % (
            command, err, errdata)
    return outdata

def getCommandOutput2(command):
    child = os.popen(command)
    data = child.read(  )
    err = child.close(  )
    if err:
        raise RuntimeError, '%s failed with exit code %d' % (command, err)
    return data

6.7.3 Discussion

This recipe shows how to execute a Unix shell command and capture the output and error streams in Python. By contrast, os.system sends both streams directly to the terminal. The presented getCommandOutput(command) function executes a command and returns the command's output. If the command fails, an exception is raised, using the text captured from the command's stderr as part of the exception's arguments.

Most of complexity of this code is due to the difficulty of capturing both the output and error streams of the child process at the same time. Normal (blocking) read calls may deadlock if the child is trying to write to one stream, and the parent is waiting for data on the other stream, so the streams must be set to nonblocking, and select must be used to wait for data on the streams.

Note that the second select call adds a 0.1-second sleep after each read. Counterintuitively, this allows the code to run much faster, since it gives the child time to put more data in the buffer. Without this, the parent may try to read only a few bytes at a time, which can be very expensive.

If you want to capture only the output, and don't mind the error stream going to the terminal, you can use the much simpler code presented in getCommandOutput2. If you want to suppress the error stream altogether, that's easy, too. You can append 2>/dev/null to the command. For example:

ls -1 2>/dev/null

Since Version 2.0, Python includes the os.popen4 function, which combines the output and error streams of the child process. However, the streams are combined in a potentially messy way, depending on how they are buffered in the child process, so this recipe can still help.

6.7.4 See Also

Documentation of the standard library modules os, popen2, fcntl, and select in the Library Reference.