Python Cookbook

Capturing the Output and Error Streams from a Unix Shell Command

Credit: Brent Burley

Problem

You need to run an external process in a Unix-like environment and capture both the output and error streams from the external process.

Solution

The popen2 module lets you capture both streams, but you also need help from fcntl to make the streams nonblocking and thus avoid deadlocks:

import os, popen2, fcntl, FCNTL, select def makeNonBlocking(fd): fl = fcntl.fcntl(fd, FCNTL.F_GETFL) try: fcntl.fcntl(fd, FCNTL.F_SETFL, fl | FCNTL.O_NDELAY) except AttributeError: fcntl.fcntl(fd, FCNTL.F_SETFL, fl | FCNTL.FNDELAY) def getCommandOutput(command): child = popen2.Popen3(command, 1) # Capture stdout and stderr from command child.tochild.close( ) # don't need to write to child's stdin outfile = child.fromchild outfd = outfile.fileno( ) errfile = child.childerr errfd = errfile.fileno( ) makeNonBlocking(outfd) # Don't deadlock! Make fd's nonblocking. makeNonBlocking(errfd) outdata = errdata = '' outeof = erreof = 0 while 1: ready = select.select([outfd,errfd],[],[]) # Wait for input if outfd in ready[0]: outchunk = outfile.read( ) if outchunk == '': outeof = 1 outdata = outdata + outchunk if errfd in ready[0]: errchunk = errfile.read( ) if errchunk == '': erreof = 1 errdata = errdata + errchunk if outeof and erreof: break select.select([],[],[],.1) # Allow a little time for buffers to fill err = child.wait( ) if err != 0: raise RuntimeError, '%s failed with exit code %d\n%s' % ( command, err, errdata) ...