O'Reilly logo

Python Cookbook, 2nd Edition by David Ascher, Anna Ravenscroft, Alex Martelli

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

2.5. Counting Lines in a File

Credit: Luther Blissett

Problem

You need to compute the number of lines in a file.

Solution

The simplest approach for reasonably sized files is to read the file as a list of lines, so that the count of lines is the length of the list. If the file's path is in a string bound to a variable named thefilepath, all the code you need to implement this approach is:

count = len(open(thefilepath, 'rU').readlines( ))

For a truly huge file, however, this simple approach may be very slow or even fail to work. If you have to worry about humongous files, a loop on the file always works:

count = -1
for count, line in enumerate(open(thefilepath, 'rU')):
    pass
count += 1

A tricky alternative, potentially faster for truly humongous files, for when the line terminator is '\n' (or has '\n' as a substring, as happens on Windows):

count = 0
thefile = open(thefilepath, 'rb')
while True:
    buffer = thefile.read(8192*1024)
    if not buffer:
        break
    count += buffer.count('\n')
thefile.close( )

The 'rb' argument to open is necessary if you're after speed—without that argument, this snippet might be very slow on Windows.

Discussion

When an external program counts a file's lines, such as wc -l on Unix-like platforms, you can of course choose to use that (e.g., via os.popen). However, it's generally simpler, faster, and more portable to do the line-counting in your own program. You can rely on almost all text files having a reasonable size, so that reading the whole file into memory at once is feasible. ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required