Writing a Regular Expression for Multiline Patterns

Contributed by: David Beazley and Brian K. Jones

Difficulty level: Easy

Visit the Python Cookbook: Strings and Text Lab to test out this Shortcut in a live practice environment.

Problem

You’re trying to match a block of text using a regular expression, but you need the match to span multiple lines.

Solution

This problem typically arises in patterns that use the dot (.) to match any character but forget to account for the fact that it doesn’t match newlines. For example, suppose you are trying to match C-style comments:

>>> comment = re.compile(r'/\*(.*?)\*/')
>>> text1 = '/* this is a comment */'
>>> text2 = '''/* this is a
...               multiline comment */
... '''
>>>
>>> comment.findall(text1)
[' this is a comment ']
>>> comment.findall(text2)
[]
>>>

To fix the problem, you can add support for newlines. For example:

>>> comment = re.compile(r'/\*((?:.|\n)*?)\*/')
>>> comment.findall(text2)
[' this is a\n              multiline comment ']
>>>

In this pattern, (?:.|\n) specifies a noncapture group (i.e., it defines a group for the purposes of matching, but that group is not captured separately or numbered).

Discussion

The re.compile() function accepts a flag, re.DOTALL, which is useful here. It makes the . in a regular expression match all characters, including newlines. For example:

>>> comment = re.compile(r'/\*(.*?)\*/', re.DOTALL)
>>> comment.findall(text2)
[' this is a\n              multiline comment ']

Using the re.DOTALL flag works fine for simple ...