pattern = re.compile('<div>(<p>.*?</p>)</div>', flags=re.DOTALL)

----------------------------------------------------------------

# This works as expected in the following case:

print(re.sub(pattern, '\\1',
             '<div><p>foo</p></div>\n'
             '<div><p>bar</p>123456789</div>\n'))

# which outputs:

<p>foo</p>
<div><p>bar</p>123456789</div>

----------------------------------------------------------------

# However, it does NOT work as I expect in this case:

print(re.sub(pattern, '\\1',
             '<div><p>foo</p>123456789</div>\n'
             '<div><p>bar</p></div>\n'))

# actual output:

<p>foo</p>123456789</div>
<div><p>bar</p>

# expected output:

<div><p>foo</p>123456789</div>
<p>bar</p>

----------------------------------------------------------------

It seems that pattern matching/substitution iterations only go haywire once the matching iteration immediately prior to it turned out not to be a match. Maybe some internal variable is not cleaned up properly in an edge(?) case triggered by the example above?

It works correctly. It finds a substring that starts with '<div><p>' and ends with '</p></div>' from left to right. The leftmost found substring starts from index 0 and ends before the final '\n'. Overlapped substrings are not found.

I don't understand... Isn't the "?" in ".*?" supposed to make the ".*" matching non-greedy, hence matching the first "</p>" rather than the last "</p>"?

Yes, it is non-greedy. But it needs matching not just '</p>', but '</p></div>'. After finding the first '</p>' it doesn't see the '</div>' after it and continue searching until found '</p></div>'.

I now see you're right of course. Not a bug after all. Thank you.

I mistakenly assumed that the group boundary ")" would delimit the end of the non-greedy match group. I.e., ".*?</p>" versus ".*?</p></div>".

I don't see a way to accomplish the "even less greedy" variant I'm looking for though...

Doh! This has a really easy solution, doesn't it; just replace "." with "[^<]": re.compile('<div>(<p>[^<]*?</p>)</div>', flags=re.DOTALL).

Sorry about the noise.

Atomic groups can help you: '<div>((?><p>.*?</p>))</div>'.

But this feature is not supported in the re module yet (see issue433030). You can use the third-party regex module which is compatible with the re module and supports atomic grouping.

>>> import regex as re
>>> pattern = re.compile('<div>((?><p>.*?</p>))</div>', flags=re.DOTALL)
>>> print(re.sub(pattern, '\\1',
...              '<div><p>foo</p>123456789</div>\n'
...              '<div><p>bar</p></div>\n'))
<div><p>foo</p>123456789</div>
<p>bar</p>