I am experiencing and issue with the following regex when using finditer.

(?=<(?P<tag>\w+)/?>(?:(?P<text>.+?)</(?P=tag)>)?)", "<test><foo2/></test>

(I know it's not the best method of dealing with HTML, and this is a simplified version)

For example:

[m.groupdict() for m in re.finditer(r"(?=<(?P<tag>\w+)/?>(?:(?P<text>.+?)</(?P=tag)>)?)", "<test><foo2/></test>")]

In Python 2.7, 3.5, and 3.6 it returns

[{'tag': 'test', 'text': '<foo2/>'}, {'tag': 'foo2', 'text': None}]

But starting with 3.7 it returns

[{'tag': 'test', 'text': '<foo2/>'}, {'tag': 'foo2', 'text': '<foo2/>'}]

The "text" group appears to be a copy of the previous "text" group.

Some other examples:

"<test>Hello</test><foo/>" => [{'tag': 'test', 'text': 'Hello'}, {'tag': 'foo', 'text': 'Hello'}] (expected: [{'tag': 'test', 'text': 'Hello'}, {'tag': 'foo', 'text': None}])
"<test>Hello</test><foo/><foo/>" => [{'tag': 'test', 'text': 'Hello'}, {'tag': 'foo', 'text': 'Hello'}, {'tag': 'foo', 'text': None}] (expected: [{'tag': 'test', 'text': 'Hello'}, {'tag': 'foo', 'text': None}, {'tag': 'foo', 'text': None}])

➜ cpython git:(70d56fb) ✗ ./python.exe
Python 3.7.0a2+ (tags/v3.7.0a2-341-g70d56fb525:70d56fb525, Jul 31 2018, 21:58:10)
[Clang 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>
➜ cpython git:(70d56fb) ✗ ./python.exe -c 'import re; print([m.groupdict() for m in re.finditer(r"(?=<(?P<tag>\w+)/?>(?:(?P<text>.+?)</(?P=tag)>)?)", "<test><foo2/></test>")])'
[{'tag': 'test', 'text': '<foo2/>'}, {'tag': 'foo2', 'text': '<foo2/>'}]

➜ cpython git:(e69fbb6) ✗ ./python.exe
Python 3.7.0a2+ (tags/v3.7.0a2-340-ge69fbb6a56:e69fbb6a56, Jul 31 2018, 22:12:06)
[Clang 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>
➜ cpython git:(e69fbb6) ✗ ./python.exe -c 'import re; print([m.groupdict() for m in re.finditer(r"(?=<(?P<tag>\w+)/?>(?:(?P<text>.+?)</(?P=tag)>)?)", "<test><foo2/></test>")])'
[{'tag': 'test', 'text': '<foo2/>'}, {'tag': 'foo2', 'text': None}]

Does this have something to do with 70d56fb(bpo-25054, bpo-1647489) ?

Thanks

This bug generates wrong results silently, so I suggest mark it as release blocker for 3.7.1

Simplify the test-case, it seem the state is not reset properly.

Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47)
>>> import re
>>> re.findall(r"(?=(<\w+>)(<\w+>)?)", "<aaa><bbb>")
[('<aaa>', '<bbb>'), ('<bbb>', '')]

Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28)
>>> import re
>>> re.findall(r"(?=(<\w+>)(<\w+>)?)", "<aaa><bbb>")
[('<aaa>', '<bbb>'), ('<bbb>', '<bbb>')]

I tried to fix it, feel free to create a new PR if you don't want this one.

PR11546 has a small question, should state->data_stack be dealloced as well?

FYI, function state_reset(SRE_STATE* state) in file _sre.c:

Serhiy Storchaka lost his sight.
Please stop any work and rest, because your left eye will have more burden, and your mental burden will make it worse.
Go to hospital ASAP.

If any other core developer want to review this patch, I would like to give a detailed explanation, the logic is not very compilcated.

Original post's bug was introduced in Python 3.7.0

When investigate the code, I found another bug about capturing groups. This bug exists since very early version.
regex module doesn't have this bug.

Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (Intel)] on win32
>>> import re
>>> re.search(r"\b(?=(\t)|(x))x", "a\tx").groups()
('', 'x')

Expected result: (None, 'x')

Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
>>> import regex
>>> regex.search(r"\b(?=(\t)|(x))x", "a\tx").groups()
(None, 'x')

New changeset 4a7f44a by Serhiy Storchaka (animalize) in branch 'master':
bpo-34294: re module, fix wrong capturing groups in rare cases. (GH-11546)
4a7f44a

New changeset 0e379d4 by Miss Islington (bot) in branch '3.7':
bpo-34294: re module, fix wrong capturing groups in rare cases. (GH-11546)
0e379d4

Thank you for your PR Ma Lin!