Author ned.deily
Recipients Alex.Burka, benjamin.peterson, ezio.melotti, georg.brandl, mrabarnett, ned.deily, samueljohn, serhiy.storchaka
Date 2013-05-27.06:01:08
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1369634470.99.0.834973791828.issue18050@psf.upfronthosting.co.za>
In-reply-to
Content
After spending some time investigating this issue, I believe that potential upgrade compatibility issues have been introduced by the changes for Issue13169.  How critical they are and, in particular, whether they violate our implicit promises of maintenance (point) release compatibility are questions for discussion.

The signature of the problem is "ImportError: cannot import name MAXREPEAT" as a result of an attempt to import re or a module that itself imports re:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nad/issue18050/u/lib/python2.7/re.py", line 105, in <module>
    import sre_compile
  File "/home/nad/issue18050/u/lib/python2.7/sre_compile.py", line 14, in <module>
    import sre_parse
  File "/home/nad/issue18050/u/lib/python2.7/sre_parse.py", line 17, in <module>
    from sre_constants import *
  File "/home/nad/issue18050/u/lib/python2.7/sre_constants.py", line 18, in <module>
    from _sre import MAXREPEAT
ImportError: cannot import name MAXREPEAT

The changes for Issue13169 moved the definition of MAXREPEAT into C code and then added an import of the new C constant into Lib/sre_constants.py to continue to provide sre_constants.MAXREPEAT for third-party modules that have been using it.  As long as the versions of the Python interpreter and the standard library Python files (sys.prefix/lib/pythonX.Y) remain in sync, there is not a problem.  However, if a situation arises where a pre-13169 interpreter is used with a post-13169 standard library, the "cannot import name MAXREPEAT" ImportError will occur.  I have found at least two situations where this can happen:

1. when a C application has statically embedded a pre-13169 interpreter and the standard library pointed to by its sys.prefix gets upgraded to a post-13169 version.  The interpreter then crashes during initialization in Lib/site.py which imports re in both Python 2 and 3 (for different purposes).

2. when a virtualenv created with a pre-13169 non-shared interpreter is used with an upgraded post-13169 standard library.  In this case, the interpreter makes it past initialization because virtualenv (at least, the current version) creates a modified site.py in the virtualenv lib/pythonX.Y that happens to not import re.  However, the import error will occur on the first use of re.  Side note: 3.3 standard library pyvenv does not seem to have this problem since the created venv symlinks to the sys.prefix interpreter and libs rather than copying it, like virtualenv does.

Note that Pythons built with --enable-shared (or --enable-framework on OS X) generally will not have a problem as long as the shared libpythonX.Y and the standard library remain consistent.  That is, in both cases above, a Python upgrade will automatically cause both the embedded app and the virtualenv to run with the newer interpreter.  AFAICT, the problems will only be seen when using a non-shared Python.

I believe the upgrades affected by this problem are:

2.7 through 2.7.3 upgraded to 2.7.4 or 2.7.5
3.3.0 upgraded to 3.3.1 or 3.3.2
3.2 through 3.2.3 upgraded to 3.2.4 or 3.2.5 (unverified)

The problem should be fixable by applying a patch along the lines suggested by Samuel.  Regardless of whether this is a compatibility break or not, I think we should fix the problem because people are already running into it.  (Nosying the release managers for their input.)

While related, the root cause of the vim problem reported above is probably more complicated because, although it appears to embed a Python interpreter, the standard library used by the OS X system vim appears to depend on $PATH, apparently incorrect behavior in vim.  Unfortunately, OS X vim users on 10.8 (probably also on 10.7) may encounter this problem when they try to use :py if they install an updated version of Python 2.7, such as from python.org or a third-party distributor like Homebrew or MacPorts.  And, when vim crashes due to the import error, it leaves the terminal settings in an unusable state.  One user workaround might be to create a shell function or alias to tweak PATH before using vim to ensure /usr/bin/python2.7 is found first.  Or simply patch re.py in the upgraded Python.
History
Date User Action Args
2013-05-27 06:01:11ned.deilysetrecipients: + ned.deily, georg.brandl, benjamin.peterson, ezio.melotti, mrabarnett, samueljohn, serhiy.storchaka, Alex.Burka
2013-05-27 06:01:10ned.deilysetmessageid: <1369634470.99.0.834973791828.issue18050@psf.upfronthosting.co.za>
2013-05-27 06:01:10ned.deilylinkissue18050 messages
2013-05-27 06:01:08ned.deilycreate