While final testing Distutils changes for Issue13590, I noticed that a few tests were failing unexpectedly on 2.7.x but passing on 3.2.x.  Upon further investigation, I discovered that in 2.7 there are two copies of "customize_compiler", the function being modified in Issue13590.  It appears that at one point prior to the decision to make no feature changes to Distutils, that customize_compiler was moved from distutils.sysconfig to distutils.ccompiler (fa69e891edf4).  Then, when the decision was made to revert changes and proceed with a new Distutils2/packaging, feature changes were reverted in 2.7 but, for unexplained reasons, only part of the changes were reverted (ad22bc22dddc).  Perhaps a clue is in the commit message:
  "reverting partially distutils to its 2.6.x state so 2.7a4 looks more
   like the 2.7b1 in this. the whole revert will occur after a4 is tagged "

From the change logs, I see no evidence that the whole revert ever happened.  This has left 2.7 Distutils with two virtually identical versions of customize_compiler with some commands and tests calling one and others calling the other.  (In 3.2.x, the only version is in distutils.sysconfig.)  They *were* virtually identically until Issue13590 changed one and missed the other.

Patch to follow which completes the revert for customize_compiler and brings 2.7.x and 3.2.2 back into sync with respect to this.  This patch is needed immediately for the upcoming 2.7.3 release.  I strongly urge a more complete audit for other discrepancies between the 2.7.x and 3.2.x versions of Distutils; this isn't the first time we've been burned recently by the effects of the reverts in 2.7.

New changeset 6240ff5dfebe by Ned Deily in branch '2.7':
Issue #13994: Earler partial revert of Distutils enhancements in 2.7
http://hg.python.org/cpython/rev/6240ff5dfebe

With the release manager's concurrence, I've applied this patch prior to the freeze for 2.7.3.

FWIW I approve of the patch.

Quoting Ned:
> this isn't the first time we've been burned by the after effects of the 2.7 Distutils
> revert.  I really think we should go through and make sure there aren't other unnecessary
> discrepancies between the current state of 2.7.x and 3.2.x/default.
The revert indiscriminately reverted new features and bug and doc fixes, some of which have been fixed again later when rediscovered.  I never did a systematic review but I could.  One of the most painful difference between 2.7 and 3.x is the use of urllib vs. httplib in the register and upload commands, which does not let me port patches easily.

The patch broke egenix-mx-base, since it relies on the customize_compiler() being available in distutils.ccompiler:

https://www.egenix.com/mailman-archives/egenix-users/2012-April/114838.html

If you make such changes to dot releases, please make absolutely sure that when you move functions from one module to another, you keep backwards compatibility aliases around.

Sorry for not thinking about this.  I’ll be more careful.

Here's the quote from mxSetup.py:

    # distutils changed a lot in Python 2.7 due to many
    # distutils.sysconfig APIs having been moved to the new
    # (top-level) sysconfig module.
    from sysconfig import \
         get_config_h_filename, parse_config_h, get_path, \
         get_config_vars, get_python_version, get_platform

    # This API was moved from distutils.sysconfig to distutils.ccompiler
    # in Python 2.7
    from distutils.ccompiler import customize_compiler

So in 2.7 the function was moved from sysconfig to ccompiler (where it belongs), and now you're reverting the change in the third dot release.

Éric Araujo wrote:
> 
> Sorry for not thinking about this.  I’ll be more careful.

No need to be sorry; these things can happen.

What I don't understand is this line in the news section:

"Complete the revert back to only having one in distutils.sysconfig as
 7.12 +  is the case in 3.x."

Back when I discussed these changes with Tarek, we both agreed that
customize_compiler() is better placed into the ccompiler module
than the sysconfig module, so I think the one in the sysconfig
module should be replaced with a reference to the version in the
ccompile module - in both 2.7 and 3.x.

That's unfortunate.  But the documented location for customize_compiler is and, AFAIK, had always been in distutils.sysconfig.  It was an inadvertent consequence of the bad revert during the 2.7 development cycle that a second copy was made available in distutils.ccompiler.  That change was not supposed to be released in 2.7 and was never documented.  So I don't think there is anything that can or needs to be done as this point in Python itself.  Other opinions?

And to recap the history here, there was a change in direction for Distutils during the 2.7 development cycle, as decided at the 2010 language summit, in particular to revert feature changes in Distutils for 2.7 to its 2.6.x state and, going forward, "Distutils in Python will be feature-frozen".

http://mail.python.org/pipermail/python-dev/2010-March/098135.html

Ned Deily wrote:
> 
> Ned Deily <nad@acm.org> added the comment:
> 
> That's unfortunate.  But the documented location for customize_compiler is and, AFAIK, had always been in distutils.sysconfig.  It was an inadvertent consequence of the bad revert during the 2.7 development cycle that a second copy was made available in distutils.ccompiler.  That change was not supposed to be released in 2.7 and was never documented.  So I don't think there is anything that can or needs to be done as this point in Python itself.  Other opinions?

Excuse me, Ned, but that's not how we do approach dot releases in Python.

Regardless of whether the documentation was fixed or not, you cannot
simply remove a non-private function without making sure that at least
the import continues to work.

Ned Deily wrote:
> 
> And to recap the history here, there was a change in direction for Distutils during the 2.7 development cycle, as decided at the 2010 language summit, in particular to revert feature changes in Distutils for 2.7 to its 2.6.x state and, going forward, "Distutils in Python will be feature-frozen".
> 
> http://mail.python.org/pipermail/python-dev/2010-March/098135.html

I know that distutils development was stopped (even though I don't
consider that a good thing), but since the code changes were let
into the wild, we have to deal with it properly now.

Marc-Andre Lemburg wrote:
> 
>> Ned Deily <nad@acm.org> added the comment:
>>
>> That's unfortunate.  But the documented location for customize_compiler is and, AFAIK, had always been in distutils.sysconfig.  It was an inadvertent consequence of the bad revert during the 2.7 development cycle that a second copy was made available in distutils.ccompiler.  That change was not supposed to be released in 2.7 and was never documented.  So I don't think there is anything that can or needs to be done as this point in Python itself.  Other opinions?
> 
> Excuse me, Ned, but that's not how we do approach dot releases in Python.
> 
> Regardless of whether the documentation was fixed or not, you cannot
> simply remove a non-private function without making sure that at least
> the import continues to work.

Turns out, the "fix" broke all our packages for Python 2.7.3 and
I can hardly believe we're the only ones affected by this.

I agree that we should always try very hard not to break anything in point releases.  But I think it is fair to say that this is an unusual case.  Looking at the commit logs (and Tarek can correct me if I misread them), it appears the change that, among other things, moved customize_compiler was committed on 2010-01-23, in time for 2.7alpha3.  The language summit decision followed soon thereafter and the faulty partial revert happened on 2010-03-05, in time for 2.7alpha4.  So, if the revert had been completed as intended, the moved module should have only been in the wild for a little over a month.  Clearly you had a vested interest in making the change to your code base but it seems unlikely that anyone else would have gone to the trouble of changing their code since existing (2.6) code would have only been broken for that one small alpha window, 2.7 alpha3.  And considering that customize_compiler is probably not used by all that many packages to begin with it, I would think it is not unlikely that you *are* the only ones affected by it.

Nevertheless, what are the alternatives?  We could add a wrapper function into distutils.ccompiler that just calls the distutils.sysconfig version.  Here's a patch that attempts to do so. That should fix that breakage for the eGenix packages.  It would be great if you could test it.

It's up to the 2.7 release manager to decide what action to take, i.e. whether the patch is needed and, if so, how quickly to schedule a new release.  As a practical matter, regardless of whether the patch is applied in Python or not, I would assume that a faster solution for your end users would be to ship a version of the eGenix packages that reverts the changes(s) there.  By the way, it looks like you'll need to eventually do that anyway since the code in mxSetup.py incorrectly assumes that the corresponding changes were also made to Python 3.2.

ink it 
is not unlikely that you *are* the only ones affected by it.

With "in the wild" I'm referring to the function being released in
the ccompiler not only in alpha releases but also in the beta
releases, the 2.7, 2.7.1 and 2.7.2 release - in every release
since early in 2010.

We were unaware of the reversal of the changes by Tarek and
the way we coded things in mxSetup.py did not show that things
were removed again, simply because we support more than just
Python 2.7 and have proper fallback solutions for most things.

Only in this particular case, we were using different strategies
based on the Python version number and so there is no fallback.

> Nevertheless, what are the alternatives?  We could add a wrapper function into distutils.ccompiler that just calls the distutils.sysconfig version.  Here's a patch that attempts to do so. That should fix that breakage for the eGenix packages.  It would be great if you could test it.

The fix is easy: simply import the customize_compiler() API in
the ccompiler module to maintain compatibility with what had
already been release. No need to add a wrapper function,
a single

from distutils.sysconfig import customize_compiler()

in ccompile.py will do just fine.

> It's up to the 2.7 release manager to decide what action to take, i.e. whether the patch is needed and, if so, how quickly to schedule a new release.  As a practical matter, regardless of whether the patch is applied in Python or not, I would assume that a faster solution for your end users would be to ship a version of the eGenix packages that reverts the changes(s) there.  By the way, it looks like you'll need to eventually do that anyway since the code in mxSetup.py incorrectly assumes that the corresponding changes were also made to Python 3.2.

We don't support Python 3.x yet, so that's a non-issue at the moment.

But yes, we will have to release new patch level releases for all
our packages to get this fixed for our users.

[MAL]
> I know that distutils development was stopped (even though I don't consider that a good thing)
This is OT, but could you tell a bit more?  The freeze appears to me to have been a necessary decision due to the impossibility of making non-trivial changes without breaking setup scripts or projects such as mxSetup.

> but since the code changes were let into the wild, we have to deal with it properly now.
This I agree with.  Saturday or Sunday I will have the time to commit a fix and test.

Does this still need to block 2.7.4?

2.7.3 broke some setup scripts, it wouldn’t be bad to fix this in 2.7.4.  I’ll make time before RC.

It's not clear to me at all what needs to happen here, so I'm just going to have to drop it for 2.7.4.

Benjamin: what’s missing is an import.

Is it too late?  I changed home today and that took a lot of my time and energy this week, so I may have missed 2.7.4.

There's a window above about 12 hours now.

On it.

New changeset d4dd297fedb1 by Éric Araujo in branch '2.7':
Add alias to restore 2.7.2 compatibility for setup scripts (#13994).
http://hg.python.org/cpython/rev/d4dd297fedb1

Benjamin, I don’t know if you are using a release clone or if you will just tag the main repo, so leaving this open for now.

Not yet tagged.

Thanks for getting this in, Eric !