On an up to date Fedora 16:

== CPython 3.3.0a1+ (default:ddddd528b2d2+, Mar 29 2012, 18:04:26) [GCC 4.6.3 20120306 (Red Hat 4.6.3-2)]
==   Linux-3.3.0-4.fc16.x86_64-x86_64-with-fedora-16-Verne little-endian
==   /home/ross/src/cpythondev/temp/build/test_python_21786
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1)
[1/1] test_distutils
test_byte_compile (distutils.tests.test_install_lib.InstallLibTestCase) ... ok
test_dont_write_bytecode (distutils.tests.test_install_lib.InstallLibTestCase) ... ok
test_finalize_options (distutils.tests.test_install_lib.InstallLibTestCase) ... ok
test_get_inputs (distutils.tests.test_install_lib.InstallLibTestCase) ... ok
test_get_outputs (distutils.tests.test_install_lib.InstallLibTestCase) ... ok
test_no_compiler (distutils.tests.test_msvc9compiler.msvc9compilerTestCase) ... skipped 'These tests are only for win32'
test_reg_class (distutils.tests.test_msvc9compiler.msvc9compilerTestCase) ... skipped 'These tests are only for win32'
test_remove_entire_manifest (distutils.tests.test_msvc9compiler.msvc9compilerTestCase) ... skipped 'These tests are only for win32'
test_remove_visual_c_ref (distutils.tests.test_msvc9compiler.msvc9compilerTestCase) ... skipped 'These tests are only for win32'
test_nt_quote_args (distutils.tests.test_spawn.SpawnTestCase) ... ok
test_spawn (distutils.tests.test_spawn.SpawnTestCase) ... ok
test_announce (distutils.tests.test_dist.DistributionTestCase) ... ok
test_command_packages_cmdline (distutils.tests.test_dist.DistributionTestCase) ... ok
test_command_packages_configfile (distutils.tests.test_dist.DistributionTestCase) ... ok
test_command_packages_unspecified (distutils.tests.test_dist.DistributionTestCase) ... ok
test_empty_options (distutils.tests.test_dist.DistributionTestCase) ... ok
test_finalize_options (distutils.tests.test_dist.DistributionTestCase) ... ok
test_get_command_packages (distutils.tests.test_dist.DistributionTestCase) ... ok
test_classifier (distutils.tests.test_dist.MetadataTestCase) ... ok
test_custom_pydistutils (distutils.tests.test_dist.MetadataTestCase) ... ok
test_download_url (distutils.tests.test_dist.MetadataTestCase) ... ok
test_fix_help_options (distutils.tests.test_dist.MetadataTestCase) ... ok
test_long_description (distutils.tests.test_dist.MetadataTestCase) ... ok
test_obsoletes (distutils.tests.test_dist.MetadataTestCase) ... ok
test_obsoletes_illegal (distutils.tests.test_dist.MetadataTestCase) ... ok
test_provides (distutils.tests.test_dist.MetadataTestCase) ... ok
test_provides_illegal (distutils.tests.test_dist.MetadataTestCase) ... ok
test_requires (distutils.tests.test_dist.MetadataTestCase) ... ok
test_requires_illegal (distutils.tests.test_dist.MetadataTestCase) ... ok
test_show_help (distutils.tests.test_dist.MetadataTestCase) ... ok
test_simple_metadata (distutils.tests.test_dist.MetadataTestCase) ... ok
test_simple_run (distutils.tests.test_install_headers.InstallHeadersTestCase) ... ok
test_finalize_options (distutils.tests.test_upload.uploadTestCase) ... ok
test_saved_password (distutils.tests.test_upload.uploadTestCase) ... ok
test_server_empty_registration (distutils.tests.test_upload.uploadTestCase) ... ok
test_server_registration (distutils.tests.test_upload.uploadTestCase) ... ok
test_upload (distutils.tests.test_upload.uploadTestCase) ... ok
test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_check_extensions_list (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_compiler_option (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_deployment_target_default (distutils.tests.test_build_ext.BuildExtTestCase) ... skipped 'test only relevant for MacOSX'
test_deployment_target_higher_ok (distutils.tests.test_build_ext.BuildExtTestCase) ... skipped 'test only relevant for MacOSX'
test_deployment_target_too_low (distutils.tests.test_build_ext.BuildExtTestCase) ... skipped 'test only relevant for MacOSX'
test_ext_fullpath (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_finalize_options (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_get_source_files (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_optional_extension (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_solaris_enable_shared (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_user_site (distutils.tests.test_build_ext.BuildExtTestCase) ... ok
test_build (distutils.tests.test_build_scripts.BuildScriptsTestCase) ... ok
test_default_settings (distutils.tests.test_build_scripts.BuildScriptsTestCase) ... ok
test_version_int (distutils.tests.test_build_scripts.BuildScriptsTestCase) ... ok
test_debug_mode (distutils.tests.test_core.CoreTestCase) ... ok
test_run_setup_provides_file (distutils.tests.test_core.CoreTestCase) ... ok
test_run_setup_uses_current_dir (distutils.tests.test_core.CoreTestCase) ... ok
test_extension_init (distutils.tests.test_extension.ExtensionTestCase) ... ok
test_read_setup_file (distutils.tests.test_extension.ExtensionTestCase) ... ok
test_check_config_h (distutils.tests.test_cygwinccompiler.CygwinCCompilerTestCase) ... ok
test_get_msvcr (distutils.tests.test_cygwinccompiler.CygwinCCompilerTestCase) ... ok
test_get_versions (distutils.tests.test_cygwinccompiler.CygwinCCompilerTestCase) ... ok
test_cmp (distutils.tests.test_version.VersionTestCase) ... ok
test_cmp_strict (distutils.tests.test_version.VersionTestCase) ... ok
test_prerelease (distutils.tests.test_version.VersionTestCase) ... ok
test_byte_compile (distutils.tests.test_build_py.BuildPyTestCase) ... ok
test_byte_compile_optimized (distutils.tests.test_build_py.BuildPyTestCase) ... ok
test_dont_write_bytecode (distutils.tests.test_build_py.BuildPyTestCase) ... ok
test_empty_package_dir (distutils.tests.test_build_py.BuildPyTestCase) ... ok
test_package_data (distutils.tests.test_build_py.BuildPyTestCase) ... ok
test_no_optimize_flag (distutils.tests.test_bdist_rpm.BuildRpmTestCase) ... error: Installed (but unpackaged) file(s) found:
   /usr/local/lib/python3.3/site-packages/foo.pyc
   /usr/local/lib/python3.3/site-packages/foo.pyo
    Installed (but unpackaged) file(s) found:
   /usr/local/lib/python3.3/site-packages/foo.pyc
   /usr/local/lib/python3.3/site-packages/foo.pyo
ERROR
test_quiet (distutils.tests.test_bdist_rpm.BuildRpmTestCase) ... error: Installed (but unpackaged) file(s) found:
   /usr/local/lib/python3.3/site-packages/foo.pyc
   /usr/local/lib/python3.3/site-packages/foo.pyo
    Installed (but unpackaged) file(s) found:
   /usr/local/lib/python3.3/site-packages/foo.pyc
   /usr/local/lib/python3.3/site-packages/foo.pyo
ERROR
test_clean (distutils.tests.test_config_cmd.ConfigTestCase) ... ok
test_dump_file (distutils.tests.test_config_cmd.ConfigTestCase) ... ok
test_finalize_options (distutils.tests.test_config_cmd.ConfigTestCase) ... ok
test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase) ... ok
test_class (distutils.tests.test_text_file.TextFileTestCase) ... ok
test_move_file_verbosity (distutils.tests.test_file_util.FileUtilTestCase) ... ok
test_check_archive_formats (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_compress_deprecated (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'The compress program is required'
test_make_archive (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_make_archive_cwd (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_make_tarball (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_make_tarball_extended (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_make_tarball_latin1 (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_make_zipfile (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_make_zipfile_no_zlib (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_tarfile_vs_tar (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok
test_minimal (distutils.tests.test_bdist_msi.BDistMSITestCase) ... skipped 'these tests require Windows'
test_build_libraries (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok
test_check_library_dist (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok
test_finalize_options (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok
test_get_source_files (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok
test_run (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok
test_add_defaults (distutils.tests.test_sdist.SDistTestCase) ... ok
test_check_metadata_deprecated (distutils.tests.test_sdist.SDistTestCase) ... ok
test_finalize_options (distutils.tests.test_sdist.SDistTestCase) ... ok
test_get_file_list (distutils.tests.test_sdist.SDistTestCase) ... ok
test_invalid_template_unknown_command (distutils.tests.test_sdist.SDistTestCase) ... ok
test_invalid_template_wrong_arguments (distutils.tests.test_sdist.SDistTestCase) ... ok
test_invalid_template_wrong_path (distutils.tests.test_sdist.SDistTestCase) ... skipped 'test relevant for Windows only'
test_make_distribution (distutils.tests.test_sdist.SDistTestCase) ... ok
test_manifest_comments (distutils.tests.test_sdist.SDistTestCase) ... ok
test_manifest_marker (distutils.tests.test_sdist.SDistTestCase) ... ok
test_manual_manifest (distutils.tests.test_sdist.SDistTestCase) ... ok
test_metadata_check_option (distutils.tests.test_sdist.SDistTestCase) ... ok
test_prune_file_list (distutils.tests.test_sdist.SDistTestCase) ... ok
test_server_empty_registration (distutils.tests.test_sdist.SDistTestCase) ... ok
test_server_registration (distutils.tests.test_sdist.SDistTestCase) ... ok
test_show_formats (distutils.tests.test_sdist.SDistTestCase) ... ok
test_customize_compiler (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok
test_get_config_h_filename (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok
test_get_config_vars (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok
test_get_python_inc (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok
test_get_python_lib (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok
test_parse_makefile_base (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok
test_parse_makefile_literal_dollar (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok
test_sysconfig_module (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok
VersionPredicate (distutils.versionpredicate)
Doctest: distutils.versionpredicate.VersionPredicate ... ok
split_provision (distutils.versionpredicate)
Doctest: distutils.versionpredicate.split_provision ... ok
test_default_settings (distutils.tests.test_install_scripts.InstallScriptsTestCase) ... ok
test_installation (distutils.tests.test_install_scripts.InstallScriptsTestCase) ... ok
test_check_metadata_deprecated (distutils.tests.test_register.RegisterTestCase) ... ok
test_create_pypirc (distutils.tests.test_register.RegisterTestCase) ... ok
test_password_not_in_file (distutils.tests.test_register.RegisterTestCase) ... ok
test_password_reset (distutils.tests.test_register.RegisterTestCase) ... ok
test_registering (distutils.tests.test_register.RegisterTestCase) ... ok
test_server_empty_registration (distutils.tests.test_register.RegisterTestCase) ... ok
test_server_registration (distutils.tests.test_register.RegisterTestCase) ... ok
test_strict (distutils.tests.test_register.RegisterTestCase) ... ok
test_copy_tree_verbosity (distutils.tests.test_dir_util.DirUtilTestCase) ... ok
test_create_tree_verbosity (distutils.tests.test_dir_util.DirUtilTestCase) ... ok
test_ensure_relative (distutils.tests.test_dir_util.DirUtilTestCase) ... ok
test_mkpath_remove_tree_verbosity (distutils.tests.test_dir_util.DirUtilTestCase) ... ok
test_mkpath_with_custom_mode (distutils.tests.test_dir_util.DirUtilTestCase) ... ok
test_formats (distutils.tests.test_bdist.BuildTestCase) ... ok
test_skip_build (distutils.tests.test_bdist.BuildTestCase) ... ok
test_simple_run (distutils.tests.test_install_data.InstallDataTestCase) ... ok
test_simple_built (distutils.tests.test_bdist_dumb.BuildDumbTestCase) ... ok
test_change_root (distutils.tests.test_util.UtilTestCase) ... ok
test_check_environ (distutils.tests.test_util.UtilTestCase) ... ok
test_convert_path (distutils.tests.test_util.UtilTestCase) ... ok
test_dont_write_bytecode (distutils.tests.test_util.UtilTestCase) ... ok
test_get_platform (distutils.tests.test_util.UtilTestCase) ... ok
test_rfc822_escape (distutils.tests.test_util.UtilTestCase) ... ok
test_split_quoted (distutils.tests.test_util.UtilTestCase) ... ok
test_strtobool (distutils.tests.test_util.UtilTestCase) ... ok
test_runtime_libdir_option (distutils.tests.test_unixccompiler.UnixCCompilerTestCase) ... ok
test_debug_print (distutils.tests.test_filelist.FileListTestCase) ... ok
test_exclude_pattern (distutils.tests.test_filelist.FileListTestCase) ... ok
test_glob_to_re (distutils.tests.test_filelist.FileListTestCase) ... ok
test_include_pattern (distutils.tests.test_filelist.FileListTestCase) ... ok
test_process_template (distutils.tests.test_filelist.FileListTestCase) ... ok
test_process_template_line (distutils.tests.test_filelist.FileListTestCase) ... ok
test_remove_duplicates (distutils.tests.test_filelist.FileListTestCase) ... ok
test_set_allfiles (distutils.tests.test_filelist.FileListTestCase) ... ok
test_translate_pattern (distutils.tests.test_filelist.FileListTestCase) ... ok
test_simple_run (distutils.tests.test_clean.cleanTestCase) ... ok
test_finalize_options (distutils.tests.test_build.BuildTestCase) ... ok
test_get_exe_bytes (distutils.tests.test_bdist_wininst.BuildWinInstTestCase) ... ok
test_check_all (distutils.tests.test_check.CheckTestCase) ... ok
test_check_document (distutils.tests.test_check.CheckTestCase) ... ok
test_check_metadata (distutils.tests.test_check.CheckTestCase) ... ok
test_check_restructuredtext (distutils.tests.test_check.CheckTestCase) ... ok
test_debug_print (distutils.tests.test_cmd.CommandTestCase) ... ok
test_dump_options (distutils.tests.test_cmd.CommandTestCase) ... ok
test_ensure_dirname (distutils.tests.test_cmd.CommandTestCase) ... ok
test_ensure_filename (distutils.tests.test_cmd.CommandTestCase) ... ok
test_ensure_string (distutils.tests.test_cmd.CommandTestCase) ... ok
test_ensure_string_list (distutils.tests.test_cmd.CommandTestCase) ... ok
test_make_file (distutils.tests.test_cmd.CommandTestCase) ... ok
test_non_ascii (distutils.tests.test_log.TestLog) ... ok
test_server_empty_registration (distutils.tests.test_config.PyPIRCCommandTestCase) ... ok
test_server_registration (distutils.tests.test_config.PyPIRCCommandTestCase) ... ok
test_newer (distutils.tests.test_dep_util.DepUtilTestCase) ... ok
test_newer_group (distutils.tests.test_dep_util.DepUtilTestCase) ... ok
test_newer_pairwise (distutils.tests.test_dep_util.DepUtilTestCase) ... ok
test_debug_mode (distutils.tests.test_install.InstallTestCase) ... ok
test_finalize_options (distutils.tests.test_install.InstallTestCase) ... ok
test_handle_extra_path (distutils.tests.test_install.InstallTestCase) ... ok
test_home_installation_scheme (distutils.tests.test_install.InstallTestCase) ... ok
test_record (distutils.tests.test_install.InstallTestCase) ... ok
test_record_extensions (distutils.tests.test_install.InstallTestCase) ... ok
test_user_site (distutils.tests.test_install.InstallTestCase) ... test test_distutils failed
[?1034hok

======================================================================
ERROR: test_no_optimize_flag (distutils.tests.test_bdist_rpm.BuildRpmTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ross/src/cpythondev/temp/Lib/distutils/tests/test_bdist_rpm.py", line 125, in test_no_optimize_flag
    cmd.run()
  File "/home/ross/src/cpythondev/temp/Lib/distutils/command/bdist_rpm.py", line 365, in run
    self.spawn(rpm_cmd)
  File "/home/ross/src/cpythondev/temp/Lib/distutils/cmd.py", line 366, in spawn
    spawn(cmd, search_path, dry_run=self.dry_run)
  File "/home/ross/src/cpythondev/temp/Lib/distutils/spawn.py", line 32, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/ross/src/cpythondev/temp/Lib/distutils/spawn.py", line 163, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'rpmbuild' failed with exit status 1

======================================================================
ERROR: test_quiet (distutils.tests.test_bdist_rpm.BuildRpmTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ross/src/cpythondev/temp/Lib/distutils/tests/test_bdist_rpm.py", line 81, in test_quiet
    cmd.run()
  File "/home/ross/src/cpythondev/temp/Lib/distutils/command/bdist_rpm.py", line 365, in run
    self.spawn(rpm_cmd)
  File "/home/ross/src/cpythondev/temp/Lib/distutils/cmd.py", line 366, in spawn
    spawn(cmd, search_path, dry_run=self.dry_run)
  File "/home/ross/src/cpythondev/temp/Lib/distutils/spawn.py", line 32, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/ross/src/cpythondev/temp/Lib/distutils/spawn.py", line 163, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'rpmbuild' failed with exit status 1

----------------------------------------------------------------------
Ran 183 tests in 1.889s

FAILED (errors=2, skipped=10)
1 test failed:
    test_distutils

The output of running rpmbuild from bash:
"""
$ rpmbuild
RPM version 4.9.1.2
Copyright (C) 1998-2002 - Red Hat, Inc.
This program may be freely redistributed under the terms of the GNU GPL

Usage: rpmbuild [-v?] [-bp] [-bc] [-bi] [-bl] [-ba] [-bb] [-bs] [-tp]
        [-tc] [-ti] [-ta] [-tb] [-ts] [--rebuild] [--recompile]
        [--buildroot=DIRECTORY] [--clean] [--nobuild] [--nodeps]
        [--nodirtokens] [--rmsource] [--rmspec] [--short-circuit]
        [--target=CPU-VENDOR-OS] [-D|--define 'MACRO EXPR'] [-E|--eval 'EXPR']
        [--macros=<FILE:...>] [--nodigest] [--nosignature]
        [--rcfile=<FILE:...>] [-r|--root ROOT] [--dbpath=DIRECTORY]
        [--querytags] [--showrc] [--quiet] [-v|--verbose] [--version]
        [-?|--help] [--usage] [--with=<option>] [--without=<option>]
        [--buildpolicy=<policy>] [--sign]
"""

Related to #1533164.  Could you bisect the problem to see if it’s related to old commits or my changes for pycache directories?

The first bad revision is:                                                      
changeset:   72818:27a36b05caed
branch:      3.2
user:        Éric Araujo <merwok@netwok.org>
date:        Sat Oct 08 00:34:13 2011 +0200
summary:     Fix distutils byte-compilation to comply with PEP 3147 (#11254).


Cheers

This failure is also affecting the new RHEL-6 buildbot I'm attempting to bring online.

Failure on trunk:
http://www.python.org/dev/buildbot/all/builders/x86%20RHEL%206%203.x/builds/12/steps/test/logs/stdio

Failure on 3.2:
http://www.python.org/dev/buildbot/all/builders/x86%20RHEL%206%203.2/builds/1/steps/test/logs/stdio

Seems like it may be an old problem come back to life:
https://www.redhat.com/archives/rhl-devel-list/2009-January/msg00389.html

(The rh-devel-list link is interesting more for the problem description than it is for the suggested workaround)

I'm wondering if there may be a deeper problem here: how certain are we that bdist_rpm isn't using the system Python to handle the byte compilation step? It would explain why the files are still being generated in the old locations.

And, in practical terms, who uses RPM to package Python software for any version of Python other than the system Python?

To answer my last question: plenty of people. Even within Fedora itself there are parallel python 2 and python 3 RPM stacks.

It occurs to me there's a way to check my theory: if we update the failing test to explicitly check the magic cookie in at least one of the precompiled pyc files (rather than just expecting the files' existence), then it should also start failing on the 2.7 RHEL 6 buildbot.

The most likely cause is that some code was updated to use pycache directories but other code that computes paths still uses the old way; this happened for bdist_wininst too.  I’ll have a look at the code and reply to your questions tomorrow.  I’ll also add links to another bug where I suggest to print the full command when distutils.spawn fails and DISTUTILS_DEBUG is set and another bug about rpm vs. rpmbuild.

this-job-thing-really-takes-much-of-my-Python-time-ly yours

> I'm wondering if there may be a deeper problem here: how certain are we that bdist_rpm isn't using the system Python
> to handle the byte compilation step? It would explain why the files are still being generated in the old locations.

I’ve had a quick look at bdist_rpm and found nothing suspect (like filename + 'c').  You may have the right idea: There are --python and --fix-python options for this command that decide what executable will be used to run setup.py, so maybe the system Python is used.  The default is 'python' instead of 'python3' though, so if that’s the cause I don’t understand how it worked at all.

> It occurs to me there's a way to check my theory: if we update the failing test to explicitly check the magic cookie in at
> least one of the precompiled pyc files (rather than just expecting the files' existence), then it should also start failing
> on the 2.7 RHEL 6 buildbot.

I’ve been looking for a way to inspect byte-compiled files for some time; please share any ideas on #13473.

To help debug this, I could apply #11599.

The rpm vs. rpmbuild bug is #11122.

#5875 and #13307 are bugs similar to this one.

Antoine (added to nosy list) indicated he wasn't seeing the failure on a Mageia system, so that's another point in favour of a Fedora/RHEL specific problem.

Also added Dave Malcolm as the Fedora Python package maintainer.

Applying #11599 to get the full failing command line sounds worthwhile, but #11122 shouldn't be related, since the RHEL buildbot definitely has rpmbuild available.

As a post-processing step, rpmbuild will attempt to byte-compile any .py files it encounters, and the results must be listed in the %files manifest. [1]

This is done by the script brp-python-bytecompile, which uses the compileall module.  However, my guess is that it's not using the correct version of python when invoking "compileall", which would explain why it's using the pre-PEP3147 location for the .pyc/.pyo files.

Can you run "file" on the .pyc files and confirm which version of Python they're bytecode for?  My guess is that it's bytecompiled them with /usr/bin/python, rather than your local build of python.

Some notes: In older versions of RPM, brp-python-bytecompile took a single optional argument: the python interpreter to use, defaulting to /usr/bin/python.  I generalized this to support multiple defaults when adding Python 3 support to Fedora: see https://bugzilla.redhat.com/show_bug.cgi?id=531117  That patch could be generalized to support /usr/local/lib.

[1] In Fedora we do this using "__os_install_post", which is defined
in /usr/lib/rpm/redhat/macros (from the redhat-rpm-config package),
which has the invocation of /usr/lib/rpm/brp-python-bytecompile   So it could be possible to override the python interpreter to use by redefining __os_install_post

__os_install_post is defined within /usr/lib/rpm/redhat/macros and contains this fragment:
    /usr/lib/rpm/brp-python-bytecompile %{__python} %{?_python_bytecompile_errors_terminate_build} \

Hence it will use %{__python} as the default when byte-compiling.

%__python has the default definition of /usr/bin/python, but this can be overridden, either in the spec file using:
  %global __python some_path_to_the_python_to_use
or in the command-line invocation of rpmbuild:
  rpmbuild --define="__python some_path_to_the_python_to_use"

You can use the --showrc option to rpmbuild to see all of the macro expansions it's using (lots of output).

So if it is indeed using the wrong python executable to do the bytecompiling, the above ought to fix it.  The simplest fix would probably be for bdist_rpm to add:
  --define="__python %s" % sys.executable
to the rpmbuild invocation, I think.

I tried the simple fix a couple of different ways on the RHEL6 buildbot. First by changing line 315 of Lib/distutils/command/bdist_rpm.py" to be:

    rpm_cmd = ['rpmbuild', '--define', '__python %s' % sys.executable]

And then a second time by adding the new "--define" after the "-ba" option. (changing the spawn command to show the full command on error indicating the new arguments *were* being passed in, though)

Neither worked - I still got the same error. Since I'm seeing the same fault on my Fedora system, I'll start poking around there instead (not sure when I'll get to it though).

The attached patch fixed the test for me on Fedora 16.

It was necessary for the `define` to be after the -ba switch.

I don't know why this wouldn't work on RHEL6 then...

New changeset 3d61e27cc570 by Nick Coghlan in branch '3.2':
Issue #14443: Tell rpmbuild to use the correct version of Python
http://hg.python.org/cpython/rev/3d61e27cc570

Applied to 3.2 and trunk (I forgot to mention the issue name in the merge commit message).

I tweaked Ross's patch slightly to:
1. Use self.python rather than sys.executable
2. Also default self.python to "python3" if neither --python nor --fix-python are specified (I don't think we currently test that, and I don't see any way we *could* test it from an uninstalled source build)

Setting to pending while I wait to see how the RHEL6 buildbot reacts. Since the patch definitely resolves the issue on Fedora, it's quite possible I just fumble-fingered the earlier manual testing. Alternatively, there may be more to this issue, since RHEL6 almost certainly has an older version of rpmbuild than F16.

Unfortunately, it seems like it's still failing on the RHEL 6 buildbot.

Dave, sending this one back in your direction. Is there a relevant difference between rpmbuild in RHEL6 and rpmbuild in current Fedora?

I'm able to reproduce this on a RHEL 6 box, and I did some investigating.  The stray .pyc files are indeed reported by "file" as "python 2.6 byte-compiled" so yes, it's using /usr/bin/python to byte-compile them

On RHEL 6, with redhat-rpm-config-9.0.3-33.el6.noarch, /usr/lib/rpm/redhat/macros defines os_install_post as:
%__os_install_post    \
    /usr/lib/rpm/redhat/brp-compress \
    %{!?__debug_package:/usr/lib/rpm/redhat/brp-strip %{__strip}} \
    /usr/lib/rpm/redhat/brp-strip-static-archive %{__strip} \
    /usr/lib/rpm/redhat/brp-strip-comment-note %{__strip} %{__objdump} \
    /usr/lib/rpm/brp-python-bytecompile \
    /usr/lib/rpm/redhat/brp-python-hardlink \
    %{!?__jar_repack:/usr/lib/rpm/redhat/brp-java-repack-jars} \
%{nil}

Note how in this definition, brp-python-bytecompile is passed in without any arguments (contrast with msg159396, which is from a later version of the macros).

Hence it unconditionally (and erroneously) uses /usr/bin/python to byte-compile any .py files found in the package payload.

The change to add %{__python} to the invocation of /usr/lib/rpm/brp-python-bytecompile appears to have been in redhat-rpm-config-9.1.0 for Fedora 13 (see https://bugzilla.redhat.com/show_bug.cgi?id=521141), whereas RHEL 6 has the earlier code.

It may be possible to work around this by providing an overridden definition of __os_install_post in the specfile.  We do this in the python26.spec for EPEL5; grep for "__os_install_post" within:
http://pkgs.fedoraproject.org/gitweb/?p=python26.git;a=blob;f=python26.spec;h=6b490b9b71f42c26b7d4ec4031685fb3230c5602;hb=refs/heads/el5

Now I'm wondering if the problem will also manifest itself on EL rebuilds like CentOS or Scientific Linux, since that will affect how specific we can get when it comes to implementing a workaround.

That is, overriding __os_install_post should definitely work, but the question then becomes how we decide that overriding it is *necessary*, such that:
- we don't bypass vendor hooks when they aren't broken (e.g. Fedora, hopefully a future version of RHEL)
- we *do* bypass them when we know they don't work properly for Python 3 projects (e.g. RHEL6, probably EL6 rebuilds)

I'm currently thinking of something like this:

# At the top of the bdist_rpm command file
try:
    # Support workaround for #14443 which affects some RPM based systems
    # such as RHEL6 (and probably derivatives)
    from rpm import expandMacro as _expand_rpm_macro
except ImportError:
    _expand_rpm_macro = None

# When generating the spec file
    if _expand_rpm_macro is not None:
        # We're on a system with a vendor installed rpm library
        # and at least RHEL6 tries to bytecompile Python modules
        # with the system Python instead of the running one
        problem = "brp-python-bytecompile  \\"
        fixed = "brp-python-bytecompile  %{__python} \\"
        vendor_hook = _expand_rpm_macro("%{__os_install_post}")
        fixed_hook = vendor_hook.replace(problem, fixed)
        if fixed_hook != vendor_hook:
            # Add fixed_hook to spec file template

Thoughts?

Ping.  I would like to have this test failure fixed for 3.3 rc.

The approach Nick suggests seems reasonable, but rpm.expandMacro isn't usable: there's no guarantee that the machine has Python bindings for rpm installed (especially not for the version of Python that we've just built).

However, we can get at the macro by shelling out to the "rpm" command:
   subprocess.getoutput('rpm --eval %{__os_install_post}')

I'm attaching a patch which does this; works on my RHEL 6.3 test box.

(with the patch, it also continues to work on a test Fedora 15 box)

New changeset a9a12ad553f0 by David Malcolm in branch 'default':
Issue #14443: ensure that brp-python-bytecompile is invoked with the correct
http://hg.python.org/cpython/rev/a9a12ad553f0

New changeset 76f2b99f3568 by David Malcolm in branch '3.2':
Issue #14443: ensure that brp-python-bytecompile is invoked with the correct
http://hg.python.org/cpython/rev/76f2b99f3568

Fixes committed to 3.2 and default: RHEL 6.3 is now green for both branches.  Closing this bug out.