test_search_cpp error on AIX (with xlc) #55400

sable · 2011-02-11T15:02:16Z

BPO	11191
Nosy	@ncoghlan, @tarekziade, @merwok, @aixtools
PRs	bpo-11191: skip the distutils test 'test_search_cpp' when AIX and not gcc as compiler, etc. #5206 bpo-11191: fix test_distutils for AIX with xlc #8709

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tarekziade'
closed_at = <Date 2018-12-28.14:05:28.264>
created_at = <Date 2011-02-11.15:02:15.840>
labels = ['3.8', 'type-bug', 'library']
title = 'test_search_cpp error on AIX (with xlc)'
updated_at = <Date 2018-12-28.14:05:28.262>
user = 'https://bugs.python.org/sable'

bugs.python.org fields:

activity = <Date 2018-12-28.14:05:28.262>
actor = 'ncoghlan'
assignee = 'tarek'
closed = True
closed_date = <Date 2018-12-28.14:05:28.264>
closer = 'ncoghlan'
components = ['Distutils']
creation = <Date 2011-02-11.15:02:15.840>
creator = 'sable'
dependencies = []
files = []
hgrepos = []
issue_num = 11191
keywords = ['patch']
message_count = 10.0
messages = ['128399', '220898', '220935', '309982', '309999', '310518', '318522', '321919', '327047', '332658']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'sable', 'tarek', 'eric.araujo', 'Michael.Felt']
pr_nums = ['5206', '8709']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11191'
versions = ['Python 3.8']

sable · 2011-02-11T15:02:15Z

I have the following error in distutils on AIX.

======================================================================
ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/distutils/unixccompiler.py", line 166, in preprocess
    self.spawn(pp_args)
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/distutils/ccompiler.py", line 911, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'xlc' failed with exit status 40

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/distutils/tests/test_config_cmd.py", line 47, in test_search_cpp
    match = cmd.search_cpp(pattern='xxx', body='// xxx')
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/distutils/command/config.py", line 203, in search_cpp
    src, out = self._preprocess(body, headers, include_dirs, lang)
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/distutils/command/config.py", line 126, in _preprocess
    self.compiler.preprocess(src, out, include_dirs=include_dirs)
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/distutils/unixccompiler.py", line 168, in preprocess
    raise CompileError(msg)
distutils.errors.CompileError: command 'xlc' failed with exit status 40

I haven't investigated yet.

BreamoreBoy · 2014-06-17T21:32:37Z

After three years can we close this test failure as "out of date"?

sable · 2014-06-18T12:37:08Z

I don't have any AIX environment to try to reproduce the issue, so feel free to close it.

aixtools · 2018-01-15T13:59:16Z

After even more years - I see the same test failing, just a bit different.

And, others....

in short:

./python Lib/test/test_distutils.py
...
Ran 245 tests in 10.337s

FAILED (errors=7, skipped=31)

I have managed to get this to:

FAILED (errors=1, skipped=37)

And this is still:

ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)

So, moving forward now.

aixtools · 2018-01-15T16:53:23Z

a) Is this normal?

root@x065:[/data/prj/python/git/python3-3.7]./python -m unittest test.test_distutils

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

In short - at the start I had:

./python Lib/test/test_distutils.py
...
ERROR: test_run (distutils.tests.test_build_clib.BuildCLibTestCase)
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
ERROR: test_build_ext (distutils.tests.test_build_ext.ParallelBuildExtTestCase)
ERROR: test_get_outputs (distutils.tests.test_build_ext.ParallelBuildExtTestCase)
ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)
ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)

That is now down to:
ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)

There were two issues for search_cpp:
a) xlc -E needs -C to include comments in the output - this should be fixed
b) xlc -E does not accept -o as an argument - output must be to stdout, and sadly I am not clever enough to fix that. Help appreciated.

As is: the error was:

ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/unixccompiler.py", line 107, in preprocess
    self.spawn(pp_args)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/spawn.py", line 159, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'xlc_r' failed with exit status 40

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/tests/test_config_cmd.py", line 49, in test_search_cpp
    match = cmd.search_cpp(pattern='xxx', body='/* xxx */')
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/command/config.py", line 201, in search_cpp
    src, out = self._preprocess(body, headers, include_dirs, lang)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/command/config.py", line 124, in _preprocess
    self.compiler.preprocess(src, out, include_dirs=include_dirs)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/unixccompiler.py", line 109, in preprocess
    raise CompileError(msg)
distutils.errors.CompileError: command 'xlc_r' failed with exit status 40

and it is now:
======================================================================
ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/data/prj/python/git/python3-3.7/Lib/distutils/tests/test_config_cmd.py", line 49, in test_search_cpp
    match = cmd.search_cpp(pattern='xxx', body='/* xxx */')
  File "/data/prj/python/git/python3-3.7/Lib/distutils/command/config.py", line 206, in search_cpp
    file = open(out)
FileNotFoundError: [Errno 2] No such file or directory: '_configtest.i'

The code I do not know how to fix is:

File: Lib/distutils/command/config.py
def _preprocess(self, body, headers, include_dirs, lang):
src = self._gen_temp_sourcefile(body, headers, lang)
out = "_configtest.i"
self.temp_files.extend([src, out])
self.compiler.preprocess(src, out, include_dirs=include_dirs)
return (src, out)

See https://github.com/aixtools/cpython/tree/bpo-11191 for my changes

aixtools · 2018-01-23T18:52:13Z

Just tested the patch presented.

For Python3-3.6.4 the patch in PR-5206 works as is, however, for Python3-3.5.4 - the patch to Lib/test/support/init.py does not work.

So, for Python3-3.5, Python3-3.6 and Python3-master - the test test_search_cpp is resolved.

As that is the "literal" question here - I am removing the patch to Lib/test/support/init.py so that the same tests fail (for different reasons). I'll open a new issue for these tests - so that this PR can be applied to all current Python3 branches (and skip the test_search_cpp test when the compiler is not gcc on AIX).

HTH!

aixtools · 2018-06-03T06:05:35Z

I hope this can be reviewed and eventually closed - not because it is X years old and unresolved - but because it is resolved for the latest branches!

Thx

aixtools · 2018-07-18T21:00:10Z

@tarek - anything specific/extra you need?

aixtools · 2018-10-04T10:47:55Z

The current PR8709 resolves two issues, not one - so will create a new issue for the second element.

Am also shorting the NEWS text, with the expanded explanation here as:

skip the distutils test 'test_search_cpp' when not gcc as compiler
because not all compilers, e.g. IBM xlc et al do not support
redirected stdout when using the argument -E (aka cpp_preprocessing)
while gcc is know to support the pair -E -o outputfile

The additional find while researching this issue:

add the argument '-C' for AIX xlc cpp processing so comments are
included in the cpp output rather than replaced by a single ' ' (space) char.

ncoghlan · 2018-12-28T14:03:21Z

New changeset ed57e13 by Nick Coghlan (Michael Felt) in branch 'master':
bpo-11191: skip unsupported test_distutils case for AIX with xlc (GH-8709)
ed57e13

sable mannequin assigned tarekziade Feb 11, 2011

sable mannequin added the stdlib Python modules in the Lib dir label Feb 11, 2011

sable mannequin changed the title ~~test_search_cpp error on AIX (aith xlc)~~ test_search_cpp error on AIX (with xlc) Feb 11, 2011

BreamoreBoy mannequin added the type-bug An unexpected behavior, bug, or error label Jun 17, 2014

aixtools added the 3.7 (EOL) end of life label Jan 15, 2018

aixtools added the 3.8 only security fixes label Jun 3, 2018

ncoghlan removed the 3.7 (EOL) end of life label Dec 28, 2018

ncoghlan closed this as completed Dec 28, 2018

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_search_cpp error on AIX (with xlc) #55400

test_search_cpp error on AIX (with xlc) #55400

sable mannequin commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

BreamoreBoy mannequin commented Jun 17, 2014

sable mannequin commented Jun 18, 2014

aixtools commented Jan 15, 2018

aixtools commented Jan 15, 2018

aixtools commented Jan 23, 2018

aixtools commented Jun 3, 2018

aixtools commented Jul 18, 2018

aixtools commented Oct 4, 2018

ncoghlan commented Dec 28, 2018

test_search_cpp error on AIX (with xlc) #55400

test_search_cpp error on AIX (with xlc) #55400

Comments

sable mannequin commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

BreamoreBoy mannequin commented Jun 17, 2014

sable mannequin commented Jun 18, 2014

aixtools commented Jan 15, 2018

aixtools commented Jan 15, 2018

aixtools commented Jan 23, 2018

aixtools commented Jun 3, 2018

aixtools commented Jul 18, 2018

aixtools commented Oct 4, 2018

ncoghlan commented Dec 28, 2018