classification
Title: Bug in subprocess._args_from_interpreter_flags causes MemoryError
Type: crash Stage: commit review
Components: Library (Lib), Windows Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, mdengler, ncoghlan, python-dev, sbt, schlamar, vstinner
Priority: normal Keywords: patch

Created on 2014-03-17 09:38 by schlamar, last changed 2015-12-13 22:06 by gregory.p.smith. This issue is now closed.

Files
File name Uploaded Description Edit
output.log schlamar, 2014-03-17 09:38
Issue20954.patch schlamar, 2014-03-17 09:40 review
20954_test.patch schlamar, 2014-03-17 10:30
Issue20954.patch schlamar, 2014-03-17 11:01 review
Issue20954.patch schlamar, 2014-03-18 08:02 review
Messages (12)
msg213844 - (view) Author: Marc Schlaich (schlamar) * Date: 2014-03-17 09:38
Steps to reproduce:

- clone pytest-cov at https://bitbucket.org/schlamar/pytest-cov/commits/ac14225a67d715b6649f8158e05d2389b78620d2
- remove `@pytest.mark.skipif` from `test_multiprocessing_subprocess` in test_pytest_cov.py
- run: `tox --develop -e py27 -- -x`

Result: it fails with MemoryError in `subprocess._args_from_interpreter_flags` because sys.flags.hash_randomization has a value of 2147483647 and it tries to build a string with such a length. Find more details in attached log. Patch is following.
msg213845 - (view) Author: Marc Schlaich (schlamar) * Date: 2014-03-17 09:40
Patch added.
msg213847 - (view) Author: Marc Schlaich (schlamar) * Date: 2014-03-17 09:53
This comes from http://bugs.python.org/issue12098. Python 3.3 is affected, too.

Reproduction can be minimized by running the following script:


import multiprocessing


def main():
    p = multiprocessing.Process(target=lambda: None)
    p.start()
    p.join()


if __name__ == '__main__':
    main()

as

$ PYTHONHASHSEED=2147483647 python test.py
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    main()
  File "test.py", line 7, in main
    p.start()
  File "c:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "c:\Python27\lib\multiprocessing\forking.py", line 258, in __init__
    cmd = get_command_line() + [rhandle]
  File "c:\Python27\lib\multiprocessing\forking.py", line 364, in get_command_line
    opts = util._args_from_interpreter_flags()
  File "c:\Python27\lib\subprocess.py", line 508, in _args_from_interpreter_flags
    args.append('-' + opt * v)
MemoryError
msg213848 - (view) Author: Marc Schlaich (schlamar) * Date: 2014-03-17 10:30
Added TestCase.
msg213849 - (view) Author: Marc Schlaich (schlamar) * Date: 2014-03-17 10:30
BTW, patches are for 2.7 branch.
msg213850 - (view) Author: Marc Schlaich (schlamar) * Date: 2014-03-17 11:01
Merged test case and fix in a single commit/patch.
msg213947 - (view) Author: Marc Schlaich (schlamar) * Date: 2014-03-18 08:02
This was fixed in #19284 for Python 3.4 (without having possible consequences in mind). I have updated my patch accordingly.

Maybe it's worth to port my test case to Python 3.4.

Removed Python 3.3 as it isn't in bugfix maintenance anymore.
msg229239 - (view) Author: Martin Dengler (mdengler) * Date: 2014-10-13 11:42
Just got hit with this in 2.7.
msg231601 - (view) Author: Martin Dengler (mdengler) * Date: 2014-11-24 11:49
FWIW, I've been using a this patch for 2.7.5 in a production setting for a month now and have had no problems.
msg256342 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-12-13 20:49
The fix for Lib/subprocess.py looks good to me, but the test doesn't fail without the fix, so the test doesn't look good to me:

$ ./python -m test.regrtest -v test_multiprocessing_subprocess

At least, the test doesn't ensure that the bug is fixed.
msg256345 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-13 21:58
New changeset d42f6ce08362 by Gregory P. Smith in branch '2.7':
Fixes issue #20954: _args_from_interpreter_flags used by multiprocessing
https://hg.python.org/cpython/rev/d42f6ce08362
msg256346 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2015-12-13 22:06
_args_from_interpreter_flags really deserves its own unittest using a stubbed out sys.flags.  if someone wants to go ahead with the test provided in the existing patch here or write one of those, feel free.

As is, i've committed a patch logically equivalent (but less hacky) to the one from #19284.  Ironically _that_ patch wasn't the right thing to do in python _3_ but that's a separate issue i'll address outside of this bug.  It is effective for Python 2 where hash randomization must be explicitly enabled.
History
Date User Action Args
2015-12-13 22:06:28gregory.p.smithsetstatus: open -> closed
resolution: fixed
2015-12-13 22:06:17gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg256346
2015-12-13 21:58:04python-devsetnosy: + python-dev
messages: + msg256345
2015-12-13 20:49:35vstinnersetmessages: + msg256342
2015-12-03 19:18:30r.david.murraysetstage: commit review
2014-11-24 11:49:45mdenglersetmessages: + msg231601
2014-10-13 11:42:23mdenglersetmessages: + msg229239
2014-10-13 11:25:23mdenglersetnosy: + mdengler
2014-03-18 08:02:38schlamarsetnosy: + ncoghlan
2014-03-18 08:02:15schlamarsetfiles: + Issue20954.patch

messages: + msg213947
versions: - Python 3.3
2014-03-17 22:56:46vstinnersetnosy: + sbt
2014-03-17 11:52:16vstinnersetnosy: + vstinner
2014-03-17 11:01:21schlamarsetfiles: + Issue20954.patch

messages: + msg213850
2014-03-17 10:30:47schlamarsetmessages: + msg213849
2014-03-17 10:30:12schlamarsetfiles: + 20954_test.patch

messages: + msg213848
2014-03-17 09:53:02schlamarsetmessages: + msg213847
versions: + Python 3.3
2014-03-17 09:40:51schlamarsetfiles: + Issue20954.patch
keywords: + patch
messages: + msg213845
2014-03-17 09:38:58schlamarcreate