classification
Title: site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: anjos, inada.naoki, ned.deily, steverweber
Priority: normal Keywords: patch

Created on 2017-04-26 08:53 by anjos, last changed 2018-11-30 11:32 by inada.naoki. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6731 merged steverweber, 2018-05-09 12:19
PR 7415 merged inada.naoki, 2018-06-05 07:25
PR 7606 merged miss-islington, 2018-06-11 00:50
PR 7607 merged miss-islington, 2018-06-11 00:51
PR 7617 merged inada.naoki, 2018-06-11 05:52
Messages (11)
msg292325 - (view) Author: André Anjos (anjos) Date: 2017-04-26 08:53
Apparently, "import site; site.main()" does not seem to work anymore on Python 3.6 and superior.

The reason is a change on the behavior of "os.path.abspath(None)". Before Python 3.6, it used to report an AttributeError which is properly caught inside "site.abs_paths" (see: https://github.com/python/cpython/blob/master/Lib/site.py#L99), making it ignore __main__, one of sys.modules, which has __file__ and __cached__ set to None.

With Python 3.6 and superior, os.path.abspath(None) reports a TypeError, which makes calling "site.main()" raise an exception and stop.

How to reproduce: On python 3.6 or superior, do "import site; site.main()".

Expected behavior: Exception is properly caught and treated inside "site.abs_paths", ignoring modules in which __file__ and/or __cached__ are set to None.
msg292327 - (view) Author: André Anjos (anjos) Date: 2017-04-26 09:13
More information: to reproduce the problem, don't use Python's "-c" command-line option. In this case "__main__" won't be inside sys.modules which mitigates the issue.
msg292329 - (view) Author: André Anjos (anjos) Date: 2017-04-26 09:55
After further investigation, the issue can only be reproduced iff the user sets PYTHONSTARTUP which triggers "__main__" to appear in sys.modules and the problem to occur.
msg316295 - (view) Author: steverweber (steverweber) * Date: 2018-05-08 19:19
ref to other projects that hit this issue.

https://gitlab.idiap.ch/bob/bob.buildout/issues/15
https://github.com/saltstack/salt/issues/45080
https://groups.google.com/forum/#!msg/comp.lang.python/tpfiHAJhl9Y/hZj1f50vBQAJ
msg318725 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-06-05 06:21
https://github.com/python/cpython/blob/f822549653d8d09bffff5b7dcddfdf12679a787c/Python/pythonrun.c#L391-L399

__file__ and __cached__ are added here.  And,

https://github.com/python/cpython/blob/f822549653d8d09bffff5b7dcddfdf12679a787c/Python/pythonrun.c#L441-L442

Only __file__ is removed here.  I feel __cached__ should be removed too.


On the other hand, if None is valid value for __cached__, site.abs_paths() should ignore it.

https://www.python.org/dev/peps/pep-3147/#file

> It is recommended that when nothing sensible can be calculated, implementations should set the __cached__ attribute to None.

It seems it's valid, and recommended for some cases.  So PR-6731 looks OK.
msg319258 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-11 00:49
New changeset 2487f30d5529948ace26559e274d7cac6abcd1a8 by Ned Deily (Steve Weber) in branch 'master':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731)
https://github.com/python/cpython/commit/2487f30d5529948ace26559e274d7cac6abcd1a8
msg319260 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-11 01:21
New changeset ec4343c3b4c0e0a7500122fac616e6488c0ab842 by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) (GH-7606)
https://github.com/python/cpython/commit/ec4343c3b4c0e0a7500122fac616e6488c0ab842
msg319261 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-11 01:23
New changeset 3e121581d008a780b8a9f1bcda5966cf0c06f6d5 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) (GH-7607)
https://github.com/python/cpython/commit/3e121581d008a780b8a9f1bcda5966cf0c06f6d5
msg319262 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-11 01:30
As I noted in the discussion on PR 6731, I think there should be a test for this so we don't break it again.  But, since it seems that the problem has affected a number of users and projects and since the fix is easy and easily testable manually, I decided to merge the PR for 3.7.0rc1 and 3.6.6rc1.  I am leaving the issue open for someone to supply a PR with a test case and for discussion of PR 7415.  Thank you all!
msg326797 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-10-01 12:10
New changeset d4c76d960b8b286b75c933780416ace9cda682fd by INADA Naoki in branch 'master':
bpo-30167: Add test for module.__cached__ is None (GH-7617)
https://github.com/python/cpython/commit/d4c76d960b8b286b75c933780416ace9cda682fd
msg330677 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-11-29 11:01
New changeset 82daa60defbd6497efdaa6c1132ecc8563122ed5 by INADA Naoki in branch 'master':
bpo-30167: Remove __cached__ from __main__ when removing __file__ (GH-7415)
https://github.com/python/cpython/commit/82daa60defbd6497efdaa6c1132ecc8563122ed5
History
Date User Action Args
2018-11-30 11:32:45inada.naokisetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-11-29 11:01:32inada.naokisetmessages: + msg330677
2018-10-01 12:10:45inada.naokisetmessages: + msg326797
2018-06-11 05:52:17inada.naokisetstage: needs patch -> patch review
pull_requests: + pull_request7238
2018-06-11 01:30:15ned.deilysetstage: patch review -> needs patch
messages: + msg319262
versions: + Python 3.8
2018-06-11 01:23:43ned.deilysetmessages: + msg319261
2018-06-11 01:21:37ned.deilysetmessages: + msg319260
2018-06-11 00:51:44miss-islingtonsetpull_requests: + pull_request7230
2018-06-11 00:50:49miss-islingtonsetpull_requests: + pull_request7229
2018-06-11 00:49:37ned.deilysetnosy: + ned.deily
messages: + msg319258
2018-06-05 07:25:04inada.naokisetpull_requests: + pull_request7039
2018-06-05 06:21:02inada.naokisetnosy: + inada.naoki
messages: + msg318725
2018-05-09 12:19:35steverwebersetkeywords: + patch
stage: patch review
pull_requests: + pull_request6432
2018-05-08 19:19:40steverwebersetnosy: + steverweber
messages: + msg316295
2017-04-26 17:05:32anjossettitle: site.main() does not work on Python 3.6 and superior with PYTHONSTARTUP is set -> site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set
2017-04-26 17:05:18anjossettitle: site.main() does not work on Python 3.6 and superior -> site.main() does not work on Python 3.6 and superior with PYTHONSTARTUP is set
2017-04-26 09:55:28anjossetmessages: + msg292329
2017-04-26 09:13:10anjossetmessages: + msg292327
2017-04-26 08:53:33anjoscreate