importlib: PYTHONCASEOK should be ignored when using python3 -E #82872

vstinner · 2019-11-05T01:55:56Z

BPO	38691
Nosy	@brettcannon, @ncoghlan, @vstinner, @ambv, @ericsnowcurrently, @corona10, @aeros, @idomic, @petdance
PRs	bpo-38691 Added a switch to ignore PYTHONCASEOK when -E or -I flags passed #18314 Revert "bpo-38691 Added a switch to ignore PYTHONCASEOK when -E or -I flags passed (#18314)" #18553 bpo-38691 Ignore pythoncaseok tests #18612 bpo-38691: Ignore pythoncaseok tests #18627

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-25.18:01:14.573>
created_at = <Date 2019-11-05.01:55:55.991>
labels = ['library', '3.9']
title = 'importlib: PYTHONCASEOK should be ignored when using python3 -E'
updated_at = <Date 2020-03-25.18:01:14.572>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-03-25.18:01:14.572>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-25.18:01:14.573>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-11-05.01:55:55.991>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 38691
keywords = ['needs review']
message_count = 18.0
messages = ['355996', '360608', '361239', '361278', '362122', '362202', '362226', '362227', '362240', '362258', '362269', '362270', '362518', '363103', '363720', '364843', '364844', '365007']
nosy_count = 9.0
nosy_names = ['brett.cannon', 'ncoghlan', 'vstinner', 'lukasz.langa', 'eric.snow', 'corona10', 'aeros', 'Ido Michael', 'petdance']
pr_nums = ['18314', '18553', '18612', '18627']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38691'
versions = ['Python 3.9']

vstinner · 2019-11-05T01:55:56Z

When using python3 -E or python3 -I, PYTHONCASEOK environment variable should be ignored by importlib. See an email sent in 2012:
https://mail.python.org/pipermail/python-dev/2012-December/123403.html

See importlib._bootstrap_external._relax_case attribute and its _make_relax_case() function.

vstinner · 2020-01-24T10:42:08Z

sys.flags.ignore_environment should be used.

idomic · 2020-02-02T14:45:48Z

Created this PR: #62514

corona10 · 2020-02-03T10:08:42Z

@ido Michael
I left review comments for you :)
Thanks for the contribution.

vstinner · 2020-02-17T09:05:15Z

New changeset d83b660 by idomic in branch 'master':
bpo-38691 Added a switch to ignore PYTHONCASEOK when -E or -I flags passed (bpo-18314)
d83b660

vstinner · 2020-02-18T13:00:01Z

Tests fail on macOS:
https://buildbot.python.org/all/#/builders/275/builds/249

I reopen the issue. The issue should be fixed soon, or the change will be reverted to repair buildobts:
https://pythondev.readthedocs.io/ci.html#revert-on-fail

======================================================================
FAIL: test_case_insensitivity (test.test_importlib.extension.test_case_sensitivity.Frozen_ExtensionModuleCaseSensitivityTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/extension/test_case_sensitivity.py", line 36, in test_case_insensitivity
    self.assertTrue(hasattr(loader, 'load_module'))
AssertionError: False is not true

======================================================================
FAIL: test_case_insensitivity (test.test_importlib.extension.test_case_sensitivity.Source_ExtensionModuleCaseSensitivityTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/extension/test_case_sensitivity.py", line 36, in test_case_insensitivity
    self.assertTrue(hasattr(loader, 'load_module'))
AssertionError: False is not true

======================================================================
FAIL: test_insensitive (test.test_importlib.source.test_case_sensitivity.Frozen_CaseSensitivityTestPEP302)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/source/test_case_sensitivity.py", line 57, in test_insensitive
    self.assertIsNotNone(insensitive)
AssertionError: unexpectedly None

======================================================================
FAIL: test_insensitive (test.test_importlib.source.test_case_sensitivity.Frozen_CaseSensitivityTestPEP451)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/source/test_case_sensitivity.py", line 57, in test_insensitive
    self.assertIsNotNone(insensitive)
AssertionError: unexpectedly None

======================================================================
FAIL: test_insensitive (test.test_importlib.source.test_case_sensitivity.Source_CaseSensitivityTestPEP302)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/source/test_case_sensitivity.py", line 57, in test_insensitive
    self.assertIsNotNone(insensitive)
AssertionError: unexpectedly None

======================================================================
FAIL: test_insensitive (test.test_importlib.source.test_case_sensitivity.Source_CaseSensitivityTestPEP451)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/source/test_case_sensitivity.py", line 57, in test_insensitive
    self.assertIsNotNone(insensitive)
AssertionError: unexpectedly None

vstinner · 2020-02-18T16:42:34Z

I'm unable to debug the issue on macOS. I prepared PR 18553 to revert the change, just to give more time to fix the issue. It's to repair the CI, so we can notice other regressions.

vstinner · 2020-02-18T16:44:47Z

Tests fail on macOS: https://buildbot.python.org/all/#/builders/275/builds/249

Tests are run with "./python.exe ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml" which runs tests with "./python.exe -E": ignore PYTHON* environment variables. Maybe the test should just take that in account.

idomic · 2020-02-19T00:50:25Z

Yes I saw those in the morning, thanks for patching it up. I will debug this over the weekend and will update.

aeros · 2020-02-19T07:06:36Z

From what I can tell, the regression seems like it could be fixed by adding "@unittest.skipIf(sys.flags.ignore_environment)" to the following tests in python/cpython/Lib/test/test_importlib/source/test_case_sensitivity.py that modify "PYTHONCASEOK":

CaseSensitivityTest.test_sensitive
CaseSensitivityTest.test_insensitive

Those tests seem ultimately pointless if environmental variables are being ignored (-E or -I), so I think in this case it's a misleading test failure, no?

vstinner · 2020-02-19T13:23:59Z

New changeset 4dee92b by Victor Stinner in branch 'master':
Revert "bpo-38691 Added a switch to ignore PYTHONCASEOK when -E or -I flags passed (bpo-18314)" (GH-18553)
4dee92b

vstinner · 2020-02-19T13:24:52Z

Yes I saw those in the morning, thanks for patching it up. I will debug this over the weekend and will update.

Ok, perfect. In the meanwhile, as announced, I reverted the change to be able to notify other regressions.

Once we get a fix, we can reapply the change with the fix.

idomic · 2020-02-23T15:19:45Z

Added a new clean PR with the code changes, will let you know once the tests are fixed: #62827

idomic · 2020-03-01T22:34:02Z

@vstinner ready for review.

vstinner · 2020-03-09T11:58:00Z

New changeset fc72ab6 by idomic in branch 'master':
bpo-38691: importlib ignores PYTHONCASEOK if -E is used (GH-18627)
fc72ab6

ambv · 2020-03-23T10:40:03Z

Re-opening as it causes refleaks all across the stable buildbots. I can reproduce on macOS Catalina as well. Run this to see for yourself:

./python.exe -E -Wd -m test -uall,-gui -l -L -R: test_importlib

Reverting #62827 fixes the issue.

ambv · 2020-03-23T11:42:24Z

Reverting #62827 fixes the issue.

Sorry, this is false. I just checked out to the last commit to importlib before #62827 and it did not refleak. But when I actually reverted just #62827, the issue persisted.

By bisecting, I found out that it's #63284 which causes the refleak and it only just so happens that this refleak appears in importlib tests.

vstinner · 2020-03-25T18:01:14Z

FYI importlib leak was fixed in bpo-40050 by:

commit 83d46e0
Author: Victor Stinner <vstinner@python.org>
Date: Tue Mar 24 18:03:34 2020 +0100

bpo-40050: Fix importlib._bootstrap_external (GH-19135)

Remove two unused imports: _thread and _weakref. Avoid creating a new
winreg builtin module if it's already available in sys.modules.

The winreg module is now stored as "winreg" rather than "_winreg".

vstinner added 3.9 only security fixes stdlib Python modules in the Lib dir labels Nov 5, 2019

vstinner added the easy label Jan 24, 2020

vstinner changed the title ~~importlib: PYTHONCASEOK should be ignored when using python3 -E~~ [easy] importlib: PYTHONCASEOK should be ignored when using python3 -E Jan 24, 2020

vstinner closed this as completed Feb 17, 2020

vstinner reopened this Feb 18, 2020

vstinner removed the easy label Feb 18, 2020

vstinner changed the title ~~[easy] importlib: PYTHONCASEOK should be ignored when using python3 -E~~ importlib: PYTHONCASEOK should be ignored when using python3 -E Feb 18, 2020

vstinner closed this as completed Mar 9, 2020

ambv reopened this Mar 23, 2020

vstinner closed this as completed Mar 25, 2020

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

importlib: PYTHONCASEOK should be ignored when using python3 -E #82872

importlib: PYTHONCASEOK should be ignored when using python3 -E #82872

vstinner commented Nov 5, 2019

vstinner commented Nov 5, 2019

vstinner commented Jan 24, 2020

idomic mannequin commented Feb 2, 2020

corona10 commented Feb 3, 2020

vstinner commented Feb 17, 2020

vstinner commented Feb 18, 2020

vstinner commented Feb 18, 2020

vstinner commented Feb 18, 2020

idomic mannequin commented Feb 19, 2020

aeros commented Feb 19, 2020

vstinner commented Feb 19, 2020

vstinner commented Feb 19, 2020

idomic mannequin commented Feb 23, 2020

idomic mannequin commented Mar 1, 2020

vstinner commented Mar 9, 2020

ambv commented Mar 23, 2020

ambv commented Mar 23, 2020

vstinner commented Mar 25, 2020

importlib: PYTHONCASEOK should be ignored when using python3 -E #82872

importlib: PYTHONCASEOK should be ignored when using python3 -E #82872

Comments

vstinner commented Nov 5, 2019

vstinner commented Nov 5, 2019

vstinner commented Jan 24, 2020

idomic mannequin commented Feb 2, 2020

corona10 commented Feb 3, 2020

vstinner commented Feb 17, 2020

vstinner commented Feb 18, 2020

vstinner commented Feb 18, 2020

vstinner commented Feb 18, 2020

idomic mannequin commented Feb 19, 2020

aeros commented Feb 19, 2020

vstinner commented Feb 19, 2020

vstinner commented Feb 19, 2020

idomic mannequin commented Feb 23, 2020

idomic mannequin commented Mar 1, 2020

vstinner commented Mar 9, 2020

ambv commented Mar 23, 2020

ambv commented Mar 23, 2020

vstinner commented Mar 25, 2020