Title: IDLE: Remove '', user_dir, and idlelib from sys.path when added
Type: behavior Stage: needs patch
Components: IDLE Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: markroseman, python-dev, terry.reedy
Priority: normal Keywords:

Created on 2015-10-27 09:25 by terry.reedy, last changed 2017-06-19 18:31 by terry.reedy.

File name Uploaded Description Edit
idle-start.txt terry.reedy, 2016-02-28 03:25
Messages (11)
msg253525 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-27 09:25
Problem: Consider module  The following is correct.

C:\Users\Terry>python -c "import run"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'run'

But in IDLE started from icon or command line
>>> import run

With respect to the goal of developing code that can run outside of IDLE, not raising ImportError is a bug. Suppose a user erroneously writes 'import run' or 'import help' (in repository IDLE).  The error could be that the file is 'runner' or 'helper' or that 'run' or 'help' is not in the same directory. The import should fail rather than the user getting a error later because the wrong file was imported.

Reason: When python runs 'somepath/', it prepends 'somepath' to sys.path.  So when python runs '.../idlelib/', it prepends '.../idlelib' to sys.path.  This occurs in both IDLE and user processes.

When 'somepath/' is either on startup or from the editor, IDLE first prepends 'somepath' to sys.path just as python does.

We are planning to add perhaps 50 new short lowercase (PEP8) names to idlelib.  This will greatly increase the possibility of accidentally matching something users write, if only in error, or even of  masking a stdlib module.

Solution: In PyShell.main and run.main, remove the prepended idlelib entry *if it is there* (see below).  On Windows, at least, for installed python, sys.path would then be identical on startup whether a file is run with python or IDLE.  

Would this affect IDLE itself?  I am sure not. AFAIK, IDLE does not exploit the presence of idlelib on sys.path, and always imports idlelib files via idlelib.

Further more, when IDLE is started from console Python with 'import idlelib.idle', which I regularly do for the repository version, '.../idlelib' is *not* prepended.  This is why IDLE should not depend on its presence and why removal should check first.  This also means that whether a file runs in IDLE depends on how IDLE is started, whether directly or via '>>> import idlelib.idle' within console python.

Note 1. idlelib.idle prepends repository 'lib' even when already present; this could be fixed also.  Another issue to track down, probably not due to IDLE, is that at least on Windows the binary directory is present twice.)

Note 2: The contents of sys.modules would still be different, along with the problem that 'import tkinter; tkinter.font' runs in IDLE.  (This example comes from a Stackoverflow question.) But this is a different issue and I cannot see a fix for it.
msg253527 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-27 11:18
When python is started without running a file (Windows icon or command line), '' is prepended to sys.path.  When IDLE is started from 3.5 icon, '' is prepended temporally after and spatially before '.../idlelib', so the latter, to be removed, is sys.path[1] and not sys.path[0].

'' in sys.path interpreted as the current working directory.

When IDLE is started from the command line with '-m idlelib' instead of '.../idlelib/' (for instance), the current working directory is added instead of .../idlelib (in second position).

Adding the startup current directory is redundant with '' until the user imports os and changes the cwd.  Then I expect the behavior of Shell to be different from python interactive mode, which would be a bug.  I will try to find a test case later.
PyShell.main manipulates 'sys.path' in lines 1521 and 1525, but these are sys.argv of the idle process, not the user process, each of which have their own sys.argv. If filenames are given on the command line to be edited, the directory of each is added to sys.path *for the idle process* (line 1521).  I suspect this is for completions. Line 1525 add the current working directory.  I don't know why.  Perhaps it is a holdover for -n single process mode.
msg260952 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-02-28 01:15
#25507 addresses the tkinter.font problem of Note 2 in the first message.
msg260961 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-02-28 03:18
In private email, eryk sun reports  "/usr/bin/idle3 instead prepends 3 directories: ['', current_directory_absolute, '/usr/bin']. If I use the -r option to run a script, it instead prepends script_directory_relative, '', '/usr/bin']. I think these should be [''] and [script_directory_absolute], respectively."

Windows does not have idle3.  None of the directories above are needed to run IDLE and I believe any could contain masking files.
msg260962 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-02-28 03:25
Upload: A listing of console (cross-platform) and icon (Windows-specific) IDLE starting methods and the effect on sys.path and current directory.  I believe that some of the details of the 'constant' part of sys.path are system dependent, but stable across methods for a particular python version.
msg277743 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-09-30 06:41
sys.path for 3.6.0b1 on Win 10.
Start python from icon (start menu), console (py -3.6) or Explorer (click on python.exe)

Start Python from Explorer by double-clicking *.py file or context menu 'Open': replace '' with
'C:/user/Terry/lib'  # directory containing file.

Running 'import sys; sys.path' (in user process) from IDLE Shell adds the following after ''.
Start IDLE from icon:
Start IDLE in console with py -3.6 -m idlelib:
'F:\\Python\\dev'  # the CWD of the console
Start IDLE in console with py -3.6 -m idlelib.idle:
'C:\\Programs\\Python36\\lib',  # duplicate added by
'F:\\Python\\dev'  # the CWD of the console
Start IDLE from Python with 'import idlelib.idle'
'C:\\Programs\\Python36\\lib',  # duplicate added by
'C:\\Programs\\Python36'  # the CWD of the running Python (dup)

Currently, inserts the directory containing idlelib into sys.path.  This is a duplicate unless running idlelib in a non-stanard location.  Check first.
msg277744 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-30 06:57
New changeset 5f788ad057ca by Terry Jan Reedy in branch '2.7':
Issue #25488: Stpp from adding a entry when it is a duplicate.

New changeset 013956a801e4 by Terry Jan Reedy in branch '3.5':
Issue #25488: Stpp from adding a entry when it is a duplicate.

New changeset 25ae49903496 by Terry Jan Reedy in branch '3.6':
Issue #25488: merge from 3.5.
msg277746 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-09-30 07:06
I usually start repository IDLE with 'import idlelib.idle' in repository console python, which I have pinned to my taskbar.  This is at least as faster as entering a command in the console (which would block the console), keeps error messages for one version together in one place and separate from anything else, and leaves my console free to enter other commands.

The import change will make idlelib.idle.main() work to restart IDLE after closing it.  Previously, idlelib.idle.idlelib.pyshell.main() was required, which I never did.  I marked the import as subject to change, as I intend to move (and edit) much of pyshell.main to
msg286891 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-02-03 21:20
I should have said that the problem with having '' at the beginning of the path is that when IDLE tries to import an stdlib module and '' (usually a user directory) contain a file with the same name, IDLE will import the user file instead and at some point likely exit.  For example, a user file named masks the stdlib, which is essential to Shell operation (#29395).
msg286892 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-02-03 21:47
The sys.pyth manipulation will be different in the idle and user processes.  If IDLE is started with -n, so both processes are the same, even more care is needed.
msg296366 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-19 18:31
#25488 and #26143 address related issues of sys.path as seen by IDLE and user code.  #26143 focuses on what IDLE sees, and user modules shadowing IDLE modules, #25488 focuses on what user code sees, and the presence of idlelib in sys.path making buggy code work.  It includes startup information relevant to #26143 also.  Both should be solved by patching IDLE in same place.
Date User Action Args
2017-06-19 18:31:31terry.reedysetmessages: + msg296366
components: + IDLE
2017-02-03 21:50:05terry.reedylinkissue29395 superseder
2017-02-03 21:47:27terry.reedysetmessages: + msg286892
2017-02-03 21:20:01terry.reedysetmessages: + msg286891
2016-09-30 07:06:06terry.reedysetmessages: + msg277746
versions: + Python 3.7
2016-09-30 06:57:20python-devsetnosy: + python-dev
messages: + msg277744
2016-09-30 06:41:43terry.reedysetmessages: + msg277743
2016-02-28 03:25:26terry.reedysetfiles: + idle-start.txt

messages: + msg260962
2016-02-28 03:18:34terry.reedysetmessages: + msg260961
2016-02-28 01:15:52terry.reedysettitle: IDLE: Remove '' and idlelib from sys.path when added -> IDLE: Remove '', user_dir, and idlelib from sys.path when added
messages: + msg260952
versions: - Python 3.4
2015-11-21 18:39:25terry.reedysettitle: IDLE: Remove idlelib from sys.path when added -> IDLE: Remove '' and idlelib from sys.path when added
2015-10-27 11:18:51terry.reedysetmessages: + msg253527
2015-10-27 09:25:13terry.reedycreate