classification
Title: IDLE: inconsistent use of HOMEDRIVE, HOMEPATH, and USERPROFILE on Windows
Type: behavior Stage: needs patch
Components: IDLE Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Divyanshu, John Gray, asvetlov, clikkeb, markroseman, r.david.murray, roger.serwy, terry.reedy
Priority: normal Keywords:

Created on 2012-04-14 10:41 by clikkeb, last changed 2017-06-30 00:19 by terry.reedy.

Messages (16)
msg158253 - (view) Author: clikkeb (clikkeb) Date: 2012-04-14 10:41
It's a common issue that IDLE cannot start on Windows because

"IDLE's subprocess didn't make connection.Either IDLE can't start a subprocess or personal firewall software is blocking the connection."

Everyone claim that the user should set the firewal so that IDLE can connect to subprocess via loopback. I found, instead, that this issue is often caused by an incorrect or missing setting of one the following variables: HOME, USERPROFILE or the combination of HOMEPATH and HOMEDRIVE; if these variables don't represent an existent and writable directory, the error occurs.
Try setting HOMEPATH to an unexistent  or unwritable directory just before launching idle.bat script.
Check IdleConf.GetUserCfgDir() in module configHandler.py, where the function os.path.expanduser is called to get the user home directory. You might also temporarly patch GetUserCfgDir, setting the userDir variable to an unexistent path just after it has been initialized via os.path.expanduser (line 202).
Should be considered a check on expanduser behaviour?

OS: Windows XP
Python version: 3.2.2

clikkeb
msg158288 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-04-14 21:54
I can confirm that setting HOMEPATH to a non-existent directory will prevent IDLE from starting when using idle.bat. If you modify idle.bat such that python.exe is called instead of pythonw.exe, then IDLE starts normally, but with this console message:

  Warning: os.path.expanduser("~") points to
  C:\Users\doesnotexist
  but the path does not exist.

Normally, the message is written to sys.stderr, which is None when using pythonw. See issue13582 for more information.
msg158367 - (view) Author: clikkeb (clikkeb) Date: 2012-04-15 20:42
Thanks for your answer.

Trying to understand how IDLE uses HOMEPATH and USERPROFILE Windows
variables, I have found the following information:
1) it seems that when executed via Windows command prompt (cmd.exe),
   os.path.expanduser refers to USERPROFILE to determine the user's
   home directory;
2) analizing the stack traces of the first calls to
   IdleConf.GetUserCfgDir, you can see that GetUserCfgDir (which
   calls expanduser) is called three times during IDLE startup:
   the first two times it refers to USERPROFILE, the third (called
   via run.py, probably after starting a subprocess) it refers to
   the combination of HOMEDRIVE and HOMEPATH. (???)
3) when you start IDLE using pythonw, sys.stderr.write(warn) seems
   to raise an AttributeError exception, which is unhandled. This
   causes IDLE to stop running when you either start IDLE that way
   and the user's home directory is unreachable.

Due to the Python's tricky behaviour in determining the Windows
user's home directory, my opinion would be to consider if it is
worth to go further with this discussion or if it could produce
a benefit to IDLE. For sure, it gave me a little bit of headache.

clikkeb.
msg158387 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-04-16 00:58
It certainly is worthwhile pursing this in some fashion, since at the very least the existing error message needs to be improved.  But perhaps there is something more that can be done to gracefully handle this case, instead.  I think the next interesting question is to find out why an invalid home directory causes idle to not start up.  There's no obvious reason why that should be the case in principle, so I suspect there's some error handling missing somewhere.

This probably also applies to 2.7; someone should test that.
msg158404 - (view) Author: clikkeb (clikkeb) Date: 2012-04-16 08:30
I think that lines 207-210 of GetUserCfgDir should be modified like this:

try:
    sys.stderr.write(warn)
except (IOError, AttributeError):    # <---- 
    pass        #^^^^^^^^^^^^^^

because when you start IDLE via pythonw.exe (that sets sys.stderr to "None"),
the function call sys.stderr.write(warn) raises the following exception:

    AttributeError: 'NoneType' object has no attribute 'write'

and IDLE stops running without displaying any exception error, because that
exception is unhandled.

There is a funcion call to sys.stderr.write also at line 222, just before a
"raise SystemExit" statement, which makes ininfluent the missing 
AttributeError exception handling.
msg158432 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-04-16 13:08
I guess IdleConf should to have flag like 'writable'.
If user environment points to invalid location (or there are no write access) this flag should be set.
It flag can affect IDLE configuration dialog: user should be notified what him changes will not be permanently saved.
Subprocess can check that flag as well if need.

BTW, there are related issue8231.
msg158534 - (view) Author: clikkeb (clikkeb) Date: 2012-04-17 08:22
It is one of the possible solutions.

In combination with the "writable flag" solution, you might create a class
variable in IdleConf (e.g. "user_cfg") that contains the user's home directory;
such variable will be initialized to an empty string by IdleConf.__init__.
On each call, GetUserCfgDir tries to initialize "user_cfg" only if it is an
empty string, otherwise it returns it's content.

Anyway, keep in mind that in GetUserConfig there's an unhandled exception (AttributeError) that might be the cause of the "IDLE doesn't start..." issue reported by users.
msg180121 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-01-17 08:35
>I think the next interesting question is to find out why an invalid home directory causes idle to not start up.  There's no obvious reason why that should be the case in principle,

As Roger showed, IDLE will start up with an invalid home directory.

> so I suspect there's some error handling missing somewhere

Yes, the uncaught exception raised when attempting to display a non-essential but user-useful messages (this is the third issue I know of about this). There are 5 places in configHandler.py where a warning is directly printed to sys.stderr. There are two similar writes in PyShell.py. The first warning is the problem here, but any of them could be a problem and all should be handled better.

Lib/idlelib\PyShell.py: 1357: sys.stderr.write("Error: %s\n" % str(msg))
Lib/idlelib\PyShell.py: 1358: sys.stderr.write(usage_msg)
Lib/idlelib\configHandler.py: 209:    sys.stderr.write(warn)
Lib/idlelib\configHandler.py: 223:    sys.stderr.write(warn)
Lib/idlelib\configHandler.py: 255:    sys.stderr.write(warning)
Lib/idlelib\configHandler.py: 367:    sys.stderr.write(warning)
Lib/idlelib\configHandler.py: 624:    sys.stderr.write(warning)

Why are warning used instead of message boxes? Perhaps easier, perhaps the latter are not available because the tk loop is not running. Possible solutions:

* Discard the message by just catching AttributeError (clikkeb) -- at each place indicated above. This would be better than nothing, but only as a temporary measure, and there is a better temporary measure below.

* Rearrange the start up process to make message boxes available, with default configuraton, before the configuration process.

* Make a list of messages and display them when it must (on exit, if that is possible with atexit) or in message boxes when it can.

* Solve the more general problem of writing to None by making sys.stdout and sys.stderr not be None, #13582. At minimum, even as a temporary measure, use something that just swallows output. Something like

import io
class DummyOut(io.IOBase):
    def write(self, s):
        return len(s)

dum = DummyOut()
print(dum.write('sixsix'))
# 6
msg185892 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-04-03 06:12
This issue triggers the problem described in #13582. It points out problems with IDLE's configuration manager w.r.t. determining the home directory. I changed the title so that it reflects that point.
msg212257 - (view) Author: Divyanshu Sharma (Divyanshu) Date: 2014-02-26 15:06
I found a weird solution for the problem. Exchange the names of python and pythonw in the python33 folder. This makes the IDLE to call both and as a result python opens in both modes simultaneosly, and the subprocess connection error don't shows up.
msg212296 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-02-26 20:07
I do not understand what you mean by "Exchange the names of python and pythonw in the python33 folder.". In any case, idle.bat cannot run both simultaneously.

Perhaps idle.bat should have an option to start with python instead of pythonw.
msg215588 - (view) Author: Divyanshu Sharma (Divyanshu) Date: 2014-04-05 09:58
I found the root cause for the error regarding python in my pc that shows the error message 

"IDLE's subprocess didn't make connection.Either IDLE can't start a subprocess or personal firewall software is blocking the connection."

It was caused due to a software known as proxifier which provides fast and somewhat unrestricted network access on proxy networks. It redirected my connection request from pythonw.exe to the port and IP on which the proxy network is established.

The problem hence can be easily sorted by creating new proxification rules in such softwares, if the cause of the problem can be traced to some similar mechanisms used that may intercept and modify the network requests for other programs.
msg228267 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-10-02 22:17
Divyanshu, please do not hijack by changing the title to a different issue.  There are multiple causes for the message, some unknown.  Your non-standard system is different from that of clikkeb.
msg252040 - (view) Author: John Gray (John Gray) Date: 2015-10-01 17:32
I hit this issue with an "H:" homedrive that is on a network share, and then in Windows is using "offline files" to keep a local copy.   .idlerc was not cached so IDLE worked when online/connected to my work network but not when I was offline. 

The temporary workaround was to mark .idlerc as "available offline" when connected.  I wanted to document this in case others hit this scenario.  

I also noticed that there is a .idlerc directory created in my local user directory c:\users\[username\.idlerc.  This one has the recent-files.lst in it.  So IDLE is creating two copies of .idlerc - one in the environment-variable-defined (and roaming) home directory, and one in the default local user directory.   I would suggest that as this bug is investigated, you keep track of both instances of .idlerc.
msg252041 - (view) Author: John Gray (John Gray) Date: 2015-10-01 17:35
For the h: drive issue mentioned above:  This was on Python 2.7.10.
msg252062 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-01 21:55
John, thank you for the additional information.  2 copies of .idlerc is definitely bad.
History
Date User Action Args
2017-06-30 00:19:23terry.reedysetassignee: terry.reedy
versions: + Python 3.7, - Python 2.7, Python 3.5
2015-10-07 00:33:28markrosemansetnosy: + markroseman
2015-10-01 21:55:03terry.reedysetmessages: + msg252062
versions: + Python 3.5, Python 3.6, - Python 3.3, Python 3.4
2015-10-01 17:35:22John Graysetmessages: + msg252041
2015-10-01 17:32:35John Graysetnosy: + John Gray
messages: + msg252040
2014-10-02 22:17:19terry.reedysetmessages: + msg228267
title: IDLE: "IDLE's subprocess didn't make connection.Either IDLE can't start a subprocess or personal firewall software is blocking the connection." -> IDLE: inconsistent use of HOMEDRIVE, HOMEPATH, and USERPROFILE on Windows
2014-07-10 18:51:09terry.reedylinkissue10722 superseder
2014-04-05 09:58:40Divyanshusetmessages: + msg215588
title: IDLE: resolving home directory for configuration uses HOMEDRIVE, HOMEPATH, and USERPROFILE inconsistently on Windows. -> IDLE: "IDLE's subprocess didn't make connection.Either IDLE can't start a subprocess or personal firewall software is blocking the connection."
2014-02-26 20:07:25terry.reedysetmessages: + msg212296
2014-02-26 15:06:58Divyanshusetnosy: + Divyanshu
messages: + msg212257
2013-06-15 18:57:15terry.reedysetversions: + Python 2.7, Python 3.4, - Python 3.2
2013-04-03 06:12:56roger.serwysetmessages: + msg185892
title: IDLE: closes when writing warnings on Windows -> IDLE: resolving home directory for configuration uses HOMEDRIVE, HOMEPATH, and USERPROFILE inconsistently on Windows.
2013-01-17 08:35:48terry.reedysetmessages: + msg180121
title: IDLE cannot connect to subprocess - New solution -> IDLE: closes when writing warnings on Windows
2012-04-17 08:22:48clikkebsetmessages: + msg158534
2012-04-16 13:08:18asvetlovsetmessages: + msg158432
2012-04-16 08:30:12clikkebsetmessages: + msg158404
2012-04-16 00:58:14r.david.murraysetversions: + Python 3.3
nosy: + r.david.murray

messages: + msg158387

type: behavior
stage: needs patch
2012-04-15 20:42:22clikkebsetmessages: + msg158367
2012-04-14 21:54:56roger.serwysetnosy: + terry.reedy, roger.serwy, asvetlov
messages: + msg158288
2012-04-14 10:41:05clikkebcreate