It seems IDEL is setting the environment variable 'HOME' on windows. Specifically I am on windows 8.1

This is extremly annoying and bad as according the code and doco, os.path.expanduser will preference this variable above others such as USERPROFILE.
https://docs.python.org/3/library/os.path.html#os.path.expanduser

This is causing scripts/libraries and in my case the AWS SDK boto3/botocore modules to break. As they are looking for config files in the users home location using os.path resolution order which should be %USERPROFILE% in my case as I don't have HOME set. But because idle is setting HOME to what looks to be my HOMEPATH and HOMEDRIVE this is making the AWS SDK unable to pickup the credential files required.

path = "~/.aws/credentials"
print(os.path.expanduser(path))

Should be
C:\Users\myUserName/.aws/credentials

But is instead
H:\/.aws/credentials

Please stop IDLE from setting this environment variable. Running the same scripts with python.exe or pythonw.exe works fine.

It also makes things none deterministic. As when I am off the domain or my network drive H:\ is unavailable it works and uses USERPROFILE.

Worked around this by setting HOME to be USERPROFILE before IDLE starts

Rather then change the system/user environment permanently. I edited the Idle script which the windows shortcut seems to run below.

C:\Python35\Lib\idlelib\idle.pyw

At the top of the script I added
------------------------------------
import os
os.environ['HOME'] = os.environ['USERPROFILE']
------------------------------------

This affects all Tkinter applications on Windows. For example, in Python 2.7:

    import Tkinter
    import ctypes
    libc = ctypes.CDLL('msvcr90')
    libc.getenv.restype = ctypes.c_char_p

    >>> libc.getenv(b"HOME")
    >>> root = Tkinter.Tk()
    >>> libc.getenv(b"HOME")
    'C:\\Users\\me'

TCL sets this environment variable in TclpSetVariables [1] and has since 1995 [2]. I don't think IDLE or any other Tkinter application should necessarily favor %USERPROFILE% over %HOMEDRIVE%%HOMEPATH%, and the default shouldn't be changed at this point. Probably the behavior should be documented for Tkinter and IDLE.

[1]: https://github.com/tcltk/tcl/blob/core_8_6_4/win/tclWinInit.c#L502
[2]: https://github.com/tcltk/tcl/blob/core_8_6_4/changes#L1378

As IDLE maintainer, this annoys me also. IDLE and in particular test_idle necessarily change the environment by calling tkinter.Tk().  When run under test.regrtest, test_idle gets slapped with a warning, and in 3.6, gets labelled a failure, even when it passes.

This issue cuts the other way too.  IDLE works for months and then stops working.  User posts "IDLE stopped working' on Stackoverflow.  In at least some cases, user installed a program that installed ab incompatible tcl/tk and set TCL_LIBRARY (which Python does not).  tk and tkinter get trapped by what for us is a bad setting.

Serhiy, could the _tkinter startup code recover from this and revert to using the one we installed?  Should I open a separate issue?

What is the problem? That user sets incorrect TCL_LIBRARY? I don't know how _tkinter can distinguish correct TCL_LIBRARY from the incorrect one.

Python does not set TCL_LIBRARY.  _tkinter finds tcl/tk installed with Python at its known location for the platform. tkinter and IDLE work fine for months.  Then user installs another program. That other software (not user, and unbeknownst to the user) sets TCL_LIBRARY to something that _tkinter cannot use.  I am guessing that _tkinter, seeing TCL_LIBRARY, tries to use alternate install, cannot, and gives up, instead of falling back to the python-tcl it was using for months.

This is Windows specific issue, I can't help with this, sorry. Zach should be more experienced in this.

For the original issue, I agree with Eryk that about the best we can do is to document that Tcl sets HOME as "%HOMEDRIVE%%HOMEPATH%" or "c:\" if both of those are not set.  I wonder about removing (or de-preferring) HOME in os.path.expanduser, but that would be a significantly backward incompatible change.  Since HOME is not set by default on Windows, setting it is the easiest way to manipulate os.path.expanduser, and changing it is going to break things.

The TCL_LIBRARY issue is completely separate.  I rewrote the logic that used to be in tkinter._fix for 3.5, it's now a part of _tkinter initialization and TCL_LIBRARY is no longer set (permanently) by tkinter (TK_LIBRARY is not set or inspected at all anymore, except by Tk itself).  I did this in an effort to avoid the environment warnings in the tkinter/IDLE tests, but apparently missed the HOME issue at that point.

Anyhow, we could theoretically unset TCL_LIBRARY (and TK_LIBRARY) in that initialization code, possibly conditional on whether TCL_VERSION can be found in the content of TCL_LIBRARY (the assumption being that a library for the same major version of Tcl ought to work anyway).  However, without that conditional, that would be another big backward compatibility issue; setting {TCL,TK}_LIBRARY is a legitimate way to point tkinter towards a non-standard Tcl/Tk library location (e.g. in an embedding situation).  With conditional clearing of {TCL,TK}_LIBRARY, we're still at the mercy of a just plain bad setting that just happens to contain the magic string that makes it look acceptable.

I'm -1 on making such a change anyway.  We've avoided permanently setting TCL_LIBRARY for many many years, other applications can figure out how to avoid it as well.  We can't try to clean up messes made by other badly behaved programs, that way lies madness.

#14576 is more or less about making better use of USERPROFILE when the value returned by expanduser to config.IdleConf.GetUserCfgDir fails.  Now that I know that tcl always sets HOME (or pieces that are joined back together to make HOME), so that expanduser never fails (?), this might make some sense - but only if they are not the same.

Zach, I was thinking that _tkinter read TCL_LIBRARY to start tcl, but your answer suggests that it is instead read by tcl.

The problem with bad TCL_LIBRARY is that if IDLE is started normally, there is no error message.  Even if the user tries starting in the console, the message is almost useless.  I "set TCL_LIBRARY=NONE" and got
   ...
  File "C:\Programs\Python35\lib\tkinter\__init__.py", line 1867, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: Can't find a usable init.tcl in the following directories:
    NONE C:/Programs/Python35/lib/tcl8.6 C:/Programs/lib/tcl8.6 C:/lib/tcl8.6 C:/Programs/library C:/library C:/tcl8.6.4/library C:/tcl8.6.4/library

Everything after NONE is nonsensical.  The IDLE doc needs a new sections "If IDLE does not start:.  Now I could write a troubleshooting note explaining for this note to ignore everything after the first item, and that the first item is an unusable setting of TCL_LIBRARY set by something or someone other than Python/Tkinter/IDLE.

I tried the same experiment with TK_LIBRARY and could see no effect.

ppperry, when I changed this to a doc issue, I added IDLE back as a component because, as I said in my last message, I want to include the facts revealed here in the IDLE doc.

eryksun, now I understand this is a bit more challenging because tkinter is the underlying library doing this and it is shared by other apps.

Though I still don't feel that just because its been like this since 1995 means it should stay this way. This is something which breaks other code following the python os.path.expanduser way(and probably other thing looking at the evitonmeny HOME). Not all people will try getting to the bottom of why, esspecially if they are a new user. It should just work.

Would iy be possible for tkinter to allow the user/app to decide the behavior?

Ie. Have a setting(s)/switch which allows the default behavior of setting and using HOME to be overiden.

By doing that existing apps remain unchanged and apps like IDLE could change this behavior to whatever they want when they are ready.

I think os.path.expanduser() behavior is by far more logical. Preference USERPROFILE if HOME is missing. USERPROFILE on windows is needed to even login. HOMEDRIVE/HOMEPATH is not.

How about wrapping the appropriate tkinter code with (untested yet)

import os
HOME = os.environ['HOME']
try:
    <run tkinter>
finally"
    os.environ['HOME'] = HOME  # will this unset?

or use a restore_env('HOME', ...) context manager?

Revising os.expanduser or apps to ignore HOME while tk is running is a related but different issue.

ntpath.expanduser() no longer uses HOME (though the doc string still refers to $HOME), so at least that problem is resolved. 

I suppose IDLE could work around the HOME issue in Windows by passing env=os.environ.copy() in the subprocess.Popen() call that creates the subprocess. os.environ will not contain the HOME value that TCL sets in the underlying process environment.