classification
Title: Unable to run IDLE without write-access to home directory
Type: enhancement Stage: needs patch
Components: IDLE Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: 25507 27534 Superseder:
Assigned To: terry.reedy Nosy List: John Gray, asvetlov, cane, louielu, markroseman, ned.deily, roger.serwy, tealduck, terry.reedy
Priority: normal Keywords: patch

Created on 2010-03-25 15:39 by cane, last changed 2017-09-10 21:42 by terry.reedy.

Files
File name Uploaded Description Edit
error.png cane, 2010-03-25 15:39 Error message
@cfgdir.diff terry.reedy, 2015-10-28 06:36 Use temp dir as backup user config dir. review
@cfgdir2.diff terry.reedy, 2015-10-29 08:59 review
Pull Requests
URL Status Linked Edit
PR 2629 merged terry.reedy, 2017-07-08 01:13
PR 2631 merged terry.reedy, 2017-07-08 02:29
Messages (31)
msg101708 - (view) Author: (cane) Date: 2010-03-25 15:39
When trying to run Python 2.6.5 & 3.1 IDLE GUI on Windows 7, I receive the following error that the "IDLE's subprocess didnt make connection. Either IDLE can't start a subprocess or personal firewall software is blocking the connection."

I've researched this error and tried following the steps to troubleshoot this error without any success.  I do not have any firewall software installed or have the Microsoft firewall enabled.  When following issue 8099 and tring to set the TCL and TK library to the idle.py I get the error that I need to check the path and permissions. (see attached screenshot)

These workstations are setup in active directory enviroment where each username that logs into the workstation is a local administrator and the referenced "M:\" drive is a home directory that is mapped for them.

When the local administrator account logs into the workstation the user can execute the IDLE GUI without any issues.

I found reference in one article that the os.py creates a directory called ".idlerc". 

I'm wondering if there is a way to hardcode a reference path that doesnt point to my "M:\" drive for this directory or is there something else that I can try to fix this issue?

Thanks,
Bryan
msg101716 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-03-25 16:13
I just reproduced this by removing write access from my user for my home directory.

It seems odd that you wouldn't have write access to your home directory in the first place, but that's tangent to the issue. I'll see if there are other acceptable locations to attempt to place the config directory.
msg156771 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-03-25 21:06
Let's close the issue next week as report doesn't make sense if issuer has no write access to own home dir.
msg156773 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2012-03-25 21:20
Please don't close it. Users in this situation can't use IDLE. We should at least try alternative locations to create this directory or perhaps prompt them for a directory they'd like to use.
msg156774 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-03-25 21:28
Good point. Thank you.
msg188115 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-04-29 22:09
Issue17864 is a duplicate of this.  Also, note the error reported here is a result of IDLE not having write access in the user's home directory and thus cannot create the .idlerc directory.  IDLE seems to handle more gracefully the case of .idlerc existing but not writable.
msg222687 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-07-10 18:54
I think the proper solution is to warn "Cannot write .idlerc to your home directory. Settings will not be saved." and continue.
msg222694 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-07-10 19:43
.idlerc is a directory that contains the user versions of config-xyz.def. There are currently 4, another will probably be added.  The 'offending' code is in configHandler.IdleConf.GetUserCfgDir() (line 195).
if not os.path.exists(userDir):
    try:
        os.mkdir(userDir)
    except OSError:
        warn = ('\n Warning: unable to create user config directory\n'+
                        userDir+'\n Check path and permissions.\n Exiting!\n\n')
        sys.stderr.write(warn)
        raise SystemExit

The last line could be replaces by 'return None'.  The calling code that uses the normal return would have to be changed to skip trying to read and write the config files. According to Ned, Idle already manages if .idlerc exists but is unwritable.

The recent files list and breakpoint lists are also stored in .idlerc. Here are the 4 hits for 'GetUserCfgDir' in C:\Programs\Python34\Lib\idlelib\*.py ...

EditorWindow.py: 145:         self.recent_files_path = os.path.join(idleConf.GetUserCfgDir(),
PyShell.py: 131:         self.breakpointPath = os.path.join(idleConf.GetUserCfgDir(),
configHandler.py: 184:         userDir=self.GetUserCfgDir()
configHandler.py: 195:     def GetUserCfgDir(self):

While Idle could continue without any of these, I like Brian's idea of asking for an alternative first. Actually, if there is no 'home' directory (if expanduser('~') below fails), Idle already tries the current directory as a backup. It could do the same if a home directory exits but is unusable.

How far should we go with this? A command-line option to set the user cfg dir?  That should be doable.

I thought about a new idlelib/config-users.def mapping users to directories, but that has its own problems of write permission, as will as cross-platform access to user name.

The OP (Bryan) asked "[is] there is a way to hardcode a reference path that doesnt point to my "M:\" drive for this directory[?]"  Yes. GetUserCfgDir starts with
        cfgDir = '.idlerc'
        userDir = os.path.expanduser('~')
Replace the second line, after each install, to point to a writable directory.
msg222709 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-07-10 22:33
The use case reported here sounds like a classroom or lab environment with many people (and likely novices) using open environment machines.  In such cases, if users don't have write access to their home directories, it seems to me that there's no need to try to preserve IDLE configurations across sessions.  Asking the user for another location in such cases would be confusing and add needless complexity.  I'd say either just create a temporary directory or create the config files as temporary files as needed.  For advanced users, I suppose a command line option could be added but has there been any demand for such a feature?
msg222716 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-07-11 02:10
I don't know of any request for a new option, so I would go for something simple that lets Idle run. Temporary files are a good idea for breakpoints (temporary anyway, I think) and maybe for recent files.
msg252042 - (view) Author: John Gray (John Gray) Date: 2015-10-01 17:40
See Issue14576 which is the same underlying issue.
msg253404 - (view) Author: Keith Teal (tealduck) Date: 2015-10-24 12:44
I would also like to add that the location of this directory is not correct for Windows software. This directory should be created in %APPDATA% where users by default do have write permissions.

If there are plans to ever make this application portable then it should support a command line option to specify the location of the configuration as well.
msg253413 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-24 17:31
Idle is not unique.  Several other apps ported from unix also put .xyx files in the home directory on Windows. What is unusual, if not unique, about IDLE is the need to run multiple versions. If .idlerc is moved, already released versions will not be able to access it.  I am not ready to make a break yet.  In any case, moving it to a subdirectory of $HOME will not solve this issue, which is not being able to write to $HOME, and it therefore a different issue.
msg253414 - (view) Author: Keith Teal (tealduck) Date: 2015-10-24 17:51
Hi Terry,

I did not just make that stuff up on my last post, that is actually the standard for Windows applications. Yes, many Linux ports get it wrong but is that any reason to ever perpetuate a bad practice?

To see the standards you can download the Windows SDK but to make things easier for you here is a link that talks about this: http://blogs.msdn.com/b/patricka/archive/2010/03/18/where-should-i-store-my-data-and-configuration-files-if-i-target-multiple-os-versions.aspx

By using non-standard conventions the application is just asking for trouble like what is being manifested with this issue. Users will be able to write to their %APPDATA% area, no administrator would lock that down as it would cause too many applications to fail (Kiosk type installations not included).
msg253561 - (view) Author: Mark Roseman (markroseman) * Date: 2015-10-27 19:41
Can I suggest that this issue continues to be about IDLE not being able to write its preferences directory/files due to permissions, and we create a new issue for the fact that IDLE is storing it in the wrong place under Windows?
msg253575 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-28 00:39
Yes, that is what this issue *is* about.  IDLE, like Python itself, expects to be run on machines that users can write to normally, that have not been crippled by bureaucratic policies, or by users. The editor is useless if the user cannot write files somewhere.  None of this is specific to Windows.

Ned's idea of a temporary directory seems easy.  Change the warning and replace'raise SystemExit' with 'td = tempfile.TemporaryDirectory(); userDir = td.name' (see msg222694).  For 2.7, however, TemporayDirectory is not available. For the underlying mkdtemp, the user "is responsible for deleting the temporary directory and its contents when done with it."  (I presume failure of apps to do this is why temp file cleanup is needed.)  My inclination for 2.7 would be to copy a stripped down copy of TemporaryDirectory with just the finalizer code.  In fact, since we do not need or want the implicit cleanup warning (do we?), we could do that for all versions.

We do not have to accommodate all possibilities. One report, possibly on Stackoverflow, was from a user whose home dir and, I presume, appdata dir, were on a remote server.  Maybe he also had an offline home dir, I don't remember.  In any case, IDLE was not happy with the setup.

As I said before, permanently and unconditionally moving user config files on Windows will break compatibility with all previous releases, so I would not do that unless we decide to break compatibility anyway, for reasons other than MS's recommendations.  However, after a general solution is applied we could consider in a separate issue using Appdata as future-looking, Windows-specific alternative to a temporary directory.  However, this would require careful though lest we end up with two userdirs because the non-writability of homedir is only temporary.
msg253588 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-28 06:36
Attached is a patch that I think will work.  I have not tested it because I do not know how to make my home directory, and only my home directory, read-only.

Brian, how did you do it?  I now have Win 10.

When I rt click user/terry and select properties, there is a tri-state box "[ ] Read-only (Only applies to files in folder)".  It initially has a solid square, and changes to blank and checkmark.  Try to apply, there is an unselectable grayed-out choice to apply to the directory only and a mandatory choice to also apply recursively to all files and subdirectories.  I am loath to do this since there are 47000 files (40000 in appdate, which seems grossly excessive, but that is the report)

I would like this tested anyway at least once on linux and mac.  Testing procedure: change name of .idlerc, lock home dir, run installed IDLE from command line.  Should exit with message.  Run patched repository IDLE.  Should run, reporting temp dir and deleting it on exit.  (Unlock home dir and rename .idlerc back).
msg253604 - (view) Author: Mark Roseman (markroseman) * Date: 2015-10-28 15:40
Checked on Linux and Mac - doesn't work correctly. mkdtemp() returns a different name every time it's called, and GetUserCfgDir() is called in three places, meaning we end up with three different tmp directories (which on quick examination didn't all get cleaned up at end).

I'd suggest changing it so that GetUserCfgDir() caches its result and returns the cached version on subsequent calls. Running out the door so don't have time to try this myself right now...
msg253637 - (view) Author: Mark Roseman (markroseman) * Date: 2015-10-28 22:40
Just a note that the 'store things in APPDATA' is issue #24765
msg253668 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-29 08:59
OK: call GetUserCfgDir once on creation of idleConf instance and set .userdir attribute.  Replace repeated calls in PyShell and Editor with attributes accesses.  I tested that, with patch, IDLE starts and rewrites both breakpoints.lst and recent-files.lst as appropriate.
msg253681 - (view) Author: Mark Roseman (markroseman) * Date: 2015-10-29 15:57
Better, but alas still not quite. On further investigation, the issue is that a new instance of idleConf is instantiated in the subprocess, which then calls mkdtemp() returning a different name. You can see this by doing 'restart shell' and noting that it will hit the warning you added in GetUserCfgDir.

There are multiple places that the subprocess does access preferences, so just eliminating them is probably not the right way to go.

I'd probably recommend that the user prefs dir be communicated to the subprocess somehow. Two suggestions are via adding a command line parameter where we launch the subprocess (build_subprocess_arglist), or have the subprocess get it via 'remotecall' when it starts up (perhaps in MyHandler.handle). Either way this would then need to be communicated to idleConf so it uses that directory.

Would there be a preference and/or any other alternatives?
msg253714 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-30 04:00
I cannot currently think of any reason why the subprocess *needs* to access idleConf, so I want to put this on hold while investigating whether the accesses can be eliminated as part of #25507.
msg253719 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-30 05:53
Without being able to make my homedir read-only, I don't, of course, see the messages.  Since run does not use idleConf (I am now sure), you should have been able to proceed after clicking away the admittedly obnoxious repeat messages.  The old extraneous user-process temporary should disappear when you restart.  Can you check that?
msg297255 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-29 05:46
Problems with patching 2.7 are no longer relevant.

To test, we should refactor config so that the attempt to find and access $HOME and .idlerc are isolated in a function that can be mocked to simulate various problems.

def get_rc():
   """Return a directory path that is readable and writable.

   If not possible, return an error indicator.  <to be determined>
   """

Testing such a function is a different issue, but I would just reuse existing code.
msg297914 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-07 22:15
> GetUserCfgDir() is called in three places, (buried in one of Mark's posts).  On the fact of it, this seems like something that should be fixed.
msg297915 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-07 22:18
My second patch did fix that.  I think I will extract that part immediately.
msg297923 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-08 02:28
New changeset 223c7e70e48eb6eed4aab3906fbe32b098faafe3 by terryjreedy in branch 'master':
bpo-8231: Call idlelib.IdleConf.GetUserCfgDir only once. (#2629)
https://github.com/python/cpython/commit/223c7e70e48eb6eed4aab3906fbe32b098faafe3
msg297926 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-08 02:47
New changeset 552f26680d3806df7c27dd7161fd7d57ac815f78 by terryjreedy in branch '3.6':
[3.6] bpo-8231: Call idlelib.IdleConf.GetUserCfgDir only once. (GH-2629) (#2631)
https://github.com/python/cpython/commit/552f26680d3806df7c27dd7161fd7d57ac815f78
msg297976 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-09 02:26
#27534 is about reducing the imports into the user runcode process. Without rechecking the reason for each import, I think it possible that this might result in config not being indirectly imported.
msg298023 - (view) Author: Louie Lu (louielu) * Date: 2017-07-10 04:34
How about only taking warning when not able to create dir at GetUserCfgDir(), then take the permission handler in other place?

e.g. when user trying to save the config in bad dir, pop-out a dialog to tell it is permission problem or dir not eixsts...etc.
msg301835 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-09-10 21:42
I closed #30918 as a duplicate of this.  It has full 'set' and expanduser info.
History
Date User Action Args
2017-09-10 21:42:14terry.reedysetmessages: + msg301835
2017-09-10 21:38:34terry.reedylinkissue30918 superseder
2017-07-10 04:34:36louielusetnosy: + louielu
messages: + msg298023
2017-07-09 02:26:03terry.reedysetdependencies: + IDLE: Reduce number and time for user process imports
messages: + msg297976
2017-07-08 02:47:39terry.reedysetmessages: + msg297926
2017-07-08 02:29:26terry.reedysetpull_requests: + pull_request2696
2017-07-08 02:28:08terry.reedysetmessages: + msg297923
2017-07-08 01:13:55terry.reedysetpull_requests: + pull_request2694
2017-07-07 23:10:48terry.reedylinkissue30869 dependencies
2017-07-07 22:18:40terry.reedysetmessages: + msg297915
2017-07-07 22:15:16terry.reedysetmessages: + msg297914
2017-06-29 05:46:25terry.reedysetassignee: terry.reedy
messages: + msg297255
versions: + Python 3.7, - Python 2.7, Python 3.4, Python 3.5
2015-10-30 05:53:51terry.reedysetmessages: + msg253719
2015-10-30 04:00:00terry.reedysetdependencies: + IDLE: user code 'import tkinter; tkinter.font' should fail
messages: + msg253714
2015-10-29 15:57:38markrosemansetmessages: + msg253681
2015-10-29 08:59:57terry.reedysetfiles: + @cfgdir2.diff

messages: + msg253668
2015-10-28 22:40:43markrosemansetmessages: + msg253637
2015-10-28 15:40:22markrosemansetmessages: + msg253604
2015-10-28 06:36:03terry.reedysetfiles: + @cfgdir.diff
keywords: + patch
messages: + msg253588
2015-10-28 00:39:28terry.reedysettype: behavior -> enhancement
messages: + msg253575
2015-10-27 19:41:56markrosemansetmessages: + msg253561
2015-10-24 21:00:02terry.reedylinkissue17864 superseder
2015-10-24 17:51:54tealducksetmessages: + msg253414
2015-10-24 17:31:41terry.reedysetmessages: + msg253413
versions: + Python 3.5, Python 3.6, - Python 3.3
2015-10-24 12:44:29tealducksetnosy: + tealduck
messages: + msg253404
2015-10-01 17:40:20John Graysetnosy: + John Gray
messages: + msg252042
2015-09-18 16:39:50markrosemansetnosy: + markroseman
2014-09-28 00:06:14terry.reedylinkissue19062 superseder
2014-07-11 02:10:24terry.reedysetmessages: + msg222716
2014-07-10 22:33:56ned.deilysetmessages: + msg222709
2014-07-10 19:51:42brian.curtinsetnosy: - brian.curtin
2014-07-10 19:43:57terry.reedysetmessages: + msg222694
2014-07-10 18:54:55terry.reedysetnosy: + terry.reedy
messages: + msg222687
2013-04-29 22:09:29ned.deilysetnosy: + ned.deily
title: Unable to run IDLE without write-access to config directory -> Unable to run IDLE without write-access to home directory
messages: + msg188115

versions: + Python 3.4, - Python 2.6, Python 3.2
2012-03-25 21:28:14asvetlovsetmessages: + msg156774
2012-03-25 21:20:40brian.curtinsetmessages: + msg156773
2012-03-25 21:06:10asvetlovsetnosy: + asvetlov
messages: + msg156771
2011-12-17 19:37:47roger.serwysetnosy: + roger.serwy

components: + IDLE, - Library (Lib), Windows
versions: + Python 2.7, Python 3.2, Python 3.3, - Python 3.1
2010-03-25 16:13:26brian.curtinsetpriority: normal

type: crash -> behavior
components: + Library (Lib)
title: Subprocess Startup Error - unable to create user config directory -> Unable to run IDLE without write-access to config directory
nosy: + brian.curtin

messages: + msg101716
stage: needs patch
2010-03-25 15:39:28canecreate