classification
Title: Setting PYTHONPYCACHEPREFIX using ~ (tilde) creates a "~" folder
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: vkbo, zach.ware
Priority: normal Keywords:

Created on 2020-10-14 08:27 by vkbo, last changed 2020-10-14 15:09 by vkbo.

Messages (4)
msg378607 - (view) Author: Veronica Olsen (vkbo) Date: 2020-10-14 08:27
On Debian Bullseye (testing):

Setting

export PYTHONPYCACHEPREFIX=~/.pycache

has previously worked. With Python 3.8.6, this still works in practice, when running python code (the pycache is still redirected), but during initialisation, the python3 executable creates an empty folder named "~" in the folder where the python exec or a python script is run.

Changing the environment variable to:

export PYTHONPYCACHEPREFIX=$HOME/.pycache

resolves the issue.

Expansion of ~ is of course a bit fickly in general, but 

I assume this is not intended. Deleting those folders from command line is rather tricky!
msg378609 - (view) Author: Veronica Olsen (vkbo) Date: 2020-10-14 09:13
Correction: The pycache is in fact redirected to the new '~' folder.

In any case, I tested a bit on Debian Bullseye (Python 3.8.6) and Ubuntu 20.04 (Python 3.8.5) and it seems ~ is not expanded in the environment variable Debian. I'm a little uncertain what is the expected behaviour here, but the sys.pycache_prefix variable returns different results on the two platforms:

With environment variable set to ~/.pycache on 3.8.6 (on debian testing), I get the following:

Python 3.8.6 (default, Sep 25 2020, 09:36:53) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.pycache_prefix
'~/.pycache'
>>> 

Doing the same on 3.8.5 on ubuntu 20.04 produces:

Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.pycache_prefix
'/home/******/.pycache'
>>> 

Still, perhaps the pycache_prefix should expand the ~ on linux?
msg378617 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-10-14 14:04
AFAIK, tilde expansion is entirely up to your shell rather than Python (unless you use something like `os.path.expanduser` in your own program).

A few debugging questions:
1) What version of what shell(s) do you have on each system?

2) What value do you get from `TESTVAR=~/test python3 -c 'import os;print(os.environ["TESTVAR"])'` on each system?  Is it the same as `TESTVAR=~/test printenv | grep TESTVAR`?

3) What results do you get from the same tests as above, but with `TESTVAR='~/test'`?
msg378620 - (view) Author: Veronica Olsen (vkbo) Date: 2020-10-14 15:09
> AFAIK, tilde expansion is entirely up to your shell rather than Python

Yes, I am uncertain what exactly is going on here. Whatever ends up in the sys.pycache_prefix variable is different with the same environment variable setting on the two systems. It may simply be an OS quirk. I am running on Debian testing after all.

It is clear from the CPython source that Python does nothing with the tilde. It eventually calls os.makedirs() in the compile() function, which I guess is when my tilde folder was made.

It would perhaps be useful if the os.path.expanduser call was added in the cache_from_source() function in importlib?

As for the results from the debug checks:

1)
Debian: GNU bash, version 5.0.18(1)-release (x86_64-pc-linux-gnu)
Ubuntu: GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)

2)
They both expand ~ to the home folder

3)
They both return ~/test
History
Date User Action Args
2020-10-14 15:09:42vkbosetmessages: + msg378620
2020-10-14 14:04:24zach.waresetnosy: + zach.ware
messages: + msg378617
2020-10-14 09:13:00vkbosetmessages: + msg378609
2020-10-14 08:27:09vkbocreate