[.]pydistutils.cfg will be deprecated in favor of a single
"distutils.cfg", located in .local/ alongside the files added for
example by the per user-site packages.

Why is this? .local/ contains machine-installed files while
.pydistutils.cfg is edited by the user.

That's what is returned by site.getuserbase()

(which uses PYTHONUSERBASE, and defaults to ~/.local)

PYTHONUSERBASE is the root of Python user-specific paths so it makes
sense to have all Python related files in there.

This came up with the discussion we had with others at Distutils-SIG

read this thread here for more info:
http://mail.python.org/pipermail/distutils-sig/2009-August/012865.html

[.]pypirc should probably go there too.

> That's what is returned by site.getuserbase()
> 
> (which uses PYTHONUSERBASE, and defaults to ~/.local)

Well, that makes it the user-specific equivalent of /usr or /usr/local.
Do you put your configuration files in /usr/local ? Why put them
in .local ?

> PYTHONUSERBASE is the root of Python user-specific paths so it makes
> sense to have all Python related files in there.

That doesn't make sense actually, since .local isn't Python-specific. :)

I've read the thread now and I think the original proposal of having a
~/.python was more sensible and more user-friendly.

+1 to what Antoine said.

> Well, that makes it the user-specific equivalent of /usr or /usr/local.
> Do you put your configuration files in /usr/local ? 
> Why put them in .local ?

The distutils.cfg file is located in the distutils package itself !

So the target is ~/.local/pythonX.X/site-packages/distutils/distutils.cfg

(sorry if that was unclear)

Now, I would totally agree to have a better ~/.NAME to reunite
user-specific *configuration* file for Python, but that would be yet
another directory for user-specific Python files.

> The distutils.cfg file is located in the distutils package itself !
> 
> So the target is ~/.local/pythonX.X/site-packages/distutils/distutils.cfg

Ah, that's horrible. -1 to that.
Configuration files should be easily findable by skimming through the
dot-files in the user's directory. ~/.python fulfills that condition.

I like ~/.python very much too (that was my initial proposal IIRC). 

I guess the best way to add this support is to provide a new api in
site.py, that returns this folder.

Distutils can move pypirc and distutils.cfg file there.

Now for the global distutils.cfg, if we want to have it somewhere else
that the distutils package directory itself, we would need a
"/etc/python" scheme for instance, that could be also added in site.py.

I don't know though how the latter would look under windows

Tarek Ziadé wrote:
> 
> Tarek Ziadé <ziade.tarek@gmail.com> added the comment:
> 
> I like ~/.python very much too (that was my initial proposal IIRC). 

+1 on that name as well.

One thing to note though: it is well possible that a user uses
multiple Python versions. With just one such directory, all
versions would look in the same directory for the configuration
files and this could lead to incompatibilities, e.g. Python 2.7
might not like that Python 3.6 needs in some config file (for
whatever reason).

> Now for the global distutils.cfg, if we want to have it somewhere else
> that the distutils package directory itself, we would need a
> "/etc/python" scheme for instance, that could be also added in site.py.
>
> I don't know though how the latter would look under windows

I think Subversion get's this pretty right, so we might want
to follow their scheme:

http://svnbook.red-bean.com/en/1.1/ch07.html#svn-ch-7-sect-1

The subversion system looks pretty nice !

So here's a modified proposal that covers the python version as well.

= per user configuration files = 

 ~/.pythonMAJOR.MINOR

Examples:

~/.python2.7
~/.python3.1

On Windows, ~ will be replaced by APPDATA if founded in the environment.
 
== global configuration files ==


On Linux/Mac:

/etc/pythonMAJOR.MINOR

examples:

/etc/python2.7/
/etc/python3.1/

On Windows:

\All users\Application Data\PythonMAJORMINOR

(\All users\Application Data is specified by the Windows Registry)

examples:

\All users\Application Data\Python26
\All users\Application Data\Python31

== standard naming for configuration files ==

The files are all named "NAME.cfg", in all platforms.

== what gets in per-user configuration directory ==

- pypi.cfg
- distutils.cfg

== what gets in global configuration directory ==

- distutils.cfg

== apis added ==

in site.py:

- getuserconfig() : returns the per-user configuration directory
- getconfig(): returns the global configuration directory

(the names are following site.py naming style)

FYI, /etc/pythonX.Y (and /etc/python) are used by Debian; the former
contains site.py and sitecustomize.py, the latter a file called
debain_config.

I can't see how this could become a problem, but maybe someone else does.

Thanks for the info Georges.

I am adding Matthias and Michael so they can give us their opinions for
their platforms (Debian/Ubuntu and Win)

Personally I'm uncomfortable with creating
yet-another-location-for-config-files.

As we now have is ~/.local/pythonX.Y I would reuse this.

But ~/.local is simply not the location for config files.

For Linux and Mac OS X put the config file wherever the 'right' place is
then, but we are starting to create a lot of new directories disparate
from the actual install. Oh well.

On Windows we create: %APPDATA%/Python/PythonX.Y/site-packages

If it is to no longer live inside distutils (which seems reasonable)
then on Windoze a 'good' place would seem to be: %APPDATA%/Python/PythonX.Y/

On Windows there is no 'right' place, the officially right place for the
system is probably the registry but I don't think many Python
programmers would thank you for that...

Noting of course that on IronPython it should be:
%APPDATA%/IronPython/PythonX.Y/

or wherever we decided in the end. :-)

/etc is definitely not the right place to put files for OS X framework 
builds; if necessary, an etc directory could be added under the 
framework version directory as a sibling of bin and lib.  It's also very 
un-OS X like to be putting things into ~/.python and ~/.local 
directories; the usual place would be in somewhere ~/Library, possibly 
~/Library/Application Support/Python or ~/Library/Frameworks.  Keep in 
mind that it's much more likely on OS X to not only have muitiple 
versions of Python installed but also more than one instance of the 
*same* version, for instance, on 10.6, an Apple-supplied 2.6.2 and a 
python.org 2.6.4.  Some thought should be given to locations for the 
files for Unix-style (non-framework) builds.

(Adding Ronald.)

Ned Deily wrote:
> 
> Ned Deily <nad@acm.org> added the comment:
> 
> /etc is definitely not the right place to put files for OS X framework 
> builds; if necessary, an etc directory could be added under the 
> framework version directory as a sibling of bin and lib.  It's also very 
> un-OS X like to be putting things into ~/.python and ~/.local 
> directories; the usual place would be in somewhere ~/Library, possibly 
> ~/Library/Application Support/Python or ~/Library/Frameworks.  Keep in 
> mind that it's much more likely on OS X to not only have muitiple 
> versions of Python installed but also more than one instance of the 
> *same* version, for instance, on 10.6, an Apple-supplied 2.6.2 and a 
> python.org 2.6.4.  Some thought should be given to locations for the 
> files for Unix-style (non-framework) builds.
> 
> (Adding Ronald.)

I think there is a misunderstanding here: we're trying to find
places where distutils would look by default, not where it
would write files.

AFAIK, there is general agreement that when writing files,
Python should use the ~/.local/pythonX.X/ directory.

OTOH, searching for config files that the user creates and
puts in place is another story. Python and distutils only
need to be able to find these and the idea is to use platform
specific standard search paths for this, so that the user
can place those config files into places that feel right on
each platform.

I don't think there's a misunderstanding.  By "putting", I meant "reading" 
or "writing".  IMO, /etc is not the place on OS X to be looking for 
python-related configuration files, certainly not for framework installs.  
Likewise for ~/.python and ~/.local.  Unfortunately, the latter is already 
out in the field; that was a step in the wrong direction for OS X.

I agree with Ned that neither ~/.local nor /etc are a good fit for OSX, 
sadly enough I wasn't paying attention when ~/.local was added as python 
already had a per-user directory on OSX: ~/Library/Python.

The common unix directories are often not a good pick for good OSX 
citizens, even if OSX is build on Unix.

The OSX filesystem structure should be described somewhere below 
<http://developer.apple.com/mac/library/documentation/MacOSX/Conceptual/
BPFileSystem/BPFileSystem.html#//apple_ref/doc/uid/10000185>, although I 
haven't found a clear explanation yet.

Technically configuration files should be stored in /Library/Preferences 
(system wide) or ~/Library/Preferences (per user), but I'd say 
practicality beats purity here and I'd store configuration in 
/Library/Python/etc or ~/Library/Python/etc. 

BTW. I haven't read most of the discussion yet, I'll probably have more 
to add when I do.  What I did notice was a proposal to store the 
distutils cfg file in ~/.local/lib/pythonX.Y/site-
packages/distutils.cfg. IMHO that way to hidden and an unexpected 
location for storing configuration files.

When ~/.local/lib was chosen for the user directory, the BaseDir Spec
was given as prior example. Why not go the full way and follow the spec?
The config file could be $XDG_CONFIG_HOME/python/distutils, with
$XDG_CONFIG_HOME defaulting to ~/.config.

References:
http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html
http://www.freedesktop.org/wiki/Software/pyxdg (deals with
freedesktop.org-approved file formats too; the part of the code that
deals with paths would make a useful addition to the stdlib, as more and
more applications follow the spec)

As a (mainly ex) windows user I would hate to have user editable data in APPDATA as it is not a location the user ever expects to visit. The home directory, or a subdirectory thereof, for user editable app specific data is more usual and more friendly.

I guess I misremembered. %APPDATA% is /usr/share, right? What I meant was the variable that expands to the “documents and settings” directory.

Pointed out by Tarek: http://github.com/ActiveState/appdirs#readme (shame it doesn‘t follow the BaseDir Spec on Free OSes).

I still think that user editable configuration data, at least on windows, is different from application data. It would definitely seem weird to me to have a config file in the %APPDATA% directory.

%APPDATA% would be fine for a cache or other application specific data.

On Windows I would expect something like: "c:\Documents and Settings\AppName"

(Note the settings in Documents and settings... - although in Windows 7 I think this might just be Documents now.)

> Well, that makes it the user-specific equivalent of /usr or /usr/local.
> Do you put your configuration files in /usr/local ? Why put them
> in .local ?

Yes, software built for /usr/local will often respect /usr/local/etc in addition to /etc.

Also, many desktop Linux applications put state into ~/.local/share/$appname.  I am not sure about "configuration files", (apps that use ~/.local will typically use ~/.config for configuration, but that is a different thing entirely) but autotools applications built with '--prefix ~/.local' will often respect ~/.local/etc.

> I agree with Ned that neither ~/.local nor /etc are a good fit for OSX, 
> sadly enough I wasn't paying attention when ~/.local was added as python 
> already had a per-user directory on OSX: ~/Library/Python.

I argued loudly for consistency in treatment of UNIX-like OSes when this decision was being made, and I stand by that position.  Why would you want to break software that works fine on BSD and Linux by requiring that it specifically test for OS X, and do the exact same thing, but in a different directory?  (See <http://en.wikipedia.org/wiki/Single_UNIX_Specification#Mac_OS_X_and_Mac_OS_X_Server>)

The addition of a per-user directory on OS X when there were no similar directories on other platforms was a weird oversight.  The user site directory in Python 2.5 and earlier does not use any features of OS X and refers to ~/Library/Python only by convention.

Plus, that convention didn't support the distutils properly anyway: if you put your library code into ~/Library/Python, where are you supposed to tell 'setup.py install' to put script files?  With ~/.local, it's obvious: ~/.local/bin.

The real OS X convention is to put your code _into your application bundle_, after all, not into random directories in ~/Library.  If you don't have an application or framework bundle, you have one foot in UNIX-land already.

The point of honoring platform conventions is to provide a consistent experience to users; some users on OS X will be expecting UNIX-like behavior, some will be expecting Framework-like behavior, and there is really no reason not to provide both.

There is no reason not to consider /etc, for that matter.  PHP, for example, still reads /etc/php.ini on OS X.

FWIW, I’m no longer strongly advocating using $XDG_CONFIG_HOME, since it seems to have little acceptance outside of graphical applications. Using it *or* ~/.python (or ~/.pythonx.y) would work for me.

Since this seems like a big change for no benefit in distutils, I’m reassigning to distutils2.

To push this issue forward, I am writing an email to python-dev today.

I’m retitling the bug, given its broader scope.

Michael Foord added another stdlib example: IDLE config files.

Python-dev thread: http://mail.python.org/pipermail/python-dev/2010-August/103011.html

It would be nice to settle this.

Personally I would like to see the defaults being:

Linux: ~/.pythonx.y
Mac OS X: ~/.pythonx.y with a fallback of ~/Library/Preferences/.pythonx.y
Windows: ~/pythonx.y perhaps with a backup of AppData/pythonx.y

For both Windows and Mac OS X I would be happy with the fallback / primary to be the other way round - it doesn't *really* matter. The API for getting the user config direction should always return a list I guess if we have fallbacks.

Someone else in the Python-dev discussion thread (Adal Chiriliuc) expressed a preferences for the documents folder on Windows over the home directory. This would be fine as well (with AppData still as a fallback). This is used by other native windows applications. (In earlier versions of Windows the documents folder was explicitly called "Documents and Settings".)

We should use pythonx.y rather than just python because (for example) you will typically have different packages installed when you have multiple versions of Python, and in unittest would then want / need different plugins enabled and configured in the unittest config files for each of the versions of Python.

I agree with Michael Foord; my comment on issue 8404 <http://bugs.python.org/issue8084#msg122935> may be of interest to anyone looking at this as well.

The issue proved more complicated than expected, due to antics of Mac OS X and Windows Vista, so I set it aside for later.  I intend to summarize all useful comments from the python-dev thread and bug reports comments and try to reach agreement for 3.3.

I'm mildly uncomfortable advocating many different new dot directories in $HOME (e.g. .python2.7 .python3.1 .python3.2).   Let's say -0.  Also, because of backward compatibility, I think most configuration files will end up being similar enough between Python versions.  Without having thought through all the details, it seems that ~/.python would work in general for configuration files that are compatible across Pythons, with X.Y subdirectories for version specific config files.

I would still prefer ~/.local/something-parallel-to-where-it's-usually-installed for its ease of documentation.  But ~/.python/X.Y isn't terribly objectionable.

A minor point about Michael Foord's fallback proposal: I don't know why everyone is so keen to stuff things into ~/Library on the mac, but dotfiles don't go in ~/Library/Preferences.  While that directory isn't exclusively populated with plist files, it's pretty close.  If you want to try for consistency with platform convention, 'org.python.distutils.cfg' would be a better name for it, but I've never hand-edited a text file in that directory either and I get the impression one isn't meant to.

But, since framework builds of Python already has ~/Library/Python/X.Y directories that they look in for other things, that location should probably be recycled if one wishes to have a fallback.

See also #12393.

no, sorry, but ~/.python is wrong on linux.

there exists a standard about where things belong:

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

it says that configuration stuff belongs into `XDG_CONFIG_HOME`, with a fallback to `~/.config/`

i propose to add the `appdirs` module to the stdlib so that programs finally don’t have to guess and hack around this kind of things.

https://pypi.python.org/pypi/appdirs

That standard is just wrong, dotfiles belong in $HOME :)  I'm only half joking, having grown up with Unix and having had the experience of flailing around for a while before I figured out a particular program's rc file was in .config instead of where I expected it to be, but I *am* half joking, because putting them all in .config is convenient. But most programs don't, so also looking in ~ makes sense to me.

I'm of mixed opinion.  I personally don't like cluttering up my $HOME with a
jillion dotfiles so I appreciate the organization XDG_CONFIG_HOME offers.  But
that also makes things less discoverable.  Looking in XDG_CONFIG_HOME first
with a fallback to $HOME seems like a workable compromise.

If this is about existing config files, then we should make sure we don't break existing installations. Also, multiple possible locations in the home dir are confusing.

If this is about new config files, then ~/.config/python/XXX is probably ok.

I'm not sure what XDG_CONFIG_HOME is but $XDG_CONFIG_HOME doesn't seem defined under my shell (Xubuntu 15.04).

XDG_CONFIG_HOME is not generally set to anything; the default of ~/.config is usually fine. However, the point is that you _can_ set it to point at a different location.  The relevant specification is here:

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

The biggest reason I strongly believe we should follow a specification is that Python is not the only software on the system.  If I am an ops person that has to deal with Python and Ruby and Java and C software all the time, *any* consistency is a big help.  We don't have to follow *this* specification, but we should follow *some* specification (although this specific one looks like a good one to me).  The Python community has no special expertise deciding on locations for configuration files, and so we should not be trying to invent a new and better place for them.

There are other advantages to following the XDG spec.  If we follow it correctly (and not, like Barry suggested, start adding random other directories like ~/.python), users can easily switch between configuration environments by switching the XDG environment variables (both XDG_CONFIG_DIRS and XDG_CONFIG_HOME), which would be a handy way of ignoring both user-specified config files _and_ system-specified ones, to get a pristine virtualenv-like separation between different config files.

Given that they are going to need to set these env vars anyway to redirect spec-compliant libraries that their application may be using, it would be nice to just have _one_ set of meta-configuration variables rather than one for Python and one for C and one for Ruby and so on.  Consider the fact that distutils respects "CFLAGS", and did not feel the need to invent "DISTUTILS_FLAGS_FOR_THE_C_COMPILER".

On Sep 01, 2015, at 07:15 PM, Glyph Lefkowitz wrote:

>There are other advantages to following the XDG spec.  If we follow it
>correctly (and not, like Barry suggested, start adding random other
>directories like ~/.python)

I was really just suggesting reading from other locations for backward
compatibility.

of course if there is a chance that some specific config file exists at a known location, it’s the only sane choice to try and look for it there iff it isn’t in its preferred location.

e.g. fontconfig made that switch some time ago: it used to be ~/.fonts.conf, now it’s ${XDG_CONFIG_HOME-$HOME/.config}/fontconfig/fonts.conf

the library will look at the new position and if it can’t find anything there, it looks at the old one.

but for new config files we should follow standards. on windows and OSX there are different ones: see the current discussion on python-ideas for specifics and problems. but on linux the only standard for this which i know is XDG-base-directories, so we should follow that for new config files.

The XDG standard seems to focus on desktop GUI applications and that's also where it's mostly used. Python has it's own installation scheme, which is documented at the top of sysconfig.py and which we've had ever since distutils became part of the stdlib.

For user installs, this bases all installations under the PYTHONUSERBASE dir, which defaults to ~/.local/ on Unix.

So user installs go into ~/.local/pythonX.X/ (I also have a ~/.local/python/X.X/ - not sure where that came from).

Python also uses ~/.pypirc for PyPI, ~/.python-eggs for setuptools, ~/.python_history for readline history.

The installation scheme currently doesn't define a dir for config files, only for "data" files, so adding one would probably be wise.

Since we already own ~/.local/pythonX.X/, why not put config files in there ?

For some extra Unix touch, we could use ~/.local/pythonX.X/etc/ and then move the cache bits to a ./var/ subdir (by defining another new "cache" entry for the schemes).

hi mark, i’ve just lengthily replied to you on python-ideas.

in short: nope! many command line tools that are relatively new (among them your examples git and pip) honor the specs, my ~/.cache, ~/.config, and ~/.local/share is full of things belonging to libraries, cli tools, my DE, and of course also GUI applications.

please let’s continue this discussion on python-ideas if you’re still not convinced ☺

On 02.09.2015 16:37, flying sheep wrote:
> 
> hi mark, i’ve just lengthily replied to you on python-ideas.
> 
> in short: nope! many command line tools that are relatively new (among them your examples git and pip) honor the specs, my ~/.cache, ~/.config, and ~/.local/share is full of things belonging to libraries, cli tools, my DE, and of course also GUI applications.
> 
> please let’s continue this discussion on python-ideas if you’re still not convinced ☺

I think we're mixing two discussions here:

1. How Python itself should install config files, data, etc.

2. How applications written using Python should get help
   from some module to determine where to install their
   config files, data, etc.

For 1. we already have a solution (in sysconfig.py). If you want to
change or amend this, I think a PEP is needed, since the consequences
will affect a lot of users.

For 2. we can use whatever convention or standard is en vogue today.
The module would have to pay close attention to providing backwards
compatibility right from the start, since these conventions are
bound to change over time.

Note that this ticket started with the location of the distutils
config file. The title has since been broadened way too much for
a ticket. I'd suggest closing it as out-of-date and going with
a proper PEP process or development of a new stdlib module
on python-ideas and -dev.

Given the experience with the platform module, I'm not convinced
that the Python release process is suitable to properly respond
to platform changes, so it may be better to simply have a
standard module on PyPI for 2.

you’re right about the two problems being mixed, however not about the standards.

also you’re intermingling conventions with actual standards.

the directory standards for all three big OSs are either not going to change or fitted with a backwards-compatibility layer (like windows did with the “My Music/Videos/*” folders):

https://developer.apple.com/library/mac/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/MacOSXDirectories/MacOSXDirectories.html

http://windows.microsoft.com/de-de/windows-8/what-appdata-folder

https://msdn.microsoft.com/en-us/library/windows/desktop/dd378457%28v=vs.85%29.aspx

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

all three OSs have stable, widely followed standards in place, and the idea of providing a python stdlib API for them received an almost unanimously positive response – with the sole exception being you.

On 03.09.2015 13:37, flying sheep wrote:
> all three OSs have stable, widely followed standards in place, and the idea of providing a python stdlib API for them received an almost unanimously positive response – with the sole exception being you.

I'm not against developing another module based on appdirs for this.

I do question the usefulness of having such a module in the stdlib,
since I've already gone through a similar experience with the platform
module I wrote several years ago. Ongoing development is not really possible
once a module is in the stdlib and older Python releases don't
benefit from new developments either.

Regarding standard vs. convention: that's all smoke and mirrors :-)
A standard that's not being adopted is not a standard. A convention
may well turn into a standard without ever being written down as
one. A module implementing any such standards or conventions will
have to address more than just one way of doing things in order
to stay compatible with existing software or new developments.
That's what I wanted to say.

But this ticket is not about that discussion anyway...

As someone else mentioned, the freedesktop standard is only really "widely followed" by desktop aps.  My own .config directory contains 8 entries, while there are 13 rc files/directories in my homedir (one of which is my X windows manager :)

That's not to mention the 50+ config files in my home directory that don't end with 'rc'....

just because people do something doesn’t mean it’s right.

i guess the tendency for CLI applications to do it wrong comes from

1. their ad-hoc beginnings. you usually start with one script file and extend it
2. availability expanduser('~') is in the stdlib and more well-known than e.g. appdirs, and people don’t like to add dependencies to small things
3. the shell and old, widely used applications using ~/.appname, so it’s more well-know to do that

on the other hand, GUI applications usually have a build system in place for UI file compilation, need a .desktop file installed, and so on, i.e. need more boilerplate and dependencies from the start, and people have to research how to do things properly. GUI libraries have their built-in standard dirs interfaces as well.

————————

another thing. on the python-ideas thread someone mentioned that

> I count 17 of those on my Windows machine (!) right now, including .idlerc, .ipython, .matplotlib, .ipylint.d etc.

this is horrible! this is so obviously the wrong thing to do. i hope having the module will reduce the number of similar atrocities.

The "tendency of CLI applications to do it wrong" comes from the fact that they are following the *older* unix (de-facto) standard, which is to put config files in the home directory as dot files.  That is, they are *not* doing it wrong, they are following the older unix de-facto standard and not the freedesktop.org standard (which, you will note, is a standard arising out of GUI applications, not CLI applications).

Not, mind, you, that I think .config is a bad standard, I'm just saying that if you want to follow standards you need to account for the legacy standard as well as the new standard.

This ticket did not reach a consensus.  It seems there would be little value in changing the existing scheme for IDLE and such; external tools can (and often do) already use appdirs; I suggest closing this.