[.]pydistutils.cfg will be deprecated in favor of a single
"distutils.cfg", located in .local/ alongside the files added for
example by the per user-site packages.

Why is this? .local/ contains machine-installed files while
.pydistutils.cfg is edited by the user.

That's what is returned by site.getuserbase()

(which uses PYTHONUSERBASE, and defaults to ~/.local)

PYTHONUSERBASE is the root of Python user-specific paths so it makes
sense to have all Python related files in there.

This came up with the discussion we had with others at Distutils-SIG

read this thread here for more info:
http://mail.python.org/pipermail/distutils-sig/2009-August/012865.html

[.]pypirc should probably go there too.

> That's what is returned by site.getuserbase()
> 
> (which uses PYTHONUSERBASE, and defaults to ~/.local)

Well, that makes it the user-specific equivalent of /usr or /usr/local.
Do you put your configuration files in /usr/local ? Why put them
in .local ?

> PYTHONUSERBASE is the root of Python user-specific paths so it makes
> sense to have all Python related files in there.

That doesn't make sense actually, since .local isn't Python-specific. :)

I've read the thread now and I think the original proposal of having a
~/.python was more sensible and more user-friendly.

+1 to what Antoine said.

> Well, that makes it the user-specific equivalent of /usr or /usr/local.
> Do you put your configuration files in /usr/local ? 
> Why put them in .local ?

The distutils.cfg file is located in the distutils package itself !

So the target is ~/.local/pythonX.X/site-packages/distutils/distutils.cfg

(sorry if that was unclear)

Now, I would totally agree to have a better ~/.NAME to reunite
user-specific *configuration* file for Python, but that would be yet
another directory for user-specific Python files.

> The distutils.cfg file is located in the distutils package itself !
> 
> So the target is ~/.local/pythonX.X/site-packages/distutils/distutils.cfg

Ah, that's horrible. -1 to that.
Configuration files should be easily findable by skimming through the
dot-files in the user's directory. ~/.python fulfills that condition.

I like ~/.python very much too (that was my initial proposal IIRC). 

I guess the best way to add this support is to provide a new api in
site.py, that returns this folder.

Distutils can move pypirc and distutils.cfg file there.

Now for the global distutils.cfg, if we want to have it somewhere else
that the distutils package directory itself, we would need a
"/etc/python" scheme for instance, that could be also added in site.py.

I don't know though how the latter would look under windows

Tarek Ziadé wrote:
> 
> Tarek Ziadé <ziade.tarek@gmail.com> added the comment:
> 
> I like ~/.python very much too (that was my initial proposal IIRC). 

+1 on that name as well.

One thing to note though: it is well possible that a user uses
multiple Python versions. With just one such directory, all
versions would look in the same directory for the configuration
files and this could lead to incompatibilities, e.g. Python 2.7
might not like that Python 3.6 needs in some config file (for
whatever reason).

> Now for the global distutils.cfg, if we want to have it somewhere else
> that the distutils package directory itself, we would need a
> "/etc/python" scheme for instance, that could be also added in site.py.
>
> I don't know though how the latter would look under windows

I think Subversion get's this pretty right, so we might want
to follow their scheme:

http://svnbook.red-bean.com/en/1.1/ch07.html#svn-ch-7-sect-1

The subversion system looks pretty nice !

So here's a modified proposal that covers the python version as well.

= per user configuration files = 

 ~/.pythonMAJOR.MINOR

Examples:

~/.python2.7
~/.python3.1

On Windows, ~ will be replaced by APPDATA if founded in the environment.
 
== global configuration files ==


On Linux/Mac:

/etc/pythonMAJOR.MINOR

examples:

/etc/python2.7/
/etc/python3.1/

On Windows:

\All users\Application Data\PythonMAJORMINOR

(\All users\Application Data is specified by the Windows Registry)

examples:

\All users\Application Data\Python26
\All users\Application Data\Python31

== standard naming for configuration files ==

The files are all named "NAME.cfg", in all platforms.

== what gets in per-user configuration directory ==

- pypi.cfg
- distutils.cfg

== what gets in global configuration directory ==

- distutils.cfg

== apis added ==

in site.py:

- getuserconfig() : returns the per-user configuration directory
- getconfig(): returns the global configuration directory

(the names are following site.py naming style)

FYI, /etc/pythonX.Y (and /etc/python) are used by Debian; the former
contains site.py and sitecustomize.py, the latter a file called
debain_config.

I can't see how this could become a problem, but maybe someone else does.

Thanks for the info Georges.

I am adding Matthias and Michael so they can give us their opinions for
their platforms (Debian/Ubuntu and Win)

Personally I'm uncomfortable with creating
yet-another-location-for-config-files.

As we now have is ~/.local/pythonX.Y I would reuse this.

But ~/.local is simply not the location for config files.

For Linux and Mac OS X put the config file wherever the 'right' place is
then, but we are starting to create a lot of new directories disparate
from the actual install. Oh well.

On Windows we create: %APPDATA%/Python/PythonX.Y/site-packages

If it is to no longer live inside distutils (which seems reasonable)
then on Windoze a 'good' place would seem to be: %APPDATA%/Python/PythonX.Y/

On Windows there is no 'right' place, the officially right place for the
system is probably the registry but I don't think many Python
programmers would thank you for that...

Noting of course that on IronPython it should be:
%APPDATA%/IronPython/PythonX.Y/

or wherever we decided in the end. :-)

/etc is definitely not the right place to put files for OS X framework 
builds; if necessary, an etc directory could be added under the 
framework version directory as a sibling of bin and lib.  It's also very 
un-OS X like to be putting things into ~/.python and ~/.local 
directories; the usual place would be in somewhere ~/Library, possibly 
~/Library/Application Support/Python or ~/Library/Frameworks.  Keep in 
mind that it's much more likely on OS X to not only have muitiple 
versions of Python installed but also more than one instance of the 
*same* version, for instance, on 10.6, an Apple-supplied 2.6.2 and a 
python.org 2.6.4.  Some thought should be given to locations for the 
files for Unix-style (non-framework) builds.

(Adding Ronald.)

Ned Deily wrote:
> 
> Ned Deily <nad@acm.org> added the comment:
> 
> /etc is definitely not the right place to put files for OS X framework 
> builds; if necessary, an etc directory could be added under the 
> framework version directory as a sibling of bin and lib.  It's also very 
> un-OS X like to be putting things into ~/.python and ~/.local 
> directories; the usual place would be in somewhere ~/Library, possibly 
> ~/Library/Application Support/Python or ~/Library/Frameworks.  Keep in 
> mind that it's much more likely on OS X to not only have muitiple 
> versions of Python installed but also more than one instance of the 
> *same* version, for instance, on 10.6, an Apple-supplied 2.6.2 and a 
> python.org 2.6.4.  Some thought should be given to locations for the 
> files for Unix-style (non-framework) builds.
> 
> (Adding Ronald.)

I think there is a misunderstanding here: we're trying to find
places where distutils would look by default, not where it
would write files.

AFAIK, there is general agreement that when writing files,
Python should use the ~/.local/pythonX.X/ directory.

OTOH, searching for config files that the user creates and
puts in place is another story. Python and distutils only
need to be able to find these and the idea is to use platform
specific standard search paths for this, so that the user
can place those config files into places that feel right on
each platform.

I don't think there's a misunderstanding.  By "putting", I meant "reading" 
or "writing".  IMO, /etc is not the place on OS X to be looking for 
python-related configuration files, certainly not for framework installs.  
Likewise for ~/.python and ~/.local.  Unfortunately, the latter is already 
out in the field; that was a step in the wrong direction for OS X.

I agree with Ned that neither ~/.local nor /etc are a good fit for OSX, 
sadly enough I wasn't paying attention when ~/.local was added as python 
already had a per-user directory on OSX: ~/Library/Python.

The common unix directories are often not a good pick for good OSX 
citizens, even if OSX is build on Unix.

The OSX filesystem structure should be described somewhere below 
<http://developer.apple.com/mac/library/documentation/MacOSX/Conceptual/
BPFileSystem/BPFileSystem.html#//apple_ref/doc/uid/10000185>, although I 
haven't found a clear explanation yet.

Technically configuration files should be stored in /Library/Preferences 
(system wide) or ~/Library/Preferences (per user), but I'd say 
practicality beats purity here and I'd store configuration in 
/Library/Python/etc or ~/Library/Python/etc. 

BTW. I haven't read most of the discussion yet, I'll probably have more 
to add when I do.  What I did notice was a proposal to store the 
distutils cfg file in ~/.local/lib/pythonX.Y/site-
packages/distutils.cfg. IMHO that way to hidden and an unexpected 
location for storing configuration files.

When ~/.local/lib was chosen for the user directory, the BaseDir Spec
was given as prior example. Why not go the full way and follow the spec?
The config file could be $XDG_CONFIG_HOME/python/distutils, with
$XDG_CONFIG_HOME defaulting to ~/.config.

References:
http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html
http://www.freedesktop.org/wiki/Software/pyxdg (deals with
freedesktop.org-approved file formats too; the part of the code that
deals with paths would make a useful addition to the stdlib, as more and
more applications follow the spec)

As a (mainly ex) windows user I would hate to have user editable data in APPDATA as it is not a location the user ever expects to visit. The home directory, or a subdirectory thereof, for user editable app specific data is more usual and more friendly.

I guess I misremembered. %APPDATA% is /usr/share, right? What I meant was the variable that expands to the “documents and settings” directory.

Pointed out by Tarek: http://github.com/ActiveState/appdirs#readme (shame it doesn‘t follow the BaseDir Spec on Free OSes).

I still think that user editable configuration data, at least on windows, is different from application data. It would definitely seem weird to me to have a config file in the %APPDATA% directory.

%APPDATA% would be fine for a cache or other application specific data.

On Windows I would expect something like: "c:\Documents and Settings\AppName"

(Note the settings in Documents and settings... - although in Windows 7 I think this might just be Documents now.)

> Well, that makes it the user-specific equivalent of /usr or /usr/local.
> Do you put your configuration files in /usr/local ? Why put them
> in .local ?

Yes, software built for /usr/local will often respect /usr/local/etc in addition to /etc.

Also, many desktop Linux applications put state into ~/.local/share/$appname.  I am not sure about "configuration files", (apps that use ~/.local will typically use ~/.config for configuration, but that is a different thing entirely) but autotools applications built with '--prefix ~/.local' will often respect ~/.local/etc.

> I agree with Ned that neither ~/.local nor /etc are a good fit for OSX, 
> sadly enough I wasn't paying attention when ~/.local was added as python 
> already had a per-user directory on OSX: ~/Library/Python.

I argued loudly for consistency in treatment of UNIX-like OSes when this decision was being made, and I stand by that position.  Why would you want to break software that works fine on BSD and Linux by requiring that it specifically test for OS X, and do the exact same thing, but in a different directory?  (See <http://en.wikipedia.org/wiki/Single_UNIX_Specification#Mac_OS_X_and_Mac_OS_X_Server>)

The addition of a per-user directory on OS X when there were no similar directories on other platforms was a weird oversight.  The user site directory in Python 2.5 and earlier does not use any features of OS X and refers to ~/Library/Python only by convention.

Plus, that convention didn't support the distutils properly anyway: if you put your library code into ~/Library/Python, where are you supposed to tell 'setup.py install' to put script files?  With ~/.local, it's obvious: ~/.local/bin.

The real OS X convention is to put your code _into your application bundle_, after all, not into random directories in ~/Library.  If you don't have an application or framework bundle, you have one foot in UNIX-land already.

The point of honoring platform conventions is to provide a consistent experience to users; some users on OS X will be expecting UNIX-like behavior, some will be expecting Framework-like behavior, and there is really no reason not to provide both.

There is no reason not to consider /etc, for that matter.  PHP, for example, still reads /etc/php.ini on OS X.

FWIW, I’m no longer strongly advocating using $XDG_CONFIG_HOME, since it seems to have little acceptance outside of graphical applications. Using it *or* ~/.python (or ~/.pythonx.y) would work for me.

Since this seems like a big change for no benefit in distutils, I’m reassigning to distutils2.

To push this issue forward, I am writing an email to python-dev today.

I’m retitling the bug, given its broader scope.

Michael Foord added another stdlib example: IDLE config files.

Python-dev thread: http://mail.python.org/pipermail/python-dev/2010-August/103011.html

It would be nice to settle this.

Personally I would like to see the defaults being:

Linux: ~/.pythonx.y
Mac OS X: ~/.pythonx.y with a fallback of ~/Library/Preferences/.pythonx.y
Windows: ~/pythonx.y perhaps with a backup of AppData/pythonx.y

For both Windows and Mac OS X I would be happy with the fallback / primary to be the other way round - it doesn't *really* matter. The API for getting the user config direction should always return a list I guess if we have fallbacks.

Someone else in the Python-dev discussion thread (Adal Chiriliuc) expressed a preferences for the documents folder on Windows over the home directory. This would be fine as well (with AppData still as a fallback). This is used by other native windows applications. (In earlier versions of Windows the documents folder was explicitly called "Documents and Settings".)

We should use pythonx.y rather than just python because (for example) you will typically have different packages installed when you have multiple versions of Python, and in unittest would then want / need different plugins enabled and configured in the unittest config files for each of the versions of Python.

I agree with Michael Foord; my comment on issue 8404 <http://bugs.python.org/issue8084#msg122935> may be of interest to anyone looking at this as well.

The issue proved more complicated than expected, due to antics of Mac OS X and Windows Vista, so I set it aside for later.  I intend to summarize all useful comments from the python-dev thread and bug reports comments and try to reach agreement for 3.3.

I'm mildly uncomfortable advocating many different new dot directories in $HOME (e.g. .python2.7 .python3.1 .python3.2).   Let's say -0.  Also, because of backward compatibility, I think most configuration files will end up being similar enough between Python versions.  Without having thought through all the details, it seems that ~/.python would work in general for configuration files that are compatible across Pythons, with X.Y subdirectories for version specific config files.

I would still prefer ~/.local/something-parallel-to-where-it's-usually-installed for its ease of documentation.  But ~/.python/X.Y isn't terribly objectionable.

A minor point about Michael Foord's fallback proposal: I don't know why everyone is so keen to stuff things into ~/Library on the mac, but dotfiles don't go in ~/Library/Preferences.  While that directory isn't exclusively populated with plist files, it's pretty close.  If you want to try for consistency with platform convention, 'org.python.distutils.cfg' would be a better name for it, but I've never hand-edited a text file in that directory either and I get the impression one isn't meant to.

But, since framework builds of Python already has ~/Library/Python/X.Y directories that they look in for other things, that location should probably be recycled if one wishes to have a fallback.

See also #12393.