Title: write a configure command
Type: enhancement Stage: resolved
Components: Distutils2 Versions: 3rd party
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: merwok Nosy List: jkloth, josip, l0nwlf, meatballhat, merwok, pitrou, schmir, tarek, titus, zubin71
Priority: normal Keywords: gsoc, needs review, patch

Created on 2010-03-28 14:41 by tarek, last changed 2014-03-13 07:57 by merwok. This issue is now closed.

Messages (18)
msg101864 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-03-28 14:41
This command will contain all options that are used to build extensions (out of the build* and install* commands) and will create a confifuration file.

build* and install* commands will be able to read back the file if present, and use its values, so we don't have to pass the options again.

See 4Suite configure command for an example
msg106460 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-05-25 18:21
My review of 4Suite’s config command is on <>. Comments welcome.

(Adding my mentor to the nosy list.)
msg106534 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-05-26 15:28
Distutils2 already has a config command used to configure some aspects of C compilation. Not sure if I should expand that or write an unrelated configure command.
msg106536 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-05-26 15:37
start a new command from scratch. The configure command is not dealing but with generating a configuration file.

config is implementing a whole compilation chain.
msg108385 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-06-22 14:31
I have a working configure command at which works. (Read my short reports on for more details.) It’s not completely finished, though:

1) I import build and install to get their user_options, to avoid duplicating these lines. Is that bad?

2) Test coverage is not exhaustive. This will be done incrementally.

3) I tried directly using Distribution and configure objects in the tests, but couldn’t get them to work; I temporarily settled for running Python (sys.executable) in a subprocess. I’m looking for help to change that.

4) As it is now, the configure command just records options without checking for conflicts (e.g. using both –prefix and –user for install). While not in the original description of the task, I think that it should be added. Prior art like 4Suite’s config command or a prototype for configure done by Tarek that I discovered only today support this idea. Is that okay? I’ll have to avoid code duplication between configure, build and install.
msg108449 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-06-23 13:34
5) Should the command write different files per Python version and platform? 4Suite’s config command does that.
msg108452 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-23 14:25
Shouldn't "config" and "configure" have more distinguishable names?
(if I understand correctly, they both are distutils2 commands)
msg108454 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-06-23 14:42
4Suite’s DistExt has a config command similar to the new Distutils2 configure command described in this bug report. The Distutils config command is an unfinished command with use similar to autotools configure scripts. From the module docstring:

“a (mostly) empty command class that exists mainly to be sub-classed by specific module distributions and applications.  The idea is that while every "config" command is different, at least they're all named the same, and users always see "config" in the list of standard commands.  Also, this is a good place to put common configure-like tasks: "try to compile this C code", or "figure out where this header file lives"”.

The module has seen 26 commits in 10 years. I find the name clash confusing too for end users and wanted to bring the issue later. If I understand autotools correctly, it encompasses both config and configure commands. Since there are no conflicts in options or functionality ( is a no-op, writes a file), I think dropping config or merging it with configure would make a lot of sense. But first things first, configure needs to be reviewed and accepted (5 open questions :) and then I’ll propose a merge into a unified config command (short names win).
msg110601 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-07-17 21:02
Quick update: I have done a basic part of this, but Tarek explained to me the real usefulness of this command, i.e. provide build and installation options for third-party code (for the same purposes that get_config_var in Python’s sysconfig). I am currently working on #8252 but do not forget this one.
msg110602 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-07-17 21:09
Tarek also answered my questions.

1) 4) build and install options should be defined and checked only once, in configure. build and install would inherit from this class and get their option from it.

3) I switched to using distutils2.core.run_setup, which Tarek said is okayish. Understanding why the usual idiom of using the classes directly doesn’t work will be better.

5) Since the cache file (the file that contains options, written by configure) is tied to a precise build, it should be specific. I can put the platform and Python version in the filename, or maybe put the file in the platform-specific build directory. Not sure about the naming in dists. I still have to think about the overlap between setup.cfg and configure.cache.
msg110622 - (view) Author: Jeremy Kloth (jkloth) * Date: 2010-07-18 01:58
WRT #5, it would be possible to use section headers within the configure.cache file to indicate a particular "platform build".  Very much like how 4Suite's `config` command stored the cache file.
msg110929 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-07-20 17:31
Adding message from jkloth removed by error (remove/reply confusion):

for #1 instead of importing `build`/`install` commands for their options, why not define the options only on the `configure` command?  The B/I commands could just "lookup" their needed options from the configure command.

On that note, how does the B/I commands get the options defined by `configure`? A quick look at the code doesn't show them using the "configured" options.
msg110931 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-07-20 17:38
> why not define the options only on the `configure` command?

This is what Tarek told me and what I thought I said two messages earlier.

> On that note, how does the B/I commands get the options defined by `configure`?

See and line 384. Nothing to change in the commands, just make Distribution pick values from the cache file at the right time (after regular config files and before command-line options).
msg110935 - (view) Author: Jeremy Kloth (jkloth) * Date: 2010-07-20 17:49
How would the build or install command get the options in this case:

python configure --prefix=$HOME build install

Remember that commands can be chained (and that commands can run other commands).  I would like to be able to do (and can do right now):

python install

and have both `configure` and `build` be run with their defaults.
msg110940 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-07-20 18:13
configure is not config. As I understand it, the config command aimed to be something like ./configure in autotools, thus usable in combination with build and install. configure is different: it is meant to be run once to create a configure.cache file which saves possibly long and complicated options. IOW, its goal was to be typed only once, not in combination with build or install. This is why build does not call configure. I have written a short doc to explain that, see configure.rst in my repo or the built version at

This message and the linked doc explain the current behavior of configure as I understood its purpose a month ago, IMO respecting what Tarek wrote in the bug report. However, Tarek he did not explain all that he had in mind here, like I wrote here on 2010-07-17 (I don’t have notes on that :/ so I’ll have to ask him again and sort out the setup.cfg/configure.cache overlap). When I get back to configure, the command line “configure --prefix ~ install” will work according to your expectations.
msg117686 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-09-30 01:33
Status update.

* configure works: it checks options (like install does) and writes to a file.

* New design for build: It uses set_undefined_options to get options from configure (previously it read the cache file, but this caused nasty issues; I’ll document those designs some day, it was instructional for me).  All tests pass.

* Same thing for install, but it’s harder to make it work.  install.finalize_options checks for conflicts (for example you can’t set --user and --prefix=x) and then fills all options (so prefix gets a value).  Now configure has the same checks, so that it does not write non-working options to the cache file.  The problem is that all options are set in configure.finalize_install_options and then passed to install with set_undefined_options, and then install.finalize_options shouts at you because you can’t set all options.  I tried a simple and stupid fix for that (disable option conflict check when options come from configure), which fixed all configure tests but caused test_core to fail.  This means that while the configure tests passed, they were only superficially working; I have to be suspicious to my own tests and out general test coverage.
msg117687 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-09-30 01:33
(BTW I haven’t pushed the work in progress yet but I will shortly.)
msg121265 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-11-16 06:36
I have turned my branch into a clean patch for default:


- You can diff against and from an earlier revision to see what code has been moved where.

- Two tests for install_dist still fail, but this could be only the tip of the iceberg.  We need more tests there.  Installation paths is a tricky business, and we have to make sure configure does not break anything.

- I have changed install to install_dist in the code, tests and docs, but only in the part of the docs I added for configure.  The rest of the file is inaccurate, but that’s outside the scope of this patch.

- Some to-do features: distribute the configure.cache file in the dist-info directory (requires adding a hook system to install_distinfo); add info about the compiler type; probably more.
Date User Action Args
2014-08-04 18:20:00merwokunlinkissue1011113 dependencies
2014-03-13 07:57:42merwoksetstatus: open -> closed
resolution: accepted -> out of date
stage: patch review -> resolved
2010-11-16 06:37:13merwoksetkeywords: + patch, needs review
2010-11-16 06:36:41merwoksetmessages: + msg121265
2010-09-30 01:33:59merwoksetmessages: + msg117687
2010-09-30 01:33:07merwoksetmessages: + msg117686
versions: + 3rd party, - Python 2.6, Python 2.5, Python 3.1, Python 2.7, Python 3.2
2010-08-20 23:08:51merwoklinkissue1011113 dependencies
2010-07-20 18:13:29merwoksetmessages: + msg110940
2010-07-20 17:49:46jklothsetmessages: + msg110935
2010-07-20 17:38:08merwoksetmessages: + msg110931
2010-07-20 17:31:56merwoksetmessages: + msg110929
2010-07-20 17:30:52merwoksetmessages: - msg110928
2010-07-20 17:28:48jklothsetmessages: + msg110928
2010-07-18 01:58:37jklothsetmessages: + msg110622
2010-07-17 21:09:23merwoksetmessages: + msg110602
2010-07-17 21:02:17merwoksetassignee: tarek -> merwok
messages: + msg110601
2010-06-23 14:42:27merwoksetmessages: + msg108454
2010-06-23 14:25:11pitrousetnosy: + pitrou
messages: + msg108452
2010-06-23 13:34:16merwoksetmessages: + msg108449
2010-06-22 14:31:05merwoksetversions: - Python 3.3
resolution: accepted
messages: + msg108385

type: enhancement
stage: patch review
2010-05-26 16:08:54schmirsetnosy: + schmir
2010-05-26 15:41:53jklothsetnosy: + jkloth
2010-05-26 15:37:10tareksetmessages: + msg106536
2010-05-26 15:28:22merwoksetmessages: + msg106534
2010-05-25 18:21:45merwoksetkeywords: + gsoc
nosy: + titus
messages: + msg106460

2010-05-01 01:40:10meatballhatsetnosy: + meatballhat
2010-04-01 13:05:18josipsetnosy: + josip
2010-03-29 17:34:41zubin71setnosy: + zubin71
2010-03-28 18:53:19l0nwlfsetnosy: + l0nwlf
2010-03-28 14:45:07merwoksetnosy: + merwok
2010-03-28 14:41:23tarekcreate