classification
Title: Add '-p'/'--path0' command line option to override sys.path[0] initialisation
Type: Stage:
Components: Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, eric.araujo, eric.snow, gregory.p.smith, kristjan.jonsson, mattheww, ncoghlan, pitrou
Priority: normal Keywords: patch

Created on 2011-11-25 00:46 by ncoghlan, last changed 2014-04-18 18:26 by mattheww.

Files
File name Uploaded Description Edit
issue13475_1.diff eric.snow, 2012-05-26 19:01 review
issue13475_2.diff eric.snow, 2012-05-26 22:20 updated patch to address Antoine's comments review
issue13475_long_only.diff eric.snow, 2012-05-30 04:17 switch to long options review
Messages (31)
msg148295 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-11-25 00:46
PEP 395 spends a lot of time discussing ways that the current automatic initialisation of sys.path[0] can go wrong, and even the proposed improvements in that PEP don't claim to fix the default behaviour for every possible scenario (just many of the most common ones).

The unittest module gets around similar problems with test autodiscovery by providing an explicit "-t" (for 'toplevel') command line option.

While '-t' is not available for the main interpreter executable (and nor is '-d' for directory), '-p' for "path0" is a possibility.

Directory execution (for example) would then be equivalent to:

  python -p dirname dirname/__main__.py

A separate '--nopath0' option could also be provided to explicitly disable path initialisation based on the script being executed (see http://lists.debian.org/debian-python/2011/11/msg00058.html)
msg148305 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2011-11-25 06:09
+1  Both the -p and --nopath0 would be great additions and a good match for PEP 395.

So "-p ." would be equivalent to the current implicit sys.path[0] initialization, right?  Would any other effects happen with "-p" than sys.path[0] initialization?  I'll follow up with my one concern (the implicit "-p .") in a follow-up message.
msg148310 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2011-11-25 08:52
The current behavior is an implicit "-p .", which causes all sorts of hard-to-figure-out problems, most of which PEP 395 is rightly trying to fix.  I'm suggesting that the next step would be to make "--nopath0" the default (rendering the flag unnecessary).  Since this changes the status quo, I'll try to make the case here that it's worth it.

---

First of all, the current implicit sys.path[0] initialization for scripts has been around since forever.  Why would we change it now?  The inspiration is three-fold: we _said_ we got rid of implicit relative imports already, PEP 395 already has made a case for changing the status quo (though not to this degree), and a "-p" flag provides a simple mechanism to get the current behavior explicitly.

Like Nick says in PEP 395, "PEP 328 eliminated implicit relative imports from imported modules. This PEP proposes that the de facto implicit relative imports from main modules that are provided by the current initialisation behaviour for sys.path[0] also be eliminated."

I'm just saying we should go all the way, and that the "-p" flag would allow that.  As far as I can tell, there are two meaningful use cases for the currently implicit sys.path[0]:  When you are first _learning_ Python (not actually using it)...  When you are working on a new project...  Let's look at both, first relative to the currently implicit "-p .", then to the explicit site.

1. (current behavior) implicit sys.path[0] initialization

When a beginner is first learning Python, they learn at the interactive prompt or by running a single script.  Pretty quickly they learn about writing their own modules and using import to load them.

Where do the modules live?  In the CWD from which they are calling the Python executable.  The current implicit "-p ." means they can import their modules right there, and not have to learn yet about sys.path.  That's nice since they can concentrate their focus.

But later they _will_ run into hard-to-figure-out problems when they start relying on what are effectively implicit relative imports.  Or they will run into one of the problems that PEP 395 explains, which are likewise head-scratchers.  All of this can be pretty frustrating when you still don't know about things like the import machinery.

Consider how that beginner uses the implicit sys.path[0] behavior on a project, once they have some experience (hopefully I don't misrepresent something Nick explained to me).  The src/ directory of the project won't be under sys.path yet.  So to quickly test out their script, they change to src/ and run their script from there (something PEP 395 rightfully proposes fine-tuning).  They conveniently don't have to manually fiddle with sys.path.  That's a pretty neat trick.

The problem I have with this is that there's no difference between the sys.path workaround for the script and it's expected normal usage.  If you aren't aware of what's going on behind-the-scenes, you may end up with some confusing import exceptions later on.

2. explicit sys.path[0] initialization

With a -p flag, it's really easy to say "-p ." and get the previous implicit behavior.  Consider the impact on our beginner.  The try to run their script that imports modules in their CWD, but end up with an ImportError because we no longer have an implicit "-p .".

How is this an improvement?  No more mysterious default behavior.  There's only one way it will fail and it will fail the same way every time.  They'll learn pretty quickly (if not taught already) you have to use the -p flag if you want to import modules that aren't in the "standard" places.  The ImportError message, special-cased for imports in __main__, could help in that regard.

<aside>
As an aside, implicit or explicit "--nopath0", the path to the file for the failed import can really help in related situations.  Including it as an attribute on the ImportError, and especially in the error message, would rock.  See issue1559549.
</aside>

For the "src/" use-case, the developer won't be able to rely on the implicit behavior anymore.  A simple, explicit "-p ." fills the gap and makes it clear that they are doing something unusual.  On top of that, they can use -p with the path to the "src/" directory from anywhere and not have to change their directory.  All the better if -p takes advantage of the proposals in PEP 395.


In the end, this boils down to opt-in vs. opt-out.  Either you opt-in to the sys.path[0] initialization by using -p, or you opt-out of the default implicit "-p ." by passing "--nopath0".  It's a case of "explicit is better than implicit" without a clear reason, to me, that the status quo is worth the exception anymore in the face of the alternative.

My only concern would be backwards compatibility.  How much do scripts out there rely on the implicit sys.path[0] initialization that aren't already broken by the removal of implicit relative imports?  I simply don't know.

However, I do know that having the -p flag that Nick has recommended would be awesome.
msg148336 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-25 16:53
I don’t understand all the issues and it’s too late for me to read all the thread, but I hope these comments are helpful:

- If our goal is to help naïve users, then one or two new options wouldn’t help (people want to run “python path/to/thing.py” and want it to Just Work™)

- Getting a very helpful error message is second-best to Just Working™

- This probably needs a wider audience, python-ideas or python-dev, and maybe a PEP to explain all the issues and all the considered solutions.
msg148350 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2011-11-25 19:22
First of all, I don't want Nick's proposal in this issue (the -p and --nopath0 flags) to be misunderstood because of me.  His is a great idea that will make a really useful shortcut available and will _not_ change any current behavior.

My (overly) long message is an attempt to justify going with a more drastic variation, to which your response applies exclusively.  I stand by what I said, but I would be quite happy with the original proposal, to which your response does not apply.  :)

Just wanted to make all that clear so I don't get in the way of a good thing.  :)

Your comments are spot on.  Your concerns are exactly the ones I would need to assuage before my alternative would be acceptable.  When I get a chance I'll take this to python-ideas.
msg148378 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-11-26 05:19
Yeah, sorry Eric (Snow), you're trying to bite off *way* more than is reasonable by proposing to removing sys.path[0] auto-initialisation. While it has its problems, it's also far too entrenched in people's expections of Python's behaviour to consider removing at this late date. The earliest such a radical change could likely be considered is Python 4.

Éric (Araujo): the "just do the right thing" aspect is covered by PEP 395. This is an orthogonal proposal that allows power users the ability to override the automatic initialisation to handle those cases where it goes wrong.

That said, I may still make this suggestion part of PEP 395 rather than a distinct proposal, even though there's no direct dependency (in either direction) between this suggestion and the other ideas in that PEP.
msg148379 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2011-11-26 05:21
Yeah, the more I think about it, the more I agree it's Python 4 fodder.
msg148394 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-26 12:50
Eric: Actually I posted my comments after reading only Nick’s message, not yours, so no worry :)

Nick:
> the "just do the right thing" aspect is covered by PEP 395.
This is great.  Thanks for clarifying.

> This is an orthogonal proposal that allows power users the ability to override the
> automatic initialisation to handle those cases where it goes wrong.
If I’m understanding correctly, #12934 is one of those issues.  I agree that an option would be better than nothing, but 1) it would not help with older Pythons 2) it would not help naïve users 3) it would complicate the command-line interface of Python.  (Not saying I’m against the idea, just listing cons.)
msg148433 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-11-27 00:30
Zbigniew posted a nice summary of some of the issues sys.path[0] autoinitialisation can cause to python-dev: http://mail.python.org/pipermail/python-dev/2011-November/114668.html
msg148458 - (view) Author: Matthew Woodcraft (mattheww) Date: 2011-11-27 20:42
The proposed --nopath0 option is something I've wished I had in the past.

If this is added, it would be good if it could be given a single-letter form too, because it's an option that would be useful in #! lines (they don't reliably support using more than one command-line argument, and single-letter switches can be combined while long-form ones can't).
msg148552 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-11-29 10:48
I realised "-P" is available for use as the short form of the "suppress sys.path[0] initialisation" option.

So "-p" would override the calculation of sys.path[0], while "-P" would switch it off completely (similar to the way "-S" and "-E" switch off other aspects of the normal initialisation process).

Whether or not to provide "--path0" and "--nopath0" as long form alternatives would then be a separate question.
msg161676 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-05-26 18:45
Here's a stab at implementing -p/-P.  There are a couple warnings that I'm not sure about and I've undoubtedly missed some detail, but it should be pretty close.
msg161677 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-26 19:07
Before this is assigned a short option form, I would like to ask whether anybody but experts will be able to make a proper use of this option. 
(I also don't understand what it adds over PYTHONPATH)

As for the patch, it lacks error checking when calling C API functions.
msg161685 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-05-26 22:20
> Before this is assigned a short option form, I would like to ask
> whether anybody but experts will be able to make a proper use of this
> option.

Do you mean relative to a long form?  And what would constitute improper use for the option?

As Nick noted earlier in this issue, the implicit setting of sys.path[0] is a convenience for some cases, and I can appreciate that.  My complaint is that it is not an obvious behavior.  Furthermore, in an import system that has eliminated implicit relative imports it can lead to confusing behavior.  I consider Nick's proposal the best solution given the constraints we have.

If people do not use it, the status quo is unaffected.  If they do use it, they will be explicitly affecting sys.path[0].  If they use it without knowing what it's for, it will at worst cause imports to happen from unexpected sources (or not happen at all).  To me this doesn't seem that different from the way things are now when someone doesn't understand about sys.path[0] initialization.

> (I also don't understand what it adds over PYTHONPATH)

It provides an explicit alternative to the default implicit insertion to sys.path[0].  If the default behavior were no implicit initialization, then I'd agree that PYTHONPATH is sufficient.

> As for the patch, it lacks error checking when calling C API
> functions.

Ah, yes.  Spaced it.  Patch updated (and warnings fixed).

This brings me to ask what the behavior should be when we have errors come back from those C API functions?  In the patch I just have it fall back to the default sys.path[0] behavior.  However, wouldn't an error indicate a deeper problem?  If so, shouldn't Py_FatalError() be called?
msg161686 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-26 22:27
> > Before this is assigned a short option form, I would like to ask
> > whether anybody but experts will be able to make a proper use of this
> > option.
> 
> Do you mean relative to a long form?

Yes.

> And what would constitute improper use for the option?

As a replacement of PYTHONPATH or "setup.py install" or "setup.py
develop", I guess.

> As Nick noted earlier in this issue, the implicit setting of
> sys.path[0] is a convenience for some cases, and I can appreciate
> that.  My complaint is that it is not an obvious behavior.
> Furthermore, in an import system that has eliminated implicit relative
> imports it can lead to confusing behavior.

Agreed. I'm not arguing against the principle.

> > (I also don't understand what it adds over PYTHONPATH)
> 
> It provides an explicit alternative to the default implicit insertion
> to sys.path[0].  If the default behavior were no implicit
> initialization, then I'd agree that PYTHONPATH is sufficient.

Ah, indeed, I'd missed that.

> This brings me to ask what the behavior should be when we have errors
> come back from those C API functions?  In the patch I just have it
> fall back to the default sys.path[0] behavior.

Well, no, errors should not pass silently. The error should be
propagated. Here, it probably means print the error and abort (or
whatever strategy the rest of the function adopts).

> However, wouldn't an error indicate a deeper problem?  If so,
> shouldn't Py_FatalError() be called?

Py_FatalError() is a low-level exit; it dumps core. PyErr_Print()
followed by a proper exit may be better.
msg161689 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-05-26 23:55
did you mean to change the title?  this isn't about overriding sys.path, but rather just about explicitly dictating the initialization of sys.path[0].
msg161691 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-27 00:01
> did you mean to change the title?

No, I think it's a bug of the roundup e-mail gateway.
msg161695 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-05-27 06:28
FWIW, I now think this should *only* be a long option. Short options are precious, and this is an unusual enough use case that I'm not yet sure it deserves one.

In particular, we may decide to use "-p" later for adding directories to sys.path, rather than specifically overriding sys.path[0]
msg161696 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-05-27 06:29
FWIW, I now think this should *only* be a long option. Short options are precious, and this is an unusual enough use case that I'm not yet sure it deserves one.

In particular, we may decide to use "-p" later for adding directories to sys.path, rather than specifically overriding sys.path[0]
msg161749 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-05-28 02:03
Long options only would be fine with me.  So "--path0" and "--nopath0"?
msg161926 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-05-30 04:17
here's an updated patch with long options only.  I'm only 95% confident about my changes to getopt.c...there's probably a better way to do it.  However, the patch stands on its own two feet.
msg162677 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-06-12 14:25
any chance on this for 3.3?
msg162708 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-06-13 13:49
I was tempted to just add this (perhaps as a -X option) but, on reflection, I'm going to go with "No, not for 3.3".

I want to take a long hard look at the whole sys.path[0] initialisation process when I update PEP 395 to account for namespace packages, and it doesn't make sense to skip ahead with part of the conclusion. (I may even end up splitting that PEP into two pieces)

I'll also look into the embedding API, to see if/how embedding applications can fully control sys.path initialisation.

Python has survived without this feature for the last couple of decades, there's no need to be hasty in adding it now.
msg165164 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-07-10 05:25
Your plan sounds good, Nick.
msg168428 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-08-17 00:39
I'd also like a command line flag to override PYTHONPATH (which could also be used in combination with -E so that you could still set the PYTHONPATH while ignoring everything else).  I'll file a separate feature request for that.
msg176160 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-11-23 09:52
Butting in here:
Early on in the startup, a search is made for "site.py" using automatic sys.path settings.  Do the suggestions here propose to override this?  I know there is a -s flag, but all these flags start to be confusing.

I have been looking for ways to completely ignore automatic settings for sys.path, either from internal heuristics (either build environment or installed environment) or PYTHONPATH.
The reason for this is that as a developer in a large project, one that has many branches, each with their potentially different version of python compiled and with their own libraries, and where a "standard" python distribution may even be installed on the machine as well, we still want to be able to run scripts with each branch' own python with relative simplicity with some sort of guarantee that we are not inadvertently pulling in modules from other, unrelated branches, or god forbid, the "installed" python.

PEP 405 seemed to go a long way, although it didn't provide a means in the pyvenv.cfg to completely override sys.path from the startup.

I wonder if there isn't a synergy here with PEP 405, e.g. would it make sense to have a pyvenv.cfg file next to the __main__.py file?
msg176522 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-11-28 07:54
This proposal only affects the initialization of sys.path[0], and not any of the other sys.path entries made by site.py or otherwise.
msg176526 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-11-28 09:55
before or after running site.py?
msg176527 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-11-28 09:59
I'd suggest this would be applied at the same time as when sys.path[0] gets set now, but that kind of messy complication is why I decided we really needed to step back and start trying to clean up what we already have before we started layering yet *more* complications onto the initialisation process.
msg176529 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-11-28 10:33
Absolutely, Nick.
Perhaps we ought to start gathering a list of real world requirements, i.e. how do people run, or want to run, python?
msg216789 - (view) Author: Matthew Woodcraft (mattheww) Date: 2014-04-18 18:26
For the record: the '-I' option (#16499) in Python 3.4 disables sys.path[0] initialisation (among other things).
History
Date User Action Args
2014-04-18 18:26:06matthewwsetmessages: + msg216789
2012-11-28 10:33:59kristjan.jonssonsetmessages: + msg176529
2012-11-28 09:59:36ncoghlansetmessages: + msg176527
2012-11-28 09:55:55kristjan.jonssonsetmessages: + msg176526
2012-11-28 07:54:39eric.snowsetmessages: + msg176522
2012-11-23 09:52:39kristjan.jonssonsetmessages: + msg176160
2012-11-23 09:38:04kristjan.jonssonsetnosy: + kristjan.jonsson
2012-08-17 00:39:39gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg168428
2012-07-10 05:25:53eric.snowsetmessages: + msg165164
2012-06-17 05:49:30ncoghlansetassignee: ncoghlan ->
2012-06-13 13:49:16ncoghlansetassignee: ncoghlan
messages: + msg162708
versions: + Python 3.4
2012-06-12 14:25:42eric.snowsetmessages: + msg162677
2012-05-30 04:17:27eric.snowsetfiles: + issue13475_long_only.diff

messages: + msg161926
2012-05-28 02:03:19eric.snowsetmessages: + msg161749
2012-05-27 06:29:28ncoghlansetmessages: + msg161696
2012-05-27 06:28:20ncoghlansetmessages: + msg161695
2012-05-27 00:01:59pitrousettitle: Add '-p'/'--path0' command line option to override sys.path -> Add '-p'/'--path0' command line option to override sys.path[0] initialisation
2012-05-27 00:01:32pitrousetmessages: + msg161691
2012-05-26 23:55:11eric.snowsetmessages: + msg161689
2012-05-26 22:27:42pitrousetmessages: + msg161686
title: Add '-p'/'--path0' command line option to override sys.path[0] initialisation -> Add '-p'/'--path0' command line option to override sys.path
2012-05-26 22:20:32eric.snowsetfiles: + issue13475_2.diff

messages: + msg161685
2012-05-26 19:07:15pitrousetnosy: + pitrou
messages: + msg161677
2012-05-26 19:01:14eric.snowsetfiles: + issue13475_1.diff
2012-05-26 19:00:56eric.snowsetfiles: - issue13475_1.diff
2012-05-26 18:57:10eric.snowsetfiles: + issue13475_1.diff
2012-05-26 18:56:44eric.snowsetfiles: - issue13475_1.diff
2012-05-26 18:45:42eric.snowsetfiles: + issue13475_1.diff
keywords: + patch
messages: + msg161676
2011-11-29 10:48:09ncoghlansetmessages: + msg148552
2011-11-27 20:42:44matthewwsetnosy: + mattheww
messages: + msg148458
2011-11-27 00:30:41ncoghlansetmessages: + msg148433
2011-11-26 23:33:12Arfreversetnosy: + Arfrever
2011-11-26 12:50:02eric.araujosetmessages: + msg148394
2011-11-26 05:21:26eric.snowsetmessages: + msg148379
2011-11-26 05:19:47ncoghlansetmessages: + msg148378
2011-11-25 19:22:05eric.snowsetmessages: + msg148350
2011-11-25 16:53:02eric.araujosetnosy: + eric.araujo
messages: + msg148336
2011-11-25 08:53:01eric.snowsetmessages: + msg148310
2011-11-25 06:09:29eric.snowsetmessages: + msg148305
2011-11-25 02:01:48eric.snowsetnosy: + eric.snow
2011-11-25 00:46:33ncoghlancreate