Title: -W option and PYTHONWARNINGS env variable does not accept module regexes
Type: Stage: patch review
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Yongjik Kim, blueyed, coldfix, kernc, mpaolini, ncoghlan, nedbat, xtreak
Priority: normal Keywords: patch

Created on 2018-09-10 23:36 by coldfix, last changed 2020-07-15 09:50 by nikratio.

Pull Requests
URL Status Linked Edit
PR 9358 open coldfix, 2018-09-17 12:43
Messages (11)
msg324959 - (view) Author: Thomas Gläßle (coldfix) * Date: 2018-09-10 23:36

This command does not report a warning, while it should:

  python -c 'import warnings; warnings.warn("This should show up")' -Wi -W'default:::.*'

If the regex `.*` is replaced by `__main__` it works as expected.

Same applies for regexes in PYTHONWARNINGS and for the `message` part of the argument. 

The reason can be found in Lib/ (def _setoption):

  module = re.escape(module)

This point-blank escape makes me think that it was intended that no regexes can be passed to message/module. On the other, the documentation reads as if it should be supported.

Specifically, the -W option is documented in [1]. While this page lists only basic examples, it refers to [2] and [3] for more details. [2] states that message/module are regexes. [3] seems to be written to specifically address the syntax of the PYTHONWARNINGS and the -W option and explicitly lists an example with a regex.


I would welcome if we could remove `re.escape` to make the implementation fit the documentation, or are there any downsides to this?

Best regards, Thomas
msg324960 - (view) Author: Thomas Gläßle (coldfix) * Date: 2018-09-10 23:39
Very sorry, the example command above should read:

  python -Wi -W'default:::.*' -c 'import warnings; warnings.warn("This should show up")'
msg324997 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-09-11 08:42
The original escape was done with 2a862c614d6d3ff50dc644150670651fdc9f3a99 which was in 2000. The doc example you are referring to at [1] was added with and there doesn't seem to be a test involving regular expression as I can see from the PR. Adding Nick to the issue since he had committed the doc and might help here.


msg325538 - (view) Author: Thomas Gläßle (coldfix) * Date: 2018-09-17 12:43
Thanks for your response. I have opened a PR at [1] that would remove the re.escape such that the implementation matches the documentation if you decide that this is fine.


Personally, I would go even further and always add the `(\..*)?$` suffix (instead of only `$`) to make it easier to match all modules in a package which is probably the most important use-case for the case of packages. What do you think?
msg328648 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2018-10-27 13:46
Another option is to make clear in the docs that the command line does not accept regular expressions, but only literal module names, and raise an error if a non-importable name (for example with asterisks in it) is specified.

And while we are here: the docs show "error:::mymodule[.*]" which doesn't even make sense as a regex!
msg328650 - (view) Author: Thomas Gläßle (coldfix) * Date: 2018-10-27 14:10
Hi, thanks for the consideration!

Is there any reason to introduce different behaviour than with filterwarnings here? This increases the number of things to remember - and except for the dot regex syntax doesn't interfere with module names, so there is no big drawback here either.

More importantly, my main use case for filterwarnings would be to select/ignore warnings based on module *or package* name. With the current interpretation as exact module name, you have to list all submodules in advance, which is quite inhibiting.

I have fixed the `[.*]` bogus example in the PR.

Best, Thomas
msg336646 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-02-26 11:15
I think the only reason I didn't mention this discrepancy in my doc updates is because I wasn't aware there *was* a discrepancy.

The weird syntax was then just an incorrect amalgamation of "optional argument" notation with an improperly escaped regex suffix.
msg348202 - (view) Author: daniel hahler (blueyed) * Date: 2019-07-19 23:03
Note that "module" might also be the filename (with ".py" removed).

I've just created issue37634 - might be worth including in your PR if it makes sense to only document this.
msg373358 - (view) Author: Marco Paolini (mpaolini) * Date: 2020-07-08 22:47
hello Thomas,

do you need any help fixing the conflicts in your PR?

even if Lib/ changed a little in the last 2 years, your PR is still good!
msg373536 - (view) Author: Thomas Gläßle (coldfix) * Date: 2020-07-11 19:56
Hi, I have rebased this on master and fixed the minor conflict. Let me know if there is anything else I can do.

Best, Thomas
msg373669 - (view) Author: Yongjik Kim (Yongjik Kim) Date: 2020-07-15 06:45
Hi, sorry if I'm interrupting, but while we're at this, could we also not escape regex for "message" part?  (Or at least amend the documentation to clarify that the message part is literal string match?)

Currently, the docs on -W just say "The meaning of each of these fields is as described in The Warnings Filter" and then "The Warnings Filter" section says that the "message" field is a regex, but currently it's only true if you run warnings.filterwarnings() directly, and not if you use the -W option.
Date User Action Args
2020-07-15 09:50:06nikratiosetnosy: - nikratio
2020-07-15 06:45:50Yongjik Kimsetnosy: + Yongjik Kim
messages: + msg373669
2020-07-11 19:56:10coldfixsetmessages: + msg373536
versions: + Python 3.9, Python 3.10
2020-07-08 22:47:37mpaolinisetnosy: + mpaolini
messages: + msg373358
2019-07-19 23:03:49blueyedsetnosy: + blueyed
messages: + msg348202
2019-02-26 11:15:29ncoghlansetmessages: + msg336646
2018-12-08 18:35:08kerncsetnosy: + kernc
2018-10-27 14:10:48coldfixsetmessages: + msg328650
2018-10-27 13:46:09nedbatsetnosy: + nedbat
messages: + msg328648
2018-10-07 13:02:32nikratiosetnosy: + nikratio

title: -W option does not accept module regexes -> -W option and PYTHONWARNINGS env variable does not accept module regexes
2018-10-07 12:26:54martin.panterlinkissue34920 superseder
2018-09-17 12:43:34coldfixsetmessages: + msg325538
2018-09-17 12:43:02coldfixsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8781
2018-09-11 08:42:50xtreaksetnosy: + ncoghlan
messages: + msg324997
2018-09-11 08:15:25xtreaksetnosy: + xtreak
2018-09-10 23:39:14coldfixsetmessages: + msg324960
2018-09-10 23:36:34coldfixcreate