This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: -W option and PYTHONWARNINGS env variable does not accept module regexes
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.11
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Yongjik Kim, blueyed, christian.heimes, coldfix, kernc, kevinoid, mpaolini, ncoghlan, nedbat, vstinner, xtreak
Priority: normal Keywords: patch, security_issue

Created on 2018-09-10 23:36 by coldfix, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9358 closed coldfix, 2018-09-17 12:43
PR 23172 kevinoid, 2022-03-12 09:00
Messages (16)
msg324959 - (view) Author: Thomas Gläßle (coldfix) * Date: 2018-09-10 23:36

This command does not report a warning, while it should:

  python -c 'import warnings; warnings.warn("This should show up")' -Wi -W'default:::.*'

If the regex `.*` is replaced by `__main__` it works as expected.

Same applies for regexes in PYTHONWARNINGS and for the `message` part of the argument. 

The reason can be found in Lib/ (def _setoption):

  module = re.escape(module)

This point-blank escape makes me think that it was intended that no regexes can be passed to message/module. On the other, the documentation reads as if it should be supported.

Specifically, the -W option is documented in [1]. While this page lists only basic examples, it refers to [2] and [3] for more details. [2] states that message/module are regexes. [3] seems to be written to specifically address the syntax of the PYTHONWARNINGS and the -W option and explicitly lists an example with a regex.


I would welcome if we could remove `re.escape` to make the implementation fit the documentation, or are there any downsides to this?

Best regards, Thomas
msg324960 - (view) Author: Thomas Gläßle (coldfix) * Date: 2018-09-10 23:39
Very sorry, the example command above should read:

  python -Wi -W'default:::.*' -c 'import warnings; warnings.warn("This should show up")'
msg324997 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-09-11 08:42
The original escape was done with 2a862c614d6d3ff50dc644150670651fdc9f3a99 which was in 2000. The doc example you are referring to at [1] was added with and there doesn't seem to be a test involving regular expression as I can see from the PR. Adding Nick to the issue since he had committed the doc and might help here.


msg325538 - (view) Author: Thomas Gläßle (coldfix) * Date: 2018-09-17 12:43
Thanks for your response. I have opened a PR at [1] that would remove the re.escape such that the implementation matches the documentation if you decide that this is fine.


Personally, I would go even further and always add the `(\..*)?$` suffix (instead of only `$`) to make it easier to match all modules in a package which is probably the most important use-case for the case of packages. What do you think?
msg328648 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2018-10-27 13:46
Another option is to make clear in the docs that the command line does not accept regular expressions, but only literal module names, and raise an error if a non-importable name (for example with asterisks in it) is specified.

And while we are here: the docs show "error:::mymodule[.*]" which doesn't even make sense as a regex!
msg328650 - (view) Author: Thomas Gläßle (coldfix) * Date: 2018-10-27 14:10
Hi, thanks for the consideration!

Is there any reason to introduce different behaviour than with filterwarnings here? This increases the number of things to remember - and except for the dot regex syntax doesn't interfere with module names, so there is no big drawback here either.

More importantly, my main use case for filterwarnings would be to select/ignore warnings based on module *or package* name. With the current interpretation as exact module name, you have to list all submodules in advance, which is quite inhibiting.

I have fixed the `[.*]` bogus example in the PR.

Best, Thomas
msg336646 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-02-26 11:15
I think the only reason I didn't mention this discrepancy in my doc updates is because I wasn't aware there *was* a discrepancy.

The weird syntax was then just an incorrect amalgamation of "optional argument" notation with an improperly escaped regex suffix.
msg348202 - (view) Author: daniel hahler (blueyed) * Date: 2019-07-19 23:03
Note that "module" might also be the filename (with ".py" removed).

I've just created issue37634 - might be worth including in your PR if it makes sense to only document this.
msg373358 - (view) Author: Marco Paolini (mpaolini) * Date: 2020-07-08 22:47
hello Thomas,

do you need any help fixing the conflicts in your PR?

even if Lib/ changed a little in the last 2 years, your PR is still good!
msg373536 - (view) Author: Thomas Gläßle (coldfix) * Date: 2020-07-11 19:56
Hi, I have rebased this on master and fixed the minor conflict. Let me know if there is anything else I can do.

Best, Thomas
msg373669 - (view) Author: Yongjik Kim (Yongjik Kim) Date: 2020-07-15 06:45
Hi, sorry if I'm interrupting, but while we're at this, could we also not escape regex for "message" part?  (Or at least amend the documentation to clarify that the message part is literal string match?)

Currently, the docs on -W just say "The meaning of each of these fields is as described in The Warnings Filter" and then "The Warnings Filter" section says that the "message" field is a regex, but currently it's only true if you run warnings.filterwarnings() directly, and not if you use the -W option.
msg408305 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-11 13:59
Adding regular expression support to -W and PYTHONWARNINGS env var turns the options into potential attack vectors. It can introduce REDOS vulnerability.
msg408496 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-12-13 23:28
Oh, I didn't know this issue. I closed my issue bpo-43862 as a duplicate.
msg408498 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-12-13 23:29
> Adding regular expression support to -W and PYTHONWARNINGS env var turns the options into potential attack vectors.

Why would an attacker control these options?

If an attacker controls how Python is run, they are more efficient way to take control of Python and execute arbitrary code, than just trigger a denial of service, no
msg408499 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-12-13 23:33
One option is to keep the current behavior by default, but support a new "/regex/" format. The /regex/ format is commonly used in Perl and sed.

Example to ignore warnings containing "deprecated" string in their message:

   python3 -W "ignore:/deprecated/"

whereas the following commands continue to match exactly the whole warning message "deprecated":

   python3 -W "ignore:deprecated"
msg408959 - (view) Author: Thomas Gläßle (coldfix) * Date: 2021-12-20 13:04
Ok, it seems at least the incorrect documentation has been fixed in the mean time.

I'm going to close this as there seems to be no capacity to deal with this.
Date User Action Args
2022-04-11 14:59:05adminsetgithub: 78805
2022-03-12 09:00:18kevinoidsetnosy: + kevinoid

pull_requests: + pull_request29932
2021-12-20 13:04:35coldfixsetstatus: open -> closed
stage: patch review -> resolved
2021-12-20 13:04:27coldfixsetmessages: + msg408959
2021-12-13 23:33:49vstinnersetmessages: + msg408499
2021-12-13 23:29:50vstinnersetmessages: + msg408498
2021-12-13 23:28:26vstinnerlinkissue43862 superseder
2021-12-13 23:28:11vstinnersetnosy: + vstinner
messages: + msg408496
2021-12-11 13:59:20christian.heimessetversions: + Python 3.11, - Python 3.6, Python 3.7, Python 3.8, Python 3.9, Python 3.10
nosy: + christian.heimes

messages: + msg408305

keywords: + security_issue
type: enhancement
2020-07-15 09:50:06nikratiosetnosy: - nikratio
2020-07-15 06:45:50Yongjik Kimsetnosy: + Yongjik Kim
messages: + msg373669
2020-07-11 19:56:10coldfixsetmessages: + msg373536
versions: + Python 3.9, Python 3.10
2020-07-08 22:47:37mpaolinisetnosy: + mpaolini
messages: + msg373358
2019-07-19 23:03:49blueyedsetnosy: + blueyed
messages: + msg348202
2019-02-26 11:15:29ncoghlansetmessages: + msg336646
2018-12-08 18:35:08kerncsetnosy: + kernc
2018-10-27 14:10:48coldfixsetmessages: + msg328650
2018-10-27 13:46:09nedbatsetnosy: + nedbat
messages: + msg328648
2018-10-07 13:02:32nikratiosetnosy: + nikratio

title: -W option does not accept module regexes -> -W option and PYTHONWARNINGS env variable does not accept module regexes
2018-10-07 12:26:54martin.panterlinkissue34920 superseder
2018-09-17 12:43:34coldfixsetmessages: + msg325538
2018-09-17 12:43:02coldfixsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8781
2018-09-11 08:42:50xtreaksetnosy: + ncoghlan
messages: + msg324997
2018-09-11 08:15:25xtreaksetnosy: + xtreak
2018-09-10 23:39:14coldfixsetmessages: + msg324960
2018-09-10 23:36:34coldfixcreate