Hi,

This command does not report a warning, while it should:

  python -c 'import warnings; warnings.warn("This should show up")' -Wi -W'default:::.*'

If the regex `.*` is replaced by `__main__` it works as expected.

Same applies for regexes in PYTHONWARNINGS and for the `message` part of the argument. 

The reason can be found in Lib/warnings.py:144 (def _setoption):

  module = re.escape(module)

This point-blank escape makes me think that it was intended that no regexes can be passed to message/module. On the other, the documentation reads as if it should be supported.

Specifically, the -W option is documented in [1]. While this page lists only basic examples, it refers to [2] and [3] for more details. [2] states that message/module are regexes. [3] seems to be written to specifically address the syntax of the PYTHONWARNINGS and the -W option and explicitly lists an example with a regex.

[1]: https://docs.python.org/3/using/cmdline.html#cmdoption-w
[2]: https://docs.python.org/3/library/warnings.html#warning-filter
[3]: https://docs.python.org/3/library/warnings.html#describing-warning-filters

I would welcome if we could remove `re.escape` to make the implementation fit the documentation, or are there any downsides to this?


Best regards, Thomas

Very sorry, the example command above should read:

  python -Wi -W'default:::.*' -c 'import warnings; warnings.warn("This should show up")'

The original escape was done with 2a862c614d6d3ff50dc644150670651fdc9f3a99 which was in 2000. The doc example you are referring to at [1] was added with https://bugs.python.org/issue31975 and there doesn't seem to be a test involving regular expression as I can see from the PR. Adding Nick to the issue since he had committed the doc and might help here.


[1] https://docs.python.org/3/library/warnings.html#describing-warning-filters

Thanks

Thanks for your response. I have opened a PR at [1] that would remove the re.escape such that the implementation matches the documentation if you decide that this is fine.

[1]: https://github.com/python/cpython/pull/9358

Personally, I would go even further and always add the `(\..*)?$` suffix (instead of only `$`) to make it easier to match all modules in a package which is probably the most important use-case for the case of packages. What do you think?

Another option is to make clear in the docs that the command line does not accept regular expressions, but only literal module names, and raise an error if a non-importable name (for example with asterisks in it) is specified.

And while we are here: the docs show "error:::mymodule[.*]" which doesn't even make sense as a regex!

Hi, thanks for the consideration!

Is there any reason to introduce different behaviour than with filterwarnings here? This increases the number of things to remember - and except for the dot regex syntax doesn't interfere with module names, so there is no big drawback here either.

More importantly, my main use case for filterwarnings would be to select/ignore warnings based on module *or package* name. With the current interpretation as exact module name, you have to list all submodules in advance, which is quite inhibiting.

I have fixed the `[.*]` bogus example in the PR.

Best, Thomas

I think the only reason I didn't mention this discrepancy in my doc updates is because I wasn't aware there *was* a discrepancy.

The weird syntax was then just an incorrect amalgamation of "optional argument" notation with an improperly escaped regex suffix.

Note that "module" might also be the filename (with ".py" removed).

I've just created issue37634 - might be worth including in your PR if it makes sense to only document this.

hello Thomas,

do you need any help fixing the conflicts in your PR?


even if Lib/warnings.py changed a little in the last 2 years, your PR is still good!

Hi, I have rebased this on master and fixed the minor conflict. Let me know if there is anything else I can do.

Best, Thomas

Hi, sorry if I'm interrupting, but while we're at this, could we also not escape regex for "message" part?  (Or at least amend the documentation to clarify that the message part is literal string match?)

Currently, the docs on -W just say "The meaning of each of these fields is as described in The Warnings Filter" and then "The Warnings Filter" section says that the "message" field is a regex, but currently it's only true if you run warnings.filterwarnings() directly, and not if you use the -W option.

Adding regular expression support to -W and PYTHONWARNINGS env var turns the options into potential attack vectors. It can introduce REDOS vulnerability.

Oh, I didn't know this issue. I closed my issue bpo-43862 as a duplicate.

> Adding regular expression support to -W and PYTHONWARNINGS env var turns the options into potential attack vectors.

Why would an attacker control these options?

If an attacker controls how Python is run, they are more efficient way to take control of Python and execute arbitrary code, than just trigger a denial of service, no

One option is to keep the current behavior by default, but support a new "/regex/" format. The /regex/ format is commonly used in Perl and sed.

Example to ignore warnings containing "deprecated" string in their message:


   python3 -W "ignore:/deprecated/"
   PYTHONWARNINGS="ignore:/deprecated/"

whereas the following commands continue to match exactly the whole warning message "deprecated":

   python3 -W "ignore:deprecated"
   PYTHONWARNINGS="ignore:deprecated"

Ok, it seems at least the incorrect documentation has been fixed in the mean time.

I'm going to close this as there seems to be no capacity to deal with this.