This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Raise non-silent warning for invalid escape sequences
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: CuriousLearner, Eric Wieser, abarry, asmeurer, dansebcar, gregory.p.smith, lukasz.langa, mark.dickinson, miss-islington, ncoghlan, njs, r.david.murray, rhettinger, serhiy.storchaka, steve.dower, terry.reedy, xtreak
Priority: normal Keywords: patch

Created on 2018-02-22 18:34 by abarry, last changed 2022-04-11 14:58 by admin.

Pull Requests
URL Status Linked Edit
PR 5849 closed abarry, 2018-02-24 14:10
PR 9652 merged serhiy.storchaka, 2018-10-01 08:01
PR 15142 merged serhiy.storchaka, 2019-08-06 11:10
PR 15195 merged gregory.p.smith, 2019-08-09 22:50
Messages (34)
msg312575 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2018-02-22 18:34
This is a follow-up to Issue27364.

Back in Python 3.6, a silent warning was added for all invalid escape sequences in str and bytes. It was suggested that it would remain a silent warning (which still impacts tests, while not visually annoying the average user) for two releases (3.6 and 3.7), then would be upgraded to a non-silent warning for two subsequent releases (3.8 and 3.9) before becoming a full-on syntax error. With the 3.7 feature freeze on and going, I think it's time to evaluate the approach we take for 3.8 :)

I suggest upgrading the DeprecationWarning to a SyntaxWarning, which is visible by default, for 3.8 and 3.9. I have cross-linked #27364 to this issue as well.

msg312610 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-02-23 01:54
+1 for the DeprecationWarning->SyntaxWarning->SyntaxError approach from me (especially as 3.7 will make the existing deprecation warning visible in interactive shells and __main__ modules by default).
msg312760 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-24 21:32
PR 5849 changes not only the Python parser, but codecs. It shouldn't. The Python parser and codecs will go different ways. The warning in the Python parser will finally be upgraded to SyntaxError (it is already replaced with SyntaxError if the warning is raised as error). The warning in codecs will become UnicodeDecodeError or ValueError. This is why the code for emitting the warning is not shared between the parser and codecs at first place.
msg326766 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-01 08:06
Since the author of PR 5849 had not the time to work on it, I have created PR 9652 which properly replaces a DeprecationWarning with a SyntaxWarning. It also updates the documentation.
msg328044 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-19 14:42
New changeset 6543912c90ffa579dc4c01e811f9609cf92197d3 by Serhiy Storchaka in branch 'master':
bpo-32912: Replace a DeprecationWarning with a SyntaxWarning (GH-9652)
msg344312 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-06-02 20:41
I'm seeing these warnings pop up from time to time with various third party packages (such as pyparsing which is invoked by ensurepip).

Am wondering whether this should be deferred for another version or so until be have good confidence that the major third party packages have all had a chance to adapt.  Otherwise, this will mostly be noise for end-users.  Also, it gets in the way of the end-user strategy of "backslash anything that looks special" as preferred over "remember exactly which things are special".  This may be a case of "the cure is worse than the disease".
msg344764 - (view) Author: Aaron Meurer (asmeurer) Date: 2019-06-05 20:09
I agree with Raymond that third party libraries are not ready for this. 

My biggest issue is that the way Python warns about this makes it very difficult for library authors to fix this. Most won't even notice. The problem is the warnings are only shown once, when the file is compiled. So if you run the code in any way that causes Python to compile it, a further run of 'python -Wd' will show nothing. I don't know if it's reasonable, but it would be nice if Python recompiled when given -Wd, or somehow saved the warning so it could be shown if that flag is given later.

As an anecdote, for SymPy's CI, we went through five (if I am counting correctly) iterations of trying to test this. Each of the first four were subtly incorrect, until we finally managed to find the correct one (for reference, 'python -We:invalid -m compileall -f -q module/').  So most library authors who will attempt to add tests against this will get it wrong. Simply adding -Wd as you would expect is wrong. If the code is already compiled (which it probably is, e.g., if you ran, it won't show the warnings. At the very least the "correct" way to test this should be documented. 

Things would probably be improved if the warnings were always shown, as at least then devs will see the error once (although most will probably be confused when the warning doesn't repeat).

Another problem is the information in the warnings. It seems the line number of the string is now shown, which is an improvement ( It would be nice if it showed the actual line and column number in the file of the invalid escape. This is especially annoying when an escape appears in a docstring. It just shows """ as the offending line. 

We have a lot of LaTeX in our docstrings in SymPy, so we had quite a few of these to fix. SymPy doesn't have invalid escapes anymore because I was proactive about it, but from what I've seen, most library authors haven't been.

By the way, this looks like a bug (python 3.8b1):

$ cat

$ python -We:invalid
  File "", line 2
SyntaxError: invalid escape sequence \p

It might not be possible to see in what I pasted, but it filled my terminal with spaces.
msg345557 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-06-14 06:20
In the particular case of pyparsing there was a bug in the docstring.

     def sub(self, repl): 
         Return Regex with an attached parse action to transform the parsed 
         result as if called using `re.sub(expr, repl, string) <>`_. 
             make_html = Regex(r"(\w+):(.*?):").sub(r"<\1>\2</\1>") 
             print(make_html.transformString("h1:main title:")) 
             # prints "<h1>main title</h1>" 

\1 and \2 were interpreted as control characters U+0001 and U+0002, so the user could see 

        make_html = Regex(r"(\w+):(.*?):").sub(r"<^A>^B</^A>")


        make_html = Regex(r"(\w+):(.*?):").sub(r"<☺>☻</☺>")


        make_html = Regex(r"(\w+):(.*?):").sub(r"<></>")

depending on the output device.

So this examples proves the usefulness of the new warning.
msg345558 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-06-14 06:36
Aaron, you have reported several different issues in your message:

* compiler warnings are emitted only once, therefore it is easy to miss them and hard to reproduce.

* warnings for multiline expressions (and multiline string literals in particularly) could be better.

* strange output of spaces in some cases.

And maybe more.

I think all of them deserve opening separate issues.
msg345639 - (view) Author: Aaron Meurer (asmeurer) Date: 2019-06-14 21:45
I agree. Please someone else do that. I don't know what already has issues and I unfortunately don't have time right now to help out with any of this. 

I simply mentioned all these things as arguments why Python should not (yet) make these warnings errors, which is the point of this issue. 

Also, for the pyparsing example, you would have gotten lucky because it also contained \w, which is not a valid escape. If it didn't, you wouldn't be warned. So clearly this will help things, but it will also be good to have linting tools that, for example, warn about any escape sequences inside docstrings.
msg345675 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2019-06-15 11:37

> Also, it gets in the way of the end-user strategy of "backslash anything that looks special"

That's not a good strategy in the first place, though: adding an extra backslash for something that doesn't need to be escaped isn't benign - it's usually an error, since it puts that backslash into the resulting string.

    >>> "abc\`d" == "abc`d"

In other words, it seems to me that getting in the way of this broken end-user strategy is a *good* thing, since it warns of possible mistakes.
msg345703 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2019-06-15 16:52

> [...] makes it very difficult for library authors to fix this.

If you're willing to use 3rd party libraries, then my own experience is that `pycodestyle` or `flake8` make this easy: you just need to run "flake8 --select W605" at the root of your repository tree to get a list of all the code locations that need to be fixed.
msg346871 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-06-28 22:11
The 'filled my terminal with spaces' bug mentioned in msg344764 was independently reported in #37433, with a PR.
msg348205 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-07-20 01:14
> In other words, it seems to me that getting in the way of 
> this broken end-user strategy is a *good* thing, since it
> warns of possible mistakes.

Here's an example from the current version of Bottle (0.12.17):

/Users/raymond/Dropbox/Public/sj205/notes2/ SyntaxWarning: invalid escape sequence \[
  _re_tok += '|([\[\{\(])'
/Users/raymond/Dropbox/Public/sj205/notes2/ SyntaxWarning: invalid escape sequence \]
  _re_tok += '|([\]\}\)])

These warnings would spew out during a recent Python course where we used 3.8b2.  It mae it difficult to teach building templates in an iterative style.  We had to abandon 3.8 and go back to 3.7 to proceed with the templating lesson.

IMO, spewing out these warnings for things users can't do anything about is a usability travesty.
msg348269 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-07-21 23:15
Here's another example from the current build of the docs using Sphinx:

(venv) ~/npython/Doc $ make html
mkdir -p build
Building NEWS from Misc/NEWS.d with blurb
PATH=./venv/bin:$PATH sphinx-build -b html -d build/doctrees -D latex_elements.papersize=  -W . build/html
Running Sphinx v2.1.0
/Users/raymond/npython/venv/lib/python3.8/site-packages/docutils/writers/latex2e/ SyntaxWarning: invalid escape sequence \l
  self.out.append('}] \leavevmode ')
msg348270 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-07-21 23:15
I think it is poor form to bombard end-users with warnings about things they can't fix.
msg348281 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-22 07:07
The warning in docutils was fixed in 1.5 months ago. It was an outliner, all other occurrences of \leavevmode were either with the double backslash or in raw string literals.

The Bottle code was written 5 years age (, it was a part of the raw string. But a bug was introduced when it it was backported to the 0.12 version (
msg348347 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-07-23 19:42
I'm starting to see this error even for plain text strings in existing code:

>>> dunder_methods = '''

d[k]          d.__getitem__(k) -> value 
                   \--> d.__missing__(k) -> value
                    \-----------> raise KeyError(k)
d[k] = v      d.__setitem__(k, v) -> None
del d[k]      d.__delitem__(k) -> None


SyntaxError: invalid escape sequence \-

These not easily suppressed messages are really annoying.  We don't have to inflict this on our users.  IMO, this is the least user friendly change to Python 3.8 and as far as I can tell, it is entirely unnecessary (we've lived without it for 28+ years).
msg348348 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-07-23 19:46
AFAICT, I'm currently one of the only people in the world using the 3.8 beta for my work on a daily basis.  I've encountered these warnings multiple times in multiple contexts.  If we think other people won't experience this recurring annoyance, we're in denial.

The entire point of beta testing is to discover problems like this.  However, the participants in this thread seem inclined to ignore the evidence and persist with the idea that inflicting this on users makes the language better.  Respectfully, I disagree.
msg348349 - (view) Author: Aaron Meurer (asmeurer) Date: 2019-07-23 19:53
Raymond, are you in agreement that these warnings should at some point eventually become syntax errors?
msg348354 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-07-23 20:40
> Raymond, are you in agreement that these warnings should 
> at some point eventually become syntax errors?

I used to think so, but after experiencing the incessant warnings, I question the value.  In inactive sessions (either with the regular REPL or the ipython REPL), they are a recurring annoyance that interferes with data exploration live demos.  Perhaps, this should be left for a lint or code analysis tool.  Why should we intentionally break code that is currently working fine.

Another issue that I've encountered is that ASCII art becomes gets flagged.  Switching to a raw string then kills the unicode escape sequences.

This isn't really a "Raymond doesn't like this" concern.  I think anyone who starts using 3.8 on a daily basis for non-toy examples will constantly run into this.  Possibly, it catches a real error, but most often it will just be a recurring distractor, especially when teaching Python.  All instructors and runners of live demos will need to memorize exactly which characters require an escape and which don't -- it creates a new burden that didn't exist before.  And if the source of the problem is in an external library, the result is unactionable by the user, merely making their experience unpleasant.
msg348357 - (view) Author: Aaron Meurer (asmeurer) Date: 2019-07-23 23:40
Well paradoxically, the bugs that this prevents are the ones it doesn't warn about. If someone writes '\tan(x)' thinking it is a string representing a LaTeX formula for the tangent of x, they won't realize that they actually created a string with a tab plus "an(x)". So actually I would argue that the end goal *is* to make people aware of which escape characters exist, or at the very least, always make strings raw if there's even the remotest chance they will contain a backslash character. 

Is it the best way to go about this? I don't know. The whole thing sort of makes me think raw strings should have been the default, but it's obviously too late to change that. 

I personally don't feel strongly about the warnings being enabled by default or not. My big gripe is that if you actually want the warnings they are difficult to get in a reproducible way. I'm actually surprised they are so annoying for you. Once a py file is compiled into a pyc file the warnings completely disappear, even if you want them!

The fact that you can't use a real escape sequence in a raw string is annoying but not the end of the world given that it's trivial to concatenate strings.
msg348375 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-24 07:56
> Another issue that I've encountered is that ASCII art becomes gets flagged.  Switching to a raw string then kills the unicode escape sequences.

If you already use escape sequences in your ASCII art, what is the problem of using them for backslashes?

> I think anyone who starts using 3.8 on a daily basis for non-toy examples will constantly run into this.

What is a better solution? The deprecation warning was emitted starting from 3.6. Deprecation warnings were silent by default in 3.6, but they become more visible in 3.7. This helped some projects (which give attention to warnings in they tests) to fix real bugs. But it was not enough, so other bugs are fixed only now, when the warnings become even more visible in 3.8. We see a value of warnings, they help to fix bugs. If we rollback this change, it will cause yet undiscovered bugs be hidden for more time.

In Python 3.0 we make many abrupt breaking changes at once. People complained. Here is an opposite example of very gradual change: two releases with a DeprecationWarning, few more (at least two) releases with a SyntaxWarning, and finally perhaps a SyntaxError.
msg349027 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-08-05 05:29
I think we haven't *actually* done a proper DeprecationWarning period for this. We tried, but because of the issue with byte-compiling, the warnings were unconditionally suppressed for most users -- even the users who are diligent enough to enable warnings and look at warnings in their test suites.

I can see a good argument for making the change, but if we're going to do it then it's obviously the kind of change that requires a proper deprecation period, and that hasn't happened.

Maybe .pyc files need to be extended to store a list of syntax-related DeprecationWarnings and SyntaxWarnings, that are re-issued every time the .pyc is loaded? Then we'd at least have the technical capability to deprecate this properly.
msg349103 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-08-06 08:08
Can the pyc compilation step done by our normal package installers be made to treat this warning as an error so that it is forced into the package owners faces instead of overlooked because it was just something on stderr?

This syntax warning is absolutely the right thing to surface for _owners/maintainers_ of a given piece of code.  Their strings are wrong and their code quality will improve as a result.

It isn't the right thing to surface to _users_ of someone else's problematic code.  (though it does serve as a red flag proving that the owners of that code or person who pinned some dep to an old version haven't proactively tested with modern python)
msg349106 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-08-06 11:11
PR 15142 reverts that change for 3.8.
msg349318 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-08-09 22:34
New changeset 4c5b6bac2408f879231c7cd38d67657dd4804e7c by Gregory P. Smith (Serhiy Storchaka) in branch '3.8':
[3.8] bpo-32912: Revert SyntaxWarning on invalid escape sequences (GH-15142)
msg349320 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-08-09 23:11
Thank you!
msg349321 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-08-09 23:22
I'm leaving this open, as we may still want to do it in 3.9+, just in a less disruptive manner.  (That, and how, hasn't been decided yet)

Follow the thread(s) on python-dev for the latest on that.
msg349322 - (view) Author: Aaron Meurer (asmeurer) Date: 2019-08-09 23:52
Are there issues tracking the things I mentioned, which should IMO happen before this becomes a hard error (making the warnings reproduce even if the file has already been compiled, and making warning message point to the correct line in multiline strings)? And is it too late to potentially get some of those things in 3.8?
msg349323 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-08-10 00:15
I haven't looked, so not that i'm aware of.  I suggest filing one for each of those.

The warning not pointing to the right line in a multiline literal sounds like a bug to me so that one, if fixed, seems reasonable for 3.8.  The release manager gets to decide.

Making the syntax warnings happen upon import of an already compiled pyc is more feature-ish, likely to be 3.9+.
msg349328 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-08-10 04:26
One possibility is to restrict the warning to a backslash followed by an alphabetic character or backslash, and that we define backslash followed by any other printable character as specifically allowed.  This would catch the likely sources of errors without breaking the likes of \, or \-.  Also, multiline strings have less of a need for a warning than a single line string.
msg349335 - (view) Author: miss-islington (miss-islington) Date: 2019-08-10 07:19
New changeset b4be87a04a2a8ccfd2480e19dc527589fce53555 by Miss Islington (bot) (Gregory P. Smith) in branch 'master':
bpo-32912: Revert SyntaxWarning on invalid escape sequences. (GH-15195)
msg410496 - (view) Author: Daniel Carpenter (dansebcar) Date: 2022-01-13 16:38
I'm not sure if this is an issue or by design, but this DeprecationWarning behaves differently to other DeprecationWarnings.

A normal DeprecationWarning triggered by code in __main__ is printed by default:

$ python -c 'import warnings; warnings.warn("test", DeprecationWarning)'
<string>:1: DeprecationWarning: test

But this one is silent:

$ python -c '"\,"'
[no output]

To see this DeprecationWarning at all, I need to type:

$ python -Wdefault -c '"\,"'
<string>:1: DeprecationWarning: invalid escape sequence '\,'

But that enables this DeprecationWarning for all modules, not just __main__ .

I've tested this with Python 3.9 on debian bullseye and the 3.10 docker image.
Date User Action Args
2022-04-11 14:58:58adminsetgithub: 77093
2022-01-13 16:38:26dansebcarsetnosy: + dansebcar
messages: + msg410496
2020-04-21 23:13:00xtreaksetnosy: + xtreak
2019-08-12 16:57:20steve.dowersetnosy: + steve.dower
2019-08-10 07:19:15miss-islingtonsetnosy: + miss-islington
messages: + msg349335
2019-08-10 04:26:11rhettingersetmessages: + msg349328
2019-08-10 00:15:00gregory.p.smithsetmessages: + msg349323
2019-08-09 23:52:14asmeurersetmessages: + msg349322
2019-08-09 23:22:49gregory.p.smithsetstatus: closed -> open
versions: + Python 3.9, - Python 3.8
messages: + msg349321

resolution: fixed ->
stage: resolved ->
2019-08-09 23:11:49rhettingersetstatus: open -> closed

messages: + msg349320
stage: patch review -> resolved
2019-08-09 22:50:16gregory.p.smithsetpull_requests: + pull_request14927
2019-08-09 22:34:28gregory.p.smithsetmessages: + msg349318
2019-08-09 18:46:42CuriousLearnersetnosy: + CuriousLearner
2019-08-06 11:11:14serhiy.storchakasetmessages: + msg349106
2019-08-06 11:10:17serhiy.storchakasetstage: resolved -> patch review
pull_requests: + pull_request14879
2019-08-06 08:08:02gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg349103
2019-08-05 16:49:19ned.deilysetnosy: + lukasz.langa
2019-08-05 05:29:05njssetnosy: + njs
messages: + msg349027
2019-07-24 07:56:05serhiy.storchakasetmessages: + msg348375
2019-07-23 23:40:53asmeurersetmessages: + msg348357
2019-07-23 20:40:18rhettingersetmessages: + msg348354
2019-07-23 19:53:06asmeurersetmessages: + msg348349
2019-07-23 19:46:52rhettingersetmessages: + msg348348
2019-07-23 19:42:18rhettingersetmessages: + msg348347
2019-07-22 07:07:52serhiy.storchakasetnosy: + Eric Wieser
messages: + msg348281
2019-07-21 23:15:59rhettingersetmessages: + msg348270
2019-07-21 23:15:26rhettingersetmessages: + msg348269
2019-07-20 01:14:29rhettingersetmessages: + msg348205
2019-06-28 22:11:50terry.reedysetnosy: + terry.reedy
messages: + msg346871
2019-06-15 16:52:55mark.dickinsonsetmessages: + msg345703
2019-06-15 11:37:34mark.dickinsonsetnosy: + mark.dickinson
messages: + msg345675
2019-06-14 21:45:33asmeurersetmessages: + msg345639
2019-06-14 06:36:15serhiy.storchakasetmessages: + msg345558
2019-06-14 06:20:31serhiy.storchakasetmessages: + msg345557
2019-06-05 20:09:45asmeurersetnosy: + asmeurer
messages: + msg344764
2019-06-02 20:41:26rhettingersetstatus: closed -> open
nosy: + rhettinger
messages: + msg344312

2018-10-19 14:42:31serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-10-19 14:42:13serhiy.storchakasetmessages: + msg328044
2018-10-01 08:06:22serhiy.storchakasetmessages: + msg326766
2018-10-01 08:01:09serhiy.storchakasetpull_requests: + pull_request9043
2018-02-24 21:32:40serhiy.storchakasetmessages: + msg312760
2018-02-24 14:10:24abarrysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request5625
2018-02-23 01:54:10ncoghlansetmessages: + msg312610
2018-02-22 19:20:53serhiy.storchakasetnosy: + ncoghlan, r.david.murray, serhiy.storchaka
2018-02-22 18:34:57abarrycreate