This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: fails with UnboundLocalError on invalid keyword in marked section
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ezio.melotti Nosy List: Pikamander2, berker.peksag, corona10, ezio.melotti, jkamdjou, kodial, leonardr, lukasz.langa, mareksuscak, xtreak
Priority: normal Keywords: patch

Created on 2018-08-23 17:30 by kodial, last changed 2022-04-11 14:59 by admin.

File name Uploaded Description Edit Pikamander2, 2020-04-17 09:17
Pull Requests
URL Status Linked Edit
PR 17643 closed jkamdjou, 2020-01-04 18:09
PR 8562 mareksuscak, 2022-03-27 17:31
PR 32256 open mareksuscak, 2022-04-02 19:39
Messages (14)
msg323962 - (view) Author: Conrad (kodial) Date: 2018-08-23 17:30
$ pip freeze | grep beautifulsoup4

$ python
Python 3.7.0 (default, Jul 23 2018, 20:24:19)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from bs4 import BeautifulSoup
>>> text = "<![hi world]>"
>>> BeautifulSoup(text, 'html.parser')
/Users/conradroche/solvvy/ml-pipeline/venv/lib/python3.7/site-packages/bs4/builder/ UserWarning: unknown status keyword 'hi ' in marked section
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/conradroche/solvvy/ml-pipeline/venv/lib/python3.7/site-packages/bs4/", line 282, in __init__
  File "/Users/conradroche/solvvy/ml-pipeline/venv/lib/python3.7/site-packages/bs4/", line 343, in _feed
  File "/Users/conradroche/solvvy/ml-pipeline/venv/lib/python3.7/site-packages/bs4/builder/", line 247, in feed
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/html/", line 111, in feed
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/html/", line 179, in goahead
    k = self.parse_html_declaration(i)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/html/", line 264, in parse_html_declaration
    return self.parse_marked_section(i)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/", line 160, in parse_marked_section
    if not match:
UnboundLocalError: local variable 'match' referenced before assignment
msg323966 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-08-23 18:12
Thanks for the report.

HTMLParser.error() was supposed to raise an exception, but the BeautifulSoup project just prints a warning here:

    def error(self, msg):

As a result of this, the code doesn't stop executing in the following branch:

        self.error('unknown status keyword %r in marked section' % rawdata[i+3:j])
    if not match:
        return -1

Note that HTMLParser.error() was removed in Python 3.5 ( and there is an open issue about the status of _markupbase.ParserBase.error(): Issue 31844.

I also think that may have caused a minor regression when it was removed the error() method and its uses from the HTMLParser class. It still calls the parse_marked_section() method of _markupbase.ParserBase() which it then calls the error() method of _markupbase.ParserBase():

    elif rawdata[i:i+3] == '<![':
        return self.parse_marked_section(i)
msg358616 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-12-18 09:07
Attaching a test case for this issue since the self.error code path was not covered in the current test suite. A PR is open proposing match variable initialisation with None .

diff --git Lib/test/ Lib/test/
index a2bfb39d16..a9aff11706 100644
--- Lib/test/
+++ Lib/test/
@@ -766,6 +766,18 @@ class AttributesTestCase(TestCaseBase):
                           [("href", "\">;")]),
                          ("data", "spam"), ("endtag", "a")])
+    def test_invalid_keyword_error_exception(self):
+        class InvalidMarkupException(Exception):
+            pass
+        class MyHTMLParser(html.parser.HTMLParser):
+            def error(self, message):
+                raise InvalidMarkupException(message)
+        parser = MyHTMLParser()
+        with self.assertRaises(InvalidMarkupException):
+            parser.feed('<![invalid>')
 if __name__ == "__main__":
msg359301 - (view) Author: Josh Kamdjou (jkamdjou) * Date: 2020-01-04 18:28
(Author of PR

Since the behavior of self.error() is determined by the subclass implementation, an Exception is not guaranteed. How should this be handled? It seems the options are:

- continue execution, in which case 'match' needs to be defined (I proposed initialization to None, which results in returning -1 on the next line)
- return a value
- raise an Exception

Happy to update the PR with @xtreak's test cases.
msg359356 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2020-01-05 17:07
HTMLParser is supposed to follow the HTML5 standard, and never raise an error.

For the example in the first comment ("<![hi world]>"), the steps should be:


I agree that the error should be fixed by setting `match` to None, and a test case that triggers the UnboundLocalError (before the fix) should be added as well (what provided by Karthikeyan looks good).

However, it also seems wrong that HTMLParser ends up calling self.error() through  Lib/ ParserBase after HTMLParser.error() and all the calls to it have been removed. is internal, so it should be safe to remove ParserBase.error() and the code that calls it as suggested in #31844 (and possibly to merge _markupbase into html.parser too).  Even if this is done and the call to self.error() is removed from ParserBase.parse_marked_section(), `match` still needs to be set to None (either in the `else` branch or before the `if/elif` block).
msg359362 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2020-01-05 17:23
> is internal, so it should be safe to remove
> ParserBase.error() and the code that calls it as suggested in #31844

Should I reopen then?
msg359428 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2020-01-06 14:44
I think so.
It might be worth double-checking if BeautifulSoup (and possibly other 3rd party libs) use and/or ParserBase.error().  If they do, giving them a heads up and/or going through a regular deprecation process might be good, otherwise PR 8562 can be reopened and merged after removing the call to self.error() in ParserBase.parse_marked_section().
msg366648 - (view) Author: Pikamander2 (Pikamander2) Date: 2020-04-17 09:17
Here's a simplified real world example that I found if it helps anyone:

import bs4

html = '''<html><head><!fill in here--> <![end--><!-- start: SSI y-xtra-topspace.shtml --><!-- --></head><body></body></html>'''

soup = bs4.BeautifulSoup(html, 'html.parser')
msg366649 - (view) Author: Pikamander2 (Pikamander2) Date: 2020-04-17 09:21
For reference, my Python version is 3.8.2 and my bs4 version is 4.8.1
msg366650 - (view) Author: Pikamander2 (Pikamander2) Date: 2020-04-17 09:35
I updated to bs4 version 4.9.0 and the same issue occurs.
msg371418 - (view) Author: Leonard Richardson (leonardr) * Date: 2020-06-12 20:56
I'm the maintainer of Beautiful Soup. I learned about this issue when one of my users filed a bug for it against Beautiful Soup (

BeautifulSoupHTMLParser (my subclass of html.parser.HTMLParser) implements error() only to prevent the NotImplementedError raised by ParserBase.error(). The docstring for my error() implementation says: "this method is called only on very strange markup and our best strategy is to pretend it didn't happen and keep going."

It shouldn't affect Beautiful Soup if, in a future version of Python, HTMLParser.error is never called and ParserBase is removed.
msg416130 - (view) Author: Marek Suscak (mareksuscak) * Date: 2022-03-27 17:31
Based on the changelog, it seems that the fix for this issue has only landed in Python 3.10 but it's still affecting versions 3.7, 3.8 and 3.9. There are a few projects such as Apache Beam that have only recently added support for Python 3.9 with no support for 3.10. Beautiful Soup is a popular tool that's affected by this bug and so my question is if backporting this bugfix to at least Python 3.9 which is still an officially supported release is on the table?
msg416546 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2022-04-02 12:01
Marek, I can merge a fix for 3.9 for this. I don't think we should be removing _markupbase.ParserBase.error() in 3.9 as GH-8562 is doing. So we'd need a new patch, like GH-17643. However, that will need a test as well.

Would you be interested in creating the patch?
msg416582 - (view) Author: Marek Suscak (mareksuscak) * Date: 2022-04-02 19:50
Alright, the PR is up. This is my first contribution to Python and I had to sign the Contributor Agreement so it may take 1 business day before you're able to accept it. In the meantime, can you please review and let me know if anything needs to change? I wasn't sure where / how to add an entry in the NEWS.d folder. Can you please help me? Thanks!
Date User Action Args
2022-04-11 14:59:05adminsetgithub: 78661
2022-04-02 19:50:03mareksuscaksetmessages: + msg416582
2022-04-02 19:39:25mareksuscaksetpull_requests: + pull_request30325
2022-04-02 12:01:07lukasz.langasetmessages: + msg416546
2022-03-27 19:43:03iritkatrielsetnosy: + lukasz.langa
2022-03-27 17:31:11mareksuscaksetnosy: + mareksuscak

messages: + msg416130
pull_requests: + pull_request30221
2020-06-14 15:35:25corona10setnosy: + corona10
2020-06-12 20:56:04leonardrsetnosy: + leonardr
messages: + msg371418
2020-04-17 09:35:20Pikamander2setmessages: + msg366650
2020-04-17 09:21:19Pikamander2setmessages: + msg366649
2020-04-17 09:17:05Pikamander2setfiles: +
nosy: + Pikamander2
messages: + msg366648

2020-01-06 14:44:47ezio.melottisetmessages: + msg359428
2020-01-05 17:23:57berker.peksagsetmessages: + msg359362
2020-01-05 17:07:33ezio.melottisetmessages: + msg359356
2020-01-04 18:28:48jkamdjousetnosy: + jkamdjou
messages: + msg359301
2020-01-04 18:09:23jkamdjousetkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request17250
2019-12-18 09:07:20xtreaksetnosy: + xtreak
messages: + msg358616
2018-09-13 16:39:29ezio.melottisetassignee: ezio.melotti
2018-08-23 18:13:11berker.peksagsetmessages: - msg323967
2018-08-23 18:12:56berker.peksagsetstatus: open
2018-08-23 18:12:27berker.peksagsetnosy: ezio.melotti, berker.peksag, kodial
messages: + msg323967
2018-08-23 18:12:19berker.peksagsetstatus: open -> (no value)

nosy: + ezio.melotti, berker.peksag
messages: + msg323966

stage: test needed
2018-08-23 17:30:37kodialcreate