The patch looks good.  I'd like the comment to say "We use urlsafe_b64decode here because some mailers apparently use the urlsafe b64 alphabet, and urlsafe_b64decode will correctly decode both the urlsafe and regular alphabets".

Also, the new header parser doesn't handle this case either, and furthermore it doesn't handle binascii errors at all (my comment in the code indicates I didn't think it could ever raise there).  The fact that it doesn't handle the error at all can be considered a different issue, but it would be nice to add the same decode fix (and a test in test_email/test_headerregistry.py) for the new header parser.  Here's one way to reproduce the issue:

>>> from email import message_from_string as mfs
>>> from email.policy import default
>>> m = mfs("From: =?iso-8859-1?B?QW5tZWxkdW5nIE5ldHphbnNjaGx1c3MgU_xkcmluZzNwLmpwZw==?=\n\n", policy=default)
>>> m['From']
Traceback (most recent call last):
  File "/home/rdmurray/python/p34/Lib/email/_encoded_words.py", line 109, in decode_b
    return base64.b64decode(padded_encoded, validate=True), defects
  File "/home/rdmurray/python/p34/Lib/base64.py", line 89, in b64decode
    raise binascii.Error('Non-base64 digit found')
binascii.Error: Non-base64 digit found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rdmurray/python/p34/Lib/email/message.py", line 391, in __getitem__
    return self.get(name)
  File "/home/rdmurray/python/p34/Lib/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/home/rdmurray/python/p34/Lib/email/policy.py", line 145, in header_fetch_parse
    return self.header_factory(name, ''.join(value.splitlines()))
  File "/home/rdmurray/python/p34/Lib/email/headerregistry.py", line 583, in __call__
    return self[name](name, value)
  File "/home/rdmurray/python/p34/Lib/email/headerregistry.py", line 194, in __new__
    cls.parse(value, kwds)
  File "/home/rdmurray/python/p34/Lib/email/headerregistry.py", line 334, in parse
    kwds['parse_tree'] = address_list = cls.value_parser(value)
  File "/home/rdmurray/python/p34/Lib/email/headerregistry.py", line 325, in value_parser
    address_list, value = parser.get_address_list(value)
  File "/home/rdmurray/python/p34/Lib/email/_header_value_parser.py", line 2313, in get_address_list
    token, value = get_address(value)
  File "/home/rdmurray/python/p34/Lib/email/_header_value_parser.py", line 2290, in get_address
    token, value = get_group(value)
  File "/home/rdmurray/python/p34/Lib/email/_header_value_parser.py", line 2246, in get_group
    token, value = get_display_name(value)
  File "/home/rdmurray/python/p34/Lib/email/_header_value_parser.py", line 2072, in get_display_name
    token, value = get_phrase(value)
  File "/home/rdmurray/python/p34/Lib/email/_header_value_parser.py", line 1747, in get_phrase
    token, value = get_word(value)
  File "/home/rdmurray/python/p34/Lib/email/_header_value_parser.py", line 1728, in get_word
    token, value = get_atom(value)
  File "/home/rdmurray/python/p34/Lib/email/_header_value_parser.py", line 1645, in get_atom
    token, value = get_encoded_word(value)
  File "/home/rdmurray/python/p34/Lib/email/_header_value_parser.py", line 1421, in get_encoded_word
    text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
  File "/home/rdmurray/python/p34/Lib/email/_encoded_words.py", line 166, in decode
    bstring, defects = _cte_decoders[cte](bstring)
  File "/home/rdmurray/python/p34/Lib/email/_encoded_words.py", line 124, in decode_b
    raise AssertionError("unexpected binascii.Error")
AssertionError: unexpected binascii.Error