"""
PEP 3101 defines format strings as intermingled character data and markup. Markup defines replacement fields and is delimited by braces. Only after markup is extracted does the PEP talk about interpreting the contents of the markup.

So, given "{0[a}b]}" the parser first parses out the character data and the markup. The first piece of markup is "{0[a}". That gives a syntax error because it's missing a right bracket.
"""

The intermingling of character data and markup is irrelevant; character data is defined as "data which is transferred unchanged from the format string to the output string", and nothing in "{0[a]}" is transferred unchanged.

Two parts of the PEP suggest that the markup in the above should be "{0[a}" rather than "{0[a}]}":

    Brace characters ('curly braces') are used to indicate a
    replacement field within the string:

    [...]

    Braces can be escaped by doubling:

and

    Note that the doubled '}' at the end, which would normally be
    escaped, is not escaped in this case.  The reason is because
    the '{{' and '}}' syntax for escapes is only applied when used
    *outside* of a format field.  Within a format field, the brace
    characters always have their normal meaning.

The first statement obviously doesn't mean that the exclusive use of braces in a format string is to indicate replacement fields, since it's immediately acknowledged that sometimes braces can occur without indicating a replacement field, when they're escaped. The second occurs specifically in the context of talking about escaping braces, so the following interpretation remains available: within a format field, a brace is a brace is a brace---that is, a pair of braces is a pair of braces, not an escape for a single brace.

In fact, though the following argument may appear Jesuitical, it does, I think, hold water: The second quotation above mentions braces within a *format field*. What is a format field? Well, we know that "The element with the braces is called a 'field'", but "format field" is more specific; the whole thing between braces isn't (necessarily!) the format field. And we know that

    Fields consist
    of a 'field name', which can either be simple or compound, and an
    optional 'format specifier'.

So, perhaps a format field is the part of the broader field where the format specifier lives. And lo, it's in the part of the PEP talking about "Format Specifiers" that we get the second quotation above.

    Each field can also specify an optional set of 'format
    specifiers' which can be used to adjust the format of that field.
    Format specifiers follow the field name, with a colon (':')
    character separating the two:

So even if you think that the claim that "within a format field, the brace characters always have their normal meaning" means not "the brace characters aren't escaped" but "the brace characters indicate a replacement field", that statement could just mean that they only have this significance in *part* of the *replacement* field---the part having to do with the formatting of the replacement text---and not the whole replacement field. So that, for instance, the following does what you'd expect:


>>> "{0[{4}]}".format({"{4}":3})
'3'

And it *does* do what you'd expect, in the *current* implementation---that is, the braces here don't have the meaning of introducing a replacement field [they're kinda-sorta parsed as if they were introduced a replacement field but that is obviously not their semantics], but are instead just treated as braces. They also aren't escaped: 

>>> "{0[{{4}}]}".format({"{{4}}":3})
'3'