There seems to be weird behaviour in BaseCookie.load() when cookie that has '[' in one of the values is being loaded.

There is no exception being thrown as the key is still legal but the cookie is not getting loaded properly and everything that was after the '[' valued cookie is being silently ignored.

>>> dd = SimpleCookie()
>>> dd
<SimpleCookie: >
>>> s = 'a=b; c=[; d=r; f=h'
>>> dd.load(s)
>>> dd
<SimpleCookie: a='b'>
>>>

There could be some history behind this that I'm unaware of that I'm not familiar with.

From what I can tell, this issue is simply due to the "[" character not being in _LegalCharsPatt (http/cookies.py). _LegalCharsPatt actually seems quite a bit more restrictive than it really should be. It's set to r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\}\{\=]", where RFC 6265 specifies:

cookie-pair       = cookie-name "=" cookie-value
cookie-name       = token
cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                       ; US-ASCII characters excluding CTLs,
                       ; whitespace DQUOTE, comma, semicolon,
                       ; and backslash
token             = <token, defined in [RFC2616], Section 2.2>

_LegalCharsPatt is used for regex matching on the cookie value, not the key (there is a distinction made between the two).

The omission of those characters is correct for the cookie keys, but not the values (RFC 2965 is a little less verbose, but nothing ruling out those characters for values).

Err, sorry, I entirely misunderstood the problem. The invalid characters are correct ([ = 5B, which indeed is illegal, I wasn't paying close enough attention to the hex values in the ABNF). It's the fact that the valid key/value pairs after the invalid one are ignored. I'll dig into the RFC and see if there's an expected behavior here and whether or not it's currently handled as expected.

Now I've confused myself and my first impression was correct. For some reason, my brain was thinking "%x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E" was the exclusion list for some reason (which is obviously horribly wrong).

So my first observation was correct in that they should simply be added to the valid character list and I'll get a patch together for that.

Attached patch to fix the issue as reported.

Something interesting that came out of this though is that due to the regex expression, if there's an invalid character in one of the cookie-octets, the rest of the cookie is ignored. I would assume that it should either a) ignore the entire cookie string or b) ignore the invalid cookie pair and accept valid pairs following. I've been unable to find that defined in any of the RFCs though.

Thanks for taking a look into that.

And yes the behaviour when invalid value is encountered is bit weird as the rest of the cookie is being silently ignored which is probably less than ideal in most cases.

Just wonder if there is any easy way of making the matching more aware as browsers may allow various things as cookie values I guess.

I do think it should be a little more permissive when parsing cookies. I've created #22983 to address that as to not conflate this issue, which the attached patch does address.

Ping for review/commit.

This is also an issue with Python 2.7.9 but not 2.7.8. There were various cookie related fixes in 2.7.9 which could have revealed this issue. Maybe this one?

https://hg.python.org/cpython/rev/9e765e65e5cb

We experimented with a version of the patch for 2.7.9. 

One issue we immediately noticed is that even though disallowed by the spec the use of commas in cookie values is widespread so we needed to add \, to the _LEGAL_VALUES_PATT.

Thanks for the report Mark, updating this patch to be more backwards compatible was on my to-do list. I've attached a new patch that simply adds the new characters to the legal value set.

It does look like that's the commit that introduced this issue, but the change was made for good reason.

Will this regression be fixed in Python 2.7, 3.2, and 3.3? If not, Django may need to vendor Python's cookie class to workaround this bug to prevent users from losing sessions and/or being unable to login to Django powered sites as reported in https://code.djangoproject.com/ticket/24492.

As I understand it, the change should also be applied to security releases
as the regression manifested by a security related patch being applied.
That said, there may be some debate as there apparently isn't much (if
anything) in the way of precedence here.

Adding Python 2.7 to the affected versions (from #23341 which was closed as a duplicate of this bug). We are very interested to know whether this will be fixed in a Python 2.7 patch as well.

This needs a review from the people who created and applied the security patch.  Demian, did you add them to nosy already?

Since this is a regression I'm going to mark it as a release blocker so Benjamin can decide whether or not it is important enough to go in to 2.7.10 even though the RC is already out.

> This needs a review from the people who created and applied the security patch.

+ Guido (committed https://hg.python.org/cpython/rev/9e765e65e5cb)

New changeset 710cdba13323 by Benjamin Peterson in branch '3.2':
allow square brackets in cookie values (closes #22931)
https://hg.python.org/cpython/rev/710cdba13323

New changeset c7b3a50a2f01 by Benjamin Peterson in branch '3.3':
merge 3.2 (#22931)
https://hg.python.org/cpython/rev/c7b3a50a2f01

New changeset a43f5515e3a2 by Benjamin Peterson in branch '3.4':
merge 3.3 (#22931)
https://hg.python.org/cpython/rev/a43f5515e3a2

New changeset c58f3e76dc6c by Benjamin Peterson in branch 'default':
merge 3.4 (#22931)
https://hg.python.org/cpython/rev/c58f3e76dc6c

New changeset 2a7b0e145945 by Benjamin Peterson in branch '2.7':
allow square brackets in cookie values (#22931)
https://hg.python.org/cpython/rev/2a7b0e145945