[Reported by Alberto Casado Martín [1]]

Message.get_content_type() hangs when very large values are split by the
regex:
ctype = paramre.split(value)[0].lower().strip() #line 439

paramre comes from line 26:
paramre = re.compile(r'\s*;\s*')

Unless the full fledged parser cited in the comment before line 26 is in
the works, I suggest splitting the string by ";" to get exactly the same
behavior in a more reliable way.


[1] http://mail.python.org/pipermail/python-dev/2008-April/078840.html

Augmented version of Daniel's patch.

This makes an internal function that does the same work.  It uses
txt.find() instead of split() or partition() because for pathologically
long strings find() is noticeably faster.  It also does the strip()
before the lower() which helps with evilly long strings.

I didn't remove the module global "paramre" because an external module
might be using it.  I did update its comment.

Do bugfixes get applied to 2.6 or 3.0?  I'm a bit out of practice.

This should really be fixed. Hanging on a rather normal email message
(not a theoretical example) is not right.

Fixed in r65700. Thanks for the report!

Antoine, I looked at your patch and I'm not sure why you applied it
instead of applying mine (or saying +1 on me applying my patch).

Yours uses str.partition which I pointed out is sub-optimal (same big-Oh
but with a larger constant factor) and also adds a function that returns
two things, one of which is thrown away after having a str.strip
performed on it.

If my patch was deficient please let me know.

Hi Jack,

> Antoine, I looked at your patch and I'm not sure why you applied it
> instead of applying mine (or saying +1 on me applying my patch).
> 
> Yours uses str.partition which I pointed out is sub-optimal (same big-Oh
> but with a larger constant factor) and also adds a function that returns
> two things, one of which is thrown away after having a str.strip
> performed on it.

I added that function so that the header splitting facility is
explicitly exposed as an internal API, as was the case with the regular
expression. I tried to mimick the behaviour of the regex as closely as
possible, which meant returning two things as well :-)

I think the point of the issue is to remove the pathological
(exponential) behaviour when parsing some headers, not to try to squeeze
out the last microseconds out of content-type parsing (which shouldn't
be, IMO, the limiting factor in email handling performance as soon as
it's not super-linear).

That said, I've timed the function against the regular expression and
the former is always faster, even for tiny strings (e.g. "a;b").

Your patch was keeping the regular expression as a module-level constant
while replacing all uses of it with a function, which I found a bit
strange (I don't think people are using paramre from the outside since
it's not documented, it's an internal not public API IMO). I also found
it strange to devote a docstring to the discussion of a performance
detail. But I don't have any strong feeling against it either, so you
can still apply it if you think it's important performance-wise.

Regards

Antoine.