This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: gettext bug while parsing plural-forms metadata
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: akuchling Nosy List: akuchling, ned.deily, nedbat, python-dev, r.david.murray, straz, terry.reedy
Priority: normal Keywords: patch

Created on 2013-05-03 18:19 by straz, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
pybug.tar.gz straz, 2013-05-10 20:07 minimal reproducible test case
issue17898.patch akuchling, 2015-04-13 14:32 Patch with fix and test
Repositories containing patches
https://bitbucket.org/akuchling/cpython/
Messages (11)
msg188318 - (view) Author: Steve Strassmann (straz) Date: 2013-05-03 18:19
The gettext.py parser used by django (lib/python2.7/gettext.py),
GNUTranslations._parse(), around line 313 does not use clean values for k,v on each
iteration ("for item in tmsg.splitlines():")

To reproduce the problem (see traceback, below), try parsing a .PO file containing two headers like this, with a comment header immediately following a plurals header. This example was created by calling msgcat to combine several .po files into a single .po file. Msgcat inserted the comment line.

"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"#-#-#-#-#  messages.po (EdX Studio)  #-#-#-#-#\n"

Parsing the first header binds the inner loop variables:
  k= plural-forms v= ['nplurals=2', ' plural=(n != 1)', '']

Parsing the second header leaves k,v untouched, which then causes an improper
attempt to parse (since it's a comment, no further parsing of k,v should occur)
 v = v.split(';')

Bug workaround: I use polib to read and immediately save the file. This reorders the metadata to avoid presenting the parser with something that will break it.

Recommended bug fix: on each iteration over tmsg.splitlines, reset the values of k,v = (None, None)

--------------------
Traceback:
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
89.                     response = middleware_method(request)
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/middleware/locale.py" in process_request
24.         translation.activate(language)
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/__init__.py" in activate
105.     return _trans.activate(language)
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in activate
201.     _active.value = translation(language)
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in translation
191.     current_translation = _fetch(language, fallback=default_translation)
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in _fetch
180.                 res = _merge(localepath)
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in _merge
156.             t = _translation(path)
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in _translation
138.                 t = gettext_module.translation('django', path, [loc], DjangoTranslation)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py" in translation
480.                 t = _translations.setdefault(key, class_(fp))
File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in __init__
76.         gettext_module.GNUTranslations.__init__(self, *args, **kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py" in __init__
180.             self._parse(fp)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py" in _parse
315.                         v = v.split(';')

Exception Type: AttributeError at /
Exception Value: 'list' object has no attribute 'split'
msg188323 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-05-03 19:27
Does this bear any relationship to issue 1475523?  (And yes, I know it is...sad...that that issue hasn't been fixed yet.)
msg188324 - (view) Author: Steve Strassmann (straz) Date: 2013-05-03 19:38
There seem to be several bugs involving this particular inner loop in gettext._parse(), but I don't think they're equivalent.

The present bug (issue17898) is that parsing a plural header breaks the following header when it happens to be a comment.

issue1475523 seems to involve multi-line handling

issue12425 seems to involve breaking when the plural-forms value is empty.

Perhaps a useful design pattern to follow for code which executes this inner loop would be to have some initialization and loop invariants which are asserted true on each iteration. For example, properly initializing k and v on each iteration.
msg188866 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-05-10 19:13
Since the other two listed inner-loop issues apply to 3.x, I would guess that this does also. Steve, can you test with 3.3? And provide a *minimal*  test case?
msg188873 - (view) Author: Steve Strassmann (straz) Date: 2013-05-10 20:07
Sorry, I haven't installed python 3.*, I just have default Mac OS python 2.7.

Here's a minimal test case. Tar expands to
 file structure:                                                                                                 
  ./test.py                                                                                                      
  ./en/LC_MESSAGES/messages.po   

$ ./test.py 

Traceback (most recent call last):
  File "./test.py", line 28, in <module>
    test()
  File "./test.py", line 23, in test
    gettext.install('messages', localedir=LOCALEDIR)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 494, in install
    t = translation(domain, localedir, fallback=True, codeset=codeset)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 479, in translat\
ion
    t = _translations.setdefault(key, class_(fp))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 180, in __init__
    self._parse(fp)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 314, in _parse
    v = v.split(';')
AttributeError: 'list' object has no attribute 'split'
msg240613 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2015-04-13 14:32
Proposed patch against 3.5.
msg240629 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2015-04-13 15:49
Adding a link to a bitbucket repo.
msg240632 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2015-04-13 15:55
I would apply this change to 3.4 and 3.5.  Should I also backport it to 2.7?  I think the same bug applies there, though I haven't verified this or tried my patch.
msg240676 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-04-13 17:30
LGTM. I think it should be backported to 2.7.
msg240891 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-04-14 14:19
New changeset c3d269c01671 by Andrew Kuchling in branch '2.7':
#17898: reset k and v so that the loop doesn't use an old value
https://hg.python.org/cpython/rev/c3d269c01671
msg240893 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-04-14 14:25
New changeset 54df02192bfc by Andrew Kuchling in branch '3.4':
#17898: reset k and v so that the loop doesn't use an old value
https://hg.python.org/cpython/rev/54df02192bfc
History
Date User Action Args
2022-04-11 14:57:45adminsetgithub: 62098
2015-04-14 14:37:15akuchlingsetstatus: open -> closed
assignee: akuchling
resolution: fixed
stage: patch review -> resolved
2015-04-14 14:25:35python-devsetmessages: + msg240893
2015-04-14 14:19:09python-devsetnosy: + python-dev
messages: + msg240891
2015-04-13 17:30:53ned.deilysetnosy: + ned.deily

messages: + msg240676
versions: + Python 3.4, Python 3.5
2015-04-13 15:55:03akuchlingsetmessages: + msg240632
2015-04-13 15:49:09akuchlingsethgrepos: + hgrepo304
messages: + msg240629
2015-04-13 14:32:25akuchlingsetfiles: + issue17898.patch

nosy: + akuchling
messages: + msg240613

keywords: + patch
stage: test needed -> patch review
2013-05-10 20:07:10strazsetfiles: + pybug.tar.gz

messages: + msg188873
2013-05-10 19:13:00terry.reedysetnosy: + terry.reedy

messages: + msg188866
stage: test needed
2013-05-03 19:38:03strazsetmessages: + msg188324
2013-05-03 19:27:09r.david.murraysetnosy: + r.david.murray
messages: + msg188323
2013-05-03 18:40:33nedbatsetnosy: + nedbat
2013-05-03 18:19:46strazcreate