classification
Title: test_urllibnet failure
Type: behavior Stage: resolved
Components: Library (Lib), Tests Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: belopolsky, dmalcolm, loewis, orsenthil, pitrou, rosslagerwall
Priority: normal Keywords: patch

Created on 2010-06-10 19:07 by pitrou, last changed 2012-03-15 05:37 by orsenthil. This issue is now closed.

Files
File name Uploaded Description Edit
issue8963.diff belopolsky, 2010-06-10 20:02 review
Messages (21)
msg107470 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 19:07
This test failure just happened to me on a py3k checkout:

test test_urllibnet failed -- Traceback (most recent call last):
  File "/home/antoine/py3k/debug/Lib/test/test_urllibnet.py", line 191, in test_data_header
    time.strptime(datevalue, dateformat)
  File "/home/antoine/py3k/debug/Lib/_strptime.py", line 461, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/home/antoine/py3k/debug/Lib/_strptime.py", line 332, in _strptime
    (data_string, format))
ValueError: time data 'Thu, 10 Jun 2010 19:03:39 GMT' does not match format '%a, %d %b %Y %H:%M:%S GMT'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antoine/py3k/debug/Lib/test/test_urllibnet.py", line 193, in test_data_header
    self.fail('Date value not in %r format', dateformat)
TypeError: fail() takes at most 2 arguments (3 given)
msg107472 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-10 19:39
How do I enable `network' resource?

I am getting

$ ./python.exe -m test.regrtest test_urllibnet
test_urllibnet
test_urllibnet skipped -- Use of the `network' resource not enabled
1 test skipped:
    test_urllibnet
Those skips are all expected on darwin.


Same on Linux ...
msg107473 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 19:45
> How do I enable `network' resource?

Use the "-unetwork" flag to regrtest.
msg107474 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-10 19:48
$ ./python.exe -m test.regrtest -unetwork test_urllibnet
test_urllibnet
1 test OK.

Also the arguments to strptime from reported error message work fine:

>>> import time
>>> time.strptime('Thu, 10 Jun 2010 19:03:39 GMT', '%a, %d %b %Y %H:%M:%S GMT')
time.struct_time(tm_year=2010, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=3, tm_sec=39, tm_wday=3, tm_yday=161, tm_isdst=-1)
>>> from datetime import *
>>> datetime.strptime('Thu, 10 Jun 2010 19:03:39 GMT', '%a, %d %b %Y %H:%M:%S GMT')
datetime.datetime(2010, 6, 10, 19, 3, 39)
msg107475 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-10 19:48
What is your locale?
msg107476 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 19:51
> What is your locale?

$ locale
LANG=fr_FR.utf8
LC_CTYPE="fr_FR.utf8"
LC_NUMERIC="fr_FR.utf8"
LC_TIME="fr_FR.utf8"
LC_COLLATE="fr_FR.utf8"
LC_MONETARY="fr_FR.utf8"
LC_MESSAGES="fr_FR.utf8"
LC_PAPER="fr_FR.utf8"
LC_NAME="fr_FR.utf8"
LC_ADDRESS="fr_FR.utf8"
LC_TELEPHONE="fr_FR.utf8"
LC_MEASUREMENT="fr_FR.utf8"
LC_IDENTIFICATION="fr_FR.utf8"
LC_ALL=
msg107477 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 20:02
Strangely, it also works here from the prompt:

>>> import time
>>> time.strptime('Thu, 10 Jun 2010 19:03:39 GMT', '%a, %d %b %Y %H:%M:%S GMT')
time.struct_time(tm_year=2010, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=3, tm_sec=39, tm_wday=3, tm_yday=161, tm_isdst=-1)
>>> import locale
>>> locale.getlocale(locale.LC_TIME)
(None, None)

Apparently, something sets the locale before running the test.
msg107478 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-10 20:02
I still cannot reproduce the failure, but please, try the attached patch.
msg107480 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 20:13
The culprit seems to be test_bad_address. If I disable this test, or replace the bad URL by a good one, everything works fine. It seems that failing to resolve the domain name changes the current locale...
msg107481 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 20:20
The patch is not sufficient, since other failures can then occur if test_strptime gets run after test_urllibnet:

======================================================================
ERROR: test_twelve_noon_midnight (test.test_strptime.Strptime12AMPMTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/py3k/debug/Lib/test/test_strptime.py", line 389, in test_twelve_noon_midnight
    eq(time.strptime('12 PM', '%I %p')[3], 12)
  File "/home/antoine/py3k/debug/Lib/_strptime.py", line 461, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/home/antoine/py3k/debug/Lib/_strptime.py", line 335, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: PM

======================================================================
FAIL: test_pattern (test.test_strptime.TimeRETests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/py3k/debug/Lib/test/test_strptime.py", line 124, in test_pattern
    pattern_string)
AssertionError: did not find abbreviated weekday in pattern string '(?P<a>lun\.|mar\.|mer\.|jeu\.|ven\.|sam\.|dim\.)\s+(?P<A>mercredi|vendredi|dimanche|samedi|lundi|mardi|jeudi)\s+(?P<d>3[0-1]|[1-2]\d|0[1-9]|[1-9]| [1-9])'

======================================================================
FAIL: test_hour (test.test_strptime.StrptimeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/py3k/debug/Lib/test/test_strptime.py", line 266, in test_hour
    (strf_output, strp_output[3], self.time_tuple[3]))
AssertionError: testing of '%I %p' directive failed; '08 ' -> 8 != 20

----------------------------------------------------------------------
msg107482 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-10 20:22
BTW, looking at test_bad_address(), I wonder why it puts extra dots in the url?  The comment above it suggests that the intent is to use a name within RFC 2606 .invalid TLD.  Not likely to be a problem in your case, but using made up TLD is not a good idea.
msg107483 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-10 20:25
> The patch is not sufficient, since other
> failures can then occur if test_strptime gets
> run after test_urllibnet

This looks like a bug in support.run_with_locale decorator.  

It is described as
#=======================================================================                                                                                                                                     
# Decorator for running a function in a different locale, correctly resetting                                                                                                                                
# it afterwards.

but apparently does not.
msg107484 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 20:27
Ok, so what it boils down to is the following behaviour:

>>> import locale, socket
>>> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> locale.getlocale(locale.LC_TIME)
(None, None)
>>> sock.connect(("invalidhost", 80))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.gaierror: [Errno -2] Nom ou service inconnu
>>> locale.getlocale(locale.LC_TIME)
('fr_FR', 'UTF8')
msg107485 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 20:44
In setipaddr() in socketmodule.c, the following line appears to change the current locale when DNS lookup fails:

    error = getaddrinfo(name, NULL, &hints, &res);

This is checked by making a call to setlocale(LC_TIME, NULL) before and after the aforementioned line.

This is with the following libc:
$ rpm -qv glibc
glibc-2.11.1-8mnb2
msg107486 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-06-10 21:10
This looks like a glibc bug to me. I suspect an unauthorized redhat change; I hope Ulrich Drepper would have never accepted a glibc that causes getaddrinfo to implicitly call setlocale - see

http://sources.redhat.com/ml/libc-alpha/2004-03/msg00161.html

You might experiment with disabling IDN support in getaddrinfo.
msg107490 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 21:17
I've reported the bug upstream at Mandriva:
https://qa.mandriva.com/show_bug.cgi?id=59736

It would be nice to know whether other distributions with a Redhat lineage are affected. Can someone with such a distribution the code in msg107484?

In the end, I'm not sure it's Python's task to workaround such bug.
However, the bug in error reporting should be fixed.
msg107494 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-10 21:37
It looks like your libc calls setlocale(LC_ALL, "") on error.  This may or may not be right and not python's problem in any case.  What is worth to investigate, however is why @run_with_locale decorator fails to restore the locale after it was modified by libc.
msg107496 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-10 22:15
Antoine,

Would s/LC_TIME/LC_ALL/ in my patch fix your problem?  I could not find an affected system, but I simulated the problem by adding locale.setlocale(locale.LC_ALL, "") call in the test.

I think the patch is worth applying. It fixes the typo in the except clause and makes the test independent of the locale settings.
msg107497 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-06-10 22:24
> Would s/LC_TIME/LC_ALL/ in my patch fix your problem?

As I explained, this wouldn't fix the later failures in test_strptime.
msg155387 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2012-03-11 15:30
I tested the code from msg107484 on Fedora 16 with no change in locale.

Probably OK to close?
msg155859 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-03-15 05:37
Closing this based on - msg155387. The upstream bug is "unfortunately" closed as OLD. We can reopen if this stumble upon this again.
History
Date User Action Args
2012-03-15 05:37:09orsenthilsetstatus: open -> closed
resolution: not a bug
messages: + msg155859

stage: needs patch -> resolved
2012-03-11 15:30:21rosslagerwallsetnosy: + rosslagerwall
messages: + msg155387
2010-06-10 22:24:47pitrousetmessages: + msg107497
2010-06-10 22:15:21belopolskysetmessages: + msg107496
2010-06-10 21:37:38belopolskysetmessages: + msg107494
2010-06-10 21:17:27pitrousetassignee: orsenthil ->

messages: + msg107490
nosy: + dmalcolm
2010-06-10 21:10:34loewissetmessages: + msg107486
2010-06-10 20:44:10pitrousetmessages: + msg107485
2010-06-10 20:28:13pitrousetversions: + Python 2.6, Python 3.1, Python 2.7
2010-06-10 20:27:48pitrousetnosy: + loewis
messages: + msg107484
2010-06-10 20:25:14belopolskysetmessages: + msg107483
2010-06-10 20:22:27belopolskysetmessages: + msg107482
2010-06-10 20:20:19pitrousetmessages: + msg107481
2010-06-10 20:13:52pitrousetmessages: + msg107480
2010-06-10 20:02:23belopolskysetfiles: + issue8963.diff
keywords: + patch
messages: + msg107478
2010-06-10 20:02:08pitrousetmessages: + msg107477
2010-06-10 19:51:27pitrousetmessages: + msg107476
2010-06-10 19:48:53belopolskysetmessages: + msg107475
2010-06-10 19:48:02belopolskysetmessages: + msg107474
2010-06-10 19:45:44pitrousetmessages: + msg107473
2010-06-10 19:39:55belopolskysetmessages: + msg107472
2010-06-10 19:07:17pitroucreate