classification
Title: urllib3 fails with type error exception, when cannot reach PyPI - urllib3.util.retry
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6, Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: Quentin.Pradet, acue, christian.heimes, deivid, martin.panter, xtreak
Priority: normal Keywords:

Created on 2018-11-28 06:53 by acue, last changed 2018-11-29 07:10 by acue. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1486 acue, 2018-11-29 07:10
Messages (6)
msg330574 - (view) Author: Arno-Can Uestuensoez (acue) Date: 2018-11-28 06:53
Hi,
I am using the installation script *adafruit-pitft.sh* which tries to apply *urllib3* which itself tries to install nested dependencies,
here *evdev* from PyPI.

When the installation is offline - e.g. by internal mirror of debian/raspbian - PyPI is not accessible. The code fails than by an type error
exception:

> adafruit-pitft.sh # or pitft.sh 
> https://github.com/adafruit/Raspberry-Pi-Installer-Scripts/blob/master/adafruit-pitft.sh
>
> ...
> 
>   _stacktrace=sys.exc_info()[2])
>  File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3 none-any.whl/urllib3/util/retry.py", line 228, in increment
>
>    total -= 1
>
> TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

For the current distribution(based on debian-9.6.0/stretch):

>  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 315, in increment
>
>    total -= 1
>
> TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

See also: https://stackoverflow.com/questions/37495375/python-pip-install-throws-typeerror-unsupported-operand-types-for-retry


The following - dirty *:) - patch enables a sounding error trace:

    # File: retry.py - in *def increment(self, ..* about line 315
    # original: total = self.total
    
    # patch: quick-and-dirty-fix
    # START:
    if isinstance(self.total, Retry):
        self.total = self.total.total
    
    if type(self.total) is not int:
        self.total = 2 # default is 10
    # END:
    
    # continue with original:
    total = self.total

    if total is not None:
        total -= 1
    
    connect = self.connect
    read = self.read
    redirect = self.redirect
    cause = 'unknown'
    status = None
    redirect_location = None
    
    if error and self._is_connection_error(error):
        # Connect retry?
        if connect is False:
            raise six.reraise(type(error), error, _stacktrace)
        elif connect is not None:
            connect -= 1


The sounding output with the temporary patch is(twice...):

> Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at/
>
>  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 
'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at/
>
>  Could not find a version that satisfies the requirement evdev (from versions: )
>
> No matching distribution found for evdev
>
> WARNING : Pip failed to install software!

In this case I did not thoroughly analysed the actual error source, thus did a quick and dirty patch only.

My proposal is to add the inplace operator to the class *Reply* with the default assignment in case of an type error and display an error message.
This is still not really the correct solution, but resolves a lot of confusion by required interpretation of the exception.

Arno-Can Uestuensoez
msg330576 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-11-28 07:39
I think this is an issue with pip specifically with Ubuntu/Debian packaging causing the error. Please see https://github.com/pypa/pip/issues/3943, https://github.com/pypa/pip/issues/4779 and https://answers.launchpad.net/ubuntu/+question/659285 . I think this is a third party issue.
msg330580 - (view) Author: Arno-Can Uestuensoez (acue) Date: 2018-11-28 09:02
@xtreak
I didn't checked the source of the assignment of the instance of *Retry* to *self.total*. For me it was important to get the display of the actual source of the failure, which is the failed access to PyPI. Thus my proposal, just to intercept the error, display a bug-report, and display the quite good error message for the failed PyPI access already contained in the code. 

Thus the intention to replace:

>  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 315, in increment
>
>    total -= 1
>
> TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

by:

> Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at/
>
>  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 
'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at/
>
>  Could not find a version that satisfies the requirement evdev (from versions: )
>
> No matching distribution found for evdev
>
> WARNING : Pip failed to install software!

Optionally with an additional error message for the faulty assignment to  the member *total*. But this is actually just a repetition counter, so even the integer replacement(>=1) should be OK. If an application uses it - wrongly - as a reference pointer, this will fail than immediately and the app recognises it's error straight away.
msg330581 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-11-28 09:09
This is the wrong bug tracker for urllib3. The urllib3 package is not part of Python's standard library. 

As xtreak pointed out, it's not even a bug in urllib3 or pip. Debian ships a modified and patched version of pip that uses an incompatible version of urllib3. Please follow up on the Debian bug tracker,
msg330588 - (view) Author: Arno-Can Uestuensoez (acue) Date: 2018-11-28 09:54
@christian.heimes , @xtreak

Thanks, just for documentation, hope this is the right place - putting one to debian later:

See: issue - 1486: https://github.com/urllib3/urllib3/issues/1486

for file: https://github.com/urllib3/urllib3/blob/master/src/urllib3/util/retry.py

Rem.: The issue is about the originator of the exception - which is not necessarily the original cause.
msg330664 - (view) Author: Arno-Can Uestuensoez (acue) Date: 2018-11-29 07:10
In order to avoid further confusion (github:  #1486):
 
> This is the wrong bug tracker for urllib3. The urllib3 package is not part of Python's standard library. 
=> OK.

# As xtreak pointed out, it's not even a bug in urllib3 or pip. Debian ships a modified and patched version of pip that uses an incompatible version of urllib3. Please follow up on the Debian bug tracker,
=> not the issue as I posted it. The file is: urllib3.util.retry, So NOK, see issue-text:
> My proposal is to add the inplace operator to the class *Reply* with the default assignment in case of an type error and display an error message.
> This is still not really the correct solution, but resolves a lot of confusion by required interpretation of the exception.

Anyhow, going to issue a bug report to debian.
History
Date User Action Args
2018-11-29 07:10:58acuesetmessages: + msg330664
pull_requests: + pull_request10033
2018-11-28 09:54:45acuesetmessages: + msg330588
2018-11-28 09:09:18christian.heimessetstatus: open -> closed

type: crash -> behavior

nosy: + christian.heimes
messages: + msg330581
resolution: third party
stage: resolved
2018-11-28 09:02:34acuesetmessages: + msg330580
2018-11-28 07:39:23xtreaksetnosy: + xtreak
messages: + msg330576
2018-11-28 06:53:30acuecreate