This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Simple curl/wget-like download functionality in urllib (like http offers server)
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.11
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, eric.smith, python-dev, rhettinger, vstinner
Priority: normal Keywords: patch

Created on 2021-10-14 06:15 by tom.pohl, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 29217 closed python-dev, 2021-10-25 19:23
Messages (7)
msg403888 - (view) Author: Tom Pohl (tom.pohl) * Date: 2021-10-14 06:15
In the context of building Docker images, it is often required to download stuff. If curl/wget are available, great, but often slim images don't offer that.

The urllib could provide a very simple download functionality (like http offers a simple server):

    from urllib.request import urlopen
    data = urlopen('https://.../install-poetry.py').read()
    # print or save data

If there's some interest, I could open a PR.
msg404076 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-10-16 12:25
Posting the idea to python-ideas list might generate interest and ideas.
msg405024 - (view) Author: Tom Pohl (tom.pohl) * Date: 2021-10-26 05:18
Thanks, Terry, for the hint.

The idea got some support on python-ideas, so I thought it is worthwhile to do a PR. As a first-time contributor, I now have to wait for approval for the pipeline to run...
msg405456 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-01 17:36
I fear that you underestimate both the complexity of this feature request and the amount of demand from users this feature is going to create. Once we start to offer a command line client for urllib, users **will** ask for more options, e.g. POST requests, custom headers, various TLS/SSL options and so on. There is a ton of hidden complexity in a download tool. curl has over 240 (!) command line options for a reason.
msg405461 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-11-01 18:12
> If curl/wget are available, great, but often slim images 
> don't offer that.

I concur with Christian.  For the most part, the standard library aims to be a collection of resources helpful for building applications like curl and wget.  The applications themselves should live on the Python Package Index (PyPI).  Also, as Christian points out, this is a non-trivial project with a large scope.


> The urllib could provide a very simple download functionality
> (like http offers a simple server):
>
>    from urllib.request import urlopen
>    data = urlopen('https://.../install-poetry.py').read()
>    # print or save data

This is a simpler request and is within our scope; however, I not sure how it differs from the existing functionality in urllib.request.urlretrieve().  Can you clarify what extra functionality is being requested?
msg405469 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-11-01 20:36
The core devs have decided to reject this. Basically for a few reasons:

- the possibility of introducing security vulnerabilities
- the ongoing maintenance burden
msg406154 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-11-11 10:55
I don't think that urllib.request is a great library. There are better replacements like https://www.python-httpx.org/ or https://urllib3.readthedocs.io/

urllib.request API is not great and it doesn't support HTTP2 (nor HTTP3).
History
Date User Action Args
2022-04-11 14:59:51adminsetgithub: 89629
2021-11-11 11:14:23tom.pohlsetnosy: - tom.pohl
2021-11-11 10:55:25vstinnersetnosy: + vstinner
messages: + msg406154
2021-11-01 20:36:28eric.smithsetstatus: open -> closed

nosy: + eric.smith
messages: + msg405469

resolution: rejected
stage: patch review -> resolved
2021-11-01 18:12:21rhettingersetnosy: + rhettinger
messages: + msg405461
2021-11-01 17:36:01christian.heimessetnosy: + christian.heimes
messages: + msg405456
2021-10-26 05:18:43tom.pohlsetmessages: + msg405024
2021-10-25 20:21:36terry.reedysetnosy: - terry.reedy
2021-10-25 19:23:27python-devsetkeywords: + patch
nosy: + python-dev

pull_requests: + pull_request27479
stage: test needed -> patch review
2021-10-16 12:25:31terry.reedysetversions: + Python 3.11
nosy: + terry.reedy

messages: + msg404076

stage: test needed
2021-10-14 06:15:12tom.pohlcreate