classification
Title: use HTTPS by default for uploading packages to pypi
Type: security Stage: committed/rejected
Components: Distutils, Distutils2 Versions: Python 3.4, Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: 17129 Superseder:
Assigned To: Nosy List: Arfrever, Giovanni.Bajo, alexis, barry, benjamin.peterson, christian.heimes, devin, dstufft, eric.araujo, fdrake, georg.brandl, jwilk, larry, loewis, pitrou, python-dev, richard, skrah, tarek, techtonik
Priority: high Keywords: patch

Created on 2011-05-31 16:51 by techtonik, last changed 2013-12-22 00:50 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
pypy.https.patch techtonik, 2011-05-31 16:51 review
Messages (48)
msg137366 - (view) Author: anatoly techtonik (techtonik) Date: 2011-05-31 16:51
Before the next version is released, I'd like to push this one line modification to reduce the risk of sniffing Python development password when people upload packages to PyPI by using https:// communication channel by default.

Distutils doesn't validate PyPI server certificate, so this change doesn't prevent from MITM attacks, but at least it makes package submissions over wireless channels and public networks safer.

Taking into account that people still release packages for Python 2.5+ (AppEngine), I'd like to see this fix backported to at least Python 2.6
msg137367 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-05-31 16:51
This should probably be discussed on catalog-SIG, not the CPython bug tracker.
msg137368 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-05-31 16:52
Oops, nevermind that, thought this was suggesting a change to PyPI itself, not distutils.
msg137369 - (view) Author: anatoly techtonik (techtonik) Date: 2011-05-31 16:54
The distutils version that exposes this vulnerability is shipped with Python, that's why it is here.
msg137383 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-31 19:13
Have you tried registering/uploading stuff through https ? if it's working as is I am +1 for this change in the next 2.7.x. 

For 2.6 don't know if it's worth a security patch. Martin, Barry, any opinion ?
msg137385 - (view) Author: anatoly techtonik (techtonik) Date: 2011-05-31 20:15
Yes. I've just uploaded http://pypi.python.org/pypi/review/r749 through https://pypi.python.org/pypi
msg137417 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-06-01 05:23
I'm +0 on changing 2.6. Many people use setuptools or distribute, so the impact of this change was higher if those were modified instead of modifying 2.6 - in particular since few people are expected to pick up these security releases, anyway.
msg137424 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-06-01 07:30
> Distutils doesn't validate PyPI server certificate, so this change
> doesn't prevent from MITM attacks, but at least it makes package
> submissions over wireless channels and public networks safer.

Is that so? It's been a while, but I think e.g. ettercap is a highly
automated tool for MITM attacks that isn't very hard to use.
msg137428 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-06-01 11:06
Given that 2.6.7 is rc2 with a final release scheduled in 2 days, I don't want to apply this to 2.6 right now.  Can you guarantee this won't regress for anybody?  If so, then I'm also +0 for 2.6 after the 2.6.7 release.
msg137435 - (view) Author: anatoly techtonik (techtonik) Date: 2011-06-01 15:11
On Wed, Jun 1, 2011 at 10:30 AM, Stefan Krah <report@bugs.python.org> wrote:
>
>> Distutils doesn't validate PyPI server certificate, so this change
>> doesn't prevent from MITM attacks, but at least it makes package
>> submissions over wireless channels and public networks safer.
>
> Is that so? It's been a while, but I think e.g. ettercap is a highly
> automated tool for MITM attacks that isn't very hard to use.

This patch won't help against properly baited ettercap, but will
prevent transit sniffing of weakly protected passwords.
--
anatoly t.
msg137436 - (view) Author: anatoly techtonik (techtonik) Date: 2011-06-01 15:14
On Wed, Jun 1, 2011 at 2:06 PM, Barry A. Warsaw <report@bugs.python.org> wrote:
>
> Barry A. Warsaw <barry@python.org> added the comment:
>
> Given that 2.6.7 is rc2 with a final release scheduled in 2 days, I don't want to apply this to 2.6 right now.  Can you guarantee this won't regress for anybody?  If so, then I'm also +0 for 2.6 after the 2.6.7 release.

Adding catalog-sig to CC. I can guarantee this for Windows. I'll be
near Linux box tomorrow and will try upload to PyPI from there. It
still will be more authoritative if more than one person can test
upload to PyPI with this patch on different systems.
--
anatoly t.
msg137437 - (view) Author: Fred L. Drake, Jr. (fdrake) (Python committer) Date: 2011-06-01 15:17
On Wed, Jun 1, 2011 at 11:14 AM, anatoly techtonik <techtonik@gmail.com> wrote:
> Adding catalog-sig to CC. I can guarantee this for Windows. I'll be
> near Linux box tomorrow and will try upload to PyPI from there. It
> still will be more authoritative if more than one person can test
> upload to PyPI with this patch on different systems.

The interesting case will be for a build that doesn't include SSL support.

  -Fred
msg137438 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-06-01 15:22
On Wed, Jun 1, 2011 at 5:17 PM, Fred Drake <fdrake@acm.org> wrote:
> On Wed, Jun 1, 2011 at 11:14 AM, anatoly techtonik <techtonik@gmail.com> wrote:
>> Adding catalog-sig to CC. I can guarantee this for Windows. I'll be
>> near Linux box tomorrow and will try upload to PyPI from there. It
>> still will be more authoritative if more than one person can test
>> upload to PyPI with this patch on different systems.
>
> The interesting case will be for a build that doesn't include SSL support.

Yeah.. We do have in packaging a way to test against a PyPI server
that gets launched on a real socket, so what we could do is:

1/ add a test that tries to upload and register via ssh
2/ make sure it fallbacks to http if _ssl is not found

>
>
>  -Fred
>
> --
> Fred L. Drake, Jr.    <fdrake at acm.org>
> "Give me the luxuries of life and I will willingly do without the necessities."
>    --Frank Lloyd Wright
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG@python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>
msg137548 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-03 16:22
Patch needs to update the default server in packaging too.
msg137569 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-06-03 18:39
I think there should be a warning that the connection is unauthenticated
(i.e. not secure). Users tend to be upset if they see 'https' and later
find out that no certificates were verified.


A reasonably secure alternative is to publish the pypi server
certificate in a couple of places (python-dev, www.python.org).
Then the user can import the certificate into the browser while
on a trusted connection and henceforth do all uploading etc.
via the browser.
msg137575 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-06-03 19:47
If users use a browser to do secure uploading, there is no need to publish the certificate. It is signed by a trusted CA (cacert), so you just need to make sure your browser knows about the cacert certificate.
msg137584 - (view) Author: anatoly techtonik (techtonik) Date: 2011-06-03 20:52
Sorry for the delay. I've just uploaded http://pypi.python.org/pypi/ctypesgen/0.r125 from Ubuntu using python2.6 with patched distutils module to https://pypi.python.org/pypi and can confirm it works without problems on Linux.

So I can guarantee there won't be any regress in 99.9% of cases and <0.1% of cases that are left, risk of breaking these flaky setups doesn't outweight the need to close this security hole.

I can fill separate bug for packaging, server certificate validation and support for non-SSL builds of python (for which there is workaround with upload -r <server> option that I've just discovered), but it is an additional delay and the only reason I invested some time into this issue is because I saw RC announcements.


P.S. To Martin. CAcert is not trusted authority on Windows, and all browsers warn about it, so we need to distribute its root certificate with Python if we want to validate ours.
msg137610 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-06-03 22:59
> I can fill separate bug for packaging, server certificate validation
> and support for non-SSL builds of python (for which there is
> workaround with upload -r <server> option that I've just discovered),
> but it is an additional delay and the only reason I invested some
> time into this issue is because I saw RC announcements.

This effort was wasted - there is zero chance that this can go into
a release candidate. Only fundamental flaws (such as being unable to
compile Python on a major platform, or serious regressions) should be
considered for changes between a rc and the final release.
msg137626 - (view) Author: anatoly techtonik (techtonik) Date: 2011-06-04 05:01
I believe that's a very personal judgement. For me exposing core Python development accounts is a fundamental flaw. The more accounts are collected, the more real are attacks through PyPI package injection.
msg137638 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-04 14:33
> I think there should be a warning that the connection is
> unauthenticated (i.e. not secure). Users tend to be upset if they see
> 'https' and later find out that no certificates were verified.

Thanks Stephan, that was on my mind but I forgot it.  I’m -1 on using https if no validation is performed.

> I believe that's a very personal judgement.

Not really; it’s an explanation of our release rules, exposed by one of the older developers.

> For me exposing core Python development accounts is a fundamental
> flaw.

What is a core Python development account?
msg137732 - (view) Author: anatoly techtonik (techtonik) Date: 2011-06-06 07:54
On Sat, Jun 4, 2011 at 5:33 PM, Éric Araujo <report@bugs.python.org> wrote:>
>> I think there should be a warning that the connection is
>> unauthenticated (i.e. not secure). Users tend to be upset if they see
>> 'https' and later find out that no certificates were verified.
>
> Thanks Stephan, that was on my mind but I forgot it.  I’m -1 on using https if no validation is performed.

It will be more professional if you could also explain why. Thanks.

>> I believe that's a very personal judgement.
> Not really; it’s an explanation of our release rules, exposed by one of the older developers.

Release rules should be clear enough not to require explanation.

>> For me exposing core Python development accounts is a fundamental
>> flaw.

> What is a core Python development account?

'core' is not the best word here, so it needs an explanation. Any
account on PyPI that uploads packages used for in enterprise
deployment schemes imposes a danger. Potential target are identified
using 'popularity package/developer activity' rating to reduce the
risk. These are the primary targets for an attack, which I called
'core'. 'primary' would be a better name probably.
msg137745 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-06 15:50
>> Thanks Stephan, that was on my mind but I forgot it.  I’m -1 on
>> using https if no validation is performed.
> It will be more professional if you could also explain why.

If you make an HTTPS connection without checking the certificate, what security does it add?

> > Not really; it’s an explanation of our release rules, exposed by
>> one of the older developers.
> Release rules should be clear enough not to require explanation.

Explanations make them clear.

> Any account on PyPI that uploads packages used for in enterprise
> deployment schemes imposes a danger.

Sidenote: I don’t want to give less security to non-enterprise users.

Anyway, I understand your point now: insecure upload and download are vulnerable to MITM attacks, and encouraging HTTPS use (through default value + docs) would help against that.  I am supportive of a patch, but it doesn’t mean the release process should not be followed.  See also #11357 and #8561 about download security.
msg137747 - (view) Author: Fred L. Drake, Jr. (fdrake) (Python committer) Date: 2011-06-06 16:01
On Mon, Jun 6, 2011 at 11:50 AM, Éric Araujo <report@bugs.python.org> wrote:
> If you make an HTTPS connection without checking the certificate, what security does it add?

I'm in favor of cert checking, myself.
msg137806 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-06-07 11:16
> If you make an HTTPS connection without checking the certificate, what 
> security does it add?

Well, it does prevent the most trivial class of attacks (sniffing).
That said, Python has support for certificate checking, especially in 3.2+, so you should use that. You could e.g. bundle the CACert root certificate with the distribution.
msg138577 - (view) Author: anatoly techtonik (techtonik) Date: 2011-06-18 09:26
This simple patch slipped off 2.7.2. Why?
msg138598 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-18 20:52
Because it’s not finished.
msg138601 - (view) Author: anatoly techtonik (techtonik) Date: 2011-06-18 21:27
What is left?
msg138603 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-18 21:29
Certificate checking.
msg138612 - (view) Author: anatoly techtonik (techtonik) Date: 2011-06-18 21:55
That's the issue12358.
msg182204 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-02-16 00:10
CVE-2013-1754  Man-in-the-middle vulnerability in package upload feature of Python's distutils
msg182871 - (view) Author: Giovanni Bajo (Giovanni.Bajo) Date: 2013-02-24 11:52
Please notice that a redesign of PyPI and package security is ongoing in catalog-sig.
msg184557 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2013-03-18 23:44
New changeset f86d46a580d8 by Benjamin Peterson in branch 'default':
use the HTTPS for pypi upload
http://hg.python.org/cpython/rev/f86d46a580d8
msg185118 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-03-24 05:59
Benjamin, you committed a change to use HTTPS instead of HTTP.  In this bug report, we were having a discussion about the false/incomplete security that this provides if there is no certificate checking.  What are your thoughts on that?
msg185133 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-03-24 12:31
This is true, but if we get proper certificate checking, this should automatically work correctly then.
msg185182 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-03-25 03:11
I’m not sure what “this” refers to (in “This is true” and “this should automatically work correctly”).

My only concern is to avoid giving a false sense of security, so my initial stance was all-or-nothing.  However with the recent trend of incremental improvements to the PyPI ecosystem, I think it’s important to do what we can and keep the momentum, so I’m okay with the commit—I just wanted to make sure that committing half a fix was intentional.  You probably know more about SSL than me and you’re the RM, so let’s ship this. :)
msg185183 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-03-25 03:23
By "this", I meant the change I made. It was made in consultation with Richard Jones (added to nosy) at the PyCon sprints.
msg185184 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-03-25 03:57
Using HTTPS without a Certificate prevents passive attacks but not active attacks. It puts things in a _better_ situation but not the ideal situation.
msg190828 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-08 21:46
This should have been backported to Python 2. I expect some related attacks on EuroPython.
msg190832 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-06-08 22:18
I would +! backporting this, but It's not massively required since it only protects against passive attacks.

It would however make things a little nicer.
msg190833 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-08 22:26
If somebody sponsor my visit to EuroPython, I will dedicate some time to prepare a demo uploading rogue packages using sniffed credentials over WiFi without owner's consent. After moving to CDN no upload logs are available, so it is even more secure for attacker to do this stuff.
msg190834 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-06-08 22:28
Uploading always hits the backend servers and thus has the same logging as before

Merely switching to HTTPS only provides protections against passive attacks. You need verification to protect against active attacks (which are simple and easy to do as well). Like I said, not a bad move and I'd be in agreement on doing it but if the powers that be decide it's too big of a change it's not going to massively decrease the security.
msg202549 - (view) Author: anatoly techtonik (techtonik) Date: 2013-11-10 19:01
How come that this CVE is still present in just released 2.7.6?
msg203879 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-11-22 21:08
Donald assesses that porting the changeset to 2.7 would “make things a little nicer”, as it protects from passive attacks only.  The change is small.  What do people think?
msg203880 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-22 21:10
Well, passive attacks are the easiest to mount by a casual attacker, so I think this is important to get in.
msg203884 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-22 21:25
How about:

- load ca cert from default verify locations
- try connect with CERT_REQUIRED
- print warning when cert validation fails and try again with CERT_NONE
- match hostname otherwise

At least this warns the user about the issue. Is there way to distinguish between CA missing and other failures?

Antoine Pitrou <report@bugs.python.org> schrieb:
>
>Antoine Pitrou added the comment:
>
>Well, passive attacks are the easiest to mount by a casual attacker, so
>I think this is important to get in.
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue12226>
>_______________________________________
msg206797 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-22 00:36
New changeset 32a39ec6bd75 by Antoine Pitrou in branch '2.7':
Issue #12226: HTTPS is now used by default when connecting to PyPI.
http://hg.python.org/cpython/rev/32a39ec6bd75
msg206798 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-22 00:48
New changeset 2b5cd6d4d149 by Antoine Pitrou in branch '3.2':
Issue #12226: HTTPS is now used by default when connecting to PyPI.
http://hg.python.org/cpython/rev/2b5cd6d4d149

New changeset e5a9755c967c by Antoine Pitrou in branch '3.3':
Issue #12226: HTTPS is now used by default when connecting to PyPI.
http://hg.python.org/cpython/rev/e5a9755c967c

New changeset 9839aa0e5b28 by Antoine Pitrou in branch 'default':
Null merge (#12226 already fixed on default)
http://hg.python.org/cpython/rev/9839aa0e5b28
msg206799 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-22 00:50
Closing as fixed, and opening a new issue for cert checking.
History
Date User Action Args
2013-12-22 00:50:04pitrousetstatus: open -> closed
resolution: fixed
messages: + msg206799

stage: committed/rejected
2013-12-22 00:48:17python-devsetmessages: + msg206798
2013-12-22 00:36:00python-devsetnosy: + python-dev
messages: + msg206797
2013-12-22 00:29:31pitrousetassignee: eric.araujo ->
versions: - Python 2.6
2013-11-22 21:25:22christian.heimessetmessages: + msg203884
2013-11-22 21:10:23pitrousetmessages: + msg203880
2013-11-22 21:08:59eric.araujosetmessages: + msg203879
2013-11-10 19:01:57techtoniksetmessages: + msg202549
2013-06-08 22:28:41dstufftsetmessages: + msg190834
2013-06-08 22:26:19techtoniksetmessages: + msg190833
2013-06-08 22:18:08dstufftsetmessages: + msg190832
2013-06-08 21:46:04techtoniksetmessages: + msg190828
2013-03-25 03:57:38dstufftsetnosy: + dstufft
messages: + msg185184
2013-03-25 03:23:42benjamin.petersonsetnosy: + richard
messages: + msg185183
2013-03-25 03:11:05eric.araujosetmessages: + msg185182
2013-03-24 12:31:53benjamin.petersonsetmessages: + msg185133
2013-03-24 05:59:50eric.araujosetmessages: + msg185118
2013-03-18 23:44:20Arfreversetmessages: + msg184557
2013-02-24 11:52:57Giovanni.Bajosetnosy: + Giovanni.Bajo
messages: + msg182871
2013-02-23 18:24:38devinsetnosy: + devin
2013-02-16 00:10:53christian.heimessetnosy: + christian.heimes
messages: + msg182204
2013-02-04 22:20:25eric.araujosetpriority: release blocker -> high
assignee: tarek -> eric.araujo
2013-02-04 19:33:56christian.heimessetdependencies: + Include CA bundle and provide access to system's CA
2013-02-04 17:10:56christian.heimessetpriority: normal -> release blocker
nosy: + larry, benjamin.peterson, georg.brandl

versions: + Python 3.4, - Python 3.1
2011-06-18 21:55:33techtoniksetmessages: + msg138612
2011-06-18 21:29:40eric.araujosetmessages: + msg138603
2011-06-18 21:27:38techtoniksetmessages: + msg138601
2011-06-18 20:54:59eric.araujolinkissue12357 superseder
2011-06-18 20:54:59eric.araujounlinkissue12357 dependencies
2011-06-18 20:54:25eric.araujolinkissue12358 superseder
2011-06-18 20:52:38eric.araujolinkissue12357 dependencies
2011-06-18 20:52:20eric.araujosetmessages: + msg138598
2011-06-18 09:26:30techtoniksetmessages: + msg138577
title: use secured channel for uploading packages to pypi -> use HTTPS by default for uploading packages to pypi
2011-06-07 11:16:05pitrousetnosy: + pitrou
messages: + msg137806
2011-06-06 16:01:09fdrakesetmessages: + msg137747
2011-06-06 15:50:56eric.araujosetmessages: + msg137745
2011-06-06 07:54:26techtoniksetmessages: + msg137732
2011-06-04 14:33:46eric.araujosetmessages: + msg137638
2011-06-04 05:01:15techtoniksetmessages: + msg137626
2011-06-03 22:59:20loewissetmessages: + msg137610
2011-06-03 20:52:44techtoniksetmessages: + msg137584
2011-06-03 19:47:06loewissetmessages: + msg137575
2011-06-03 18:39:04skrahsetmessages: + msg137569
2011-06-03 16:22:05eric.araujosetmessages: + msg137548
2011-06-01 15:41:35jwilksetnosy: + jwilk
2011-06-01 15:22:52tareksetmessages: + msg137438
2011-06-01 15:17:47fdrakesetnosy: + fdrake
messages: + msg137437
2011-06-01 15:14:59techtoniksetmessages: + msg137436
2011-06-01 15:11:11techtoniksetmessages: + msg137435
2011-06-01 11:06:14barrysetmessages: + msg137428
2011-06-01 07:30:14skrahsetnosy: + skrah
messages: + msg137424
2011-06-01 05:23:45loewissetmessages: + msg137417
2011-05-31 20:15:26techtoniksetmessages: + msg137385
2011-05-31 19:13:19tareksetnosy: + loewis, barry
messages: + msg137383
2011-05-31 18:24:25Arfreversetnosy: + Arfrever

versions: + Python 3.2, Python 3.3
2011-05-31 16:54:23brian.curtinsetnosy: - brian.curtin
2011-05-31 16:54:08techtoniksetmessages: + msg137369
2011-05-31 16:52:43brian.curtinsetnosy: + brian.curtin
messages: + msg137368
2011-05-31 16:51:54brian.curtinsetnosy: - brian.curtin
2011-05-31 16:51:45brian.curtinsetnosy: + brian.curtin
messages: + msg137367
2011-05-31 16:51:04techtonikcreate