classification
Title: Upload failed (400): Digests do not match on .tar.gz ending with x0d binary code
Type: security Stage: resolved
Components: Distutils Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: bbayles, dstufft, eric.araujo, llecaroz, pitrou
Priority: normal Keywords: patch

Created on 2017-12-13 15:48 by llecaroz, last changed 2018-02-23 18:23 by eric.araujo. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5264 merged bbayles, 2018-01-21 22:42
PR 5330 merged bbayles, 2018-01-26 02:01
PR 5331 merged bbayles, 2018-01-26 02:02
Messages (7)
msg308205 - (view) Author: Louis Lecaroz (llecaroz) Date: 2017-12-13 15:48
Hi,
.tar.gz files can end with x0d bytes or whatever you want

When running setup.py sdist upload, depending on the project, the .tar.gz file, as said can sometimes end with x0d. When doing the upload, the line https://github.com/python/cpython/blob/master/Lib/distutils/command/upload.py#L162 (if value and value[-1:] == b'\r') will remove the ending char of the .tar.gz generating a 400 response error from the server like: 

Upload failed (400): Digests do not match, found: 09f23b52764a6802a87dd753009c2d3d, expected: 972b8e9d3dc8cf6ba6b4b1ad5991f013
error: Upload failed (400): Digests do not match, found: 09f23b52764a6802a87dd753009c2d3d, expected: 972b8e9d3dc8cf6ba6b4b1ad5991f013

As this line is generic & run on all key/values, I clearly understand that this check was initially written to eliminate certainly some issues on values in text format. 

But the mistake here, is that you are also changing the content of the 'content' key which contains the .tar.gz as value, and because you remove the ending 0D, you change the .tar.gz content to be uploaded. As consequence, the server will return a 400 error about a wrong digest/crc.

I was able to make the code working with all .tar.gz files by changing this line to:

                if value and value[-1:] == '\r' and not key=='content':

With a such fix, the .tar.gz content will not see its ending \r to be removed & the computed CRC from the server will be the same as computed by md5(content).hexdigest() in upload.py
msg308691 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-19 23:29
I agree with the suggested fix.  Do you want to submit a PR?
msg310720 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2018-01-26 00:02
New changeset 2fc98ae115e2a2095a0bcf388c27a878aafdb454 by Éric Araujo (Bo Bayles) in branch 'master':
bpo-32304: Fix distutils upload for sdists ending with \x0d (GH-5264)
https://github.com/python/cpython/commit/2fc98ae115e2a2095a0bcf388c27a878aafdb454
msg310832 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2018-01-27 02:19
New changeset 995c60d2656c022359aac3fe713d8464c8db5716 by Éric Araujo (Bo Bayles) in branch '3.6':
[3.6] bpo-32304: Fix distutils upload for tar files ending with b'\r' (GH-5264) (GH-5330)
https://github.com/python/cpython/commit/995c60d2656c022359aac3fe713d8464c8db5716
msg311141 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2018-01-29 14:31
New changeset f5a793522d539afc84ac7888c9ad189097c43a75 by Éric Araujo (Bo Bayles) in branch '2.7':
bpo-32304: Fix distutils upload for tar files ending with b'\r' (GH-5264) (GH-5331)
https://github.com/python/cpython/commit/f5a793522d539afc84ac7888c9ad189097c43a75
msg312624 - (view) Author: Louis Lecaroz (llecaroz) Date: 2018-02-23 09:57
Hi,

First of all, thank you so much for having fixed this bug, I checked in 3.5 & it seems that this fix needs to be also backport in 3.5 branch & certainly others branches (like 3.4) ?

Thx in advance for your coming feedback
Best regards
Louis
msg312662 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2018-02-23 18:22
3.5 only accepts security fixes:
https://devguide.python.org/#status-of-python-branches
History
Date User Action Args
2018-02-23 18:23:31eric.araujosetstatus: open -> closed
stage: backport needed -> resolved
versions: - Python 3.8
2018-02-23 18:22:39eric.araujosetmessages: + msg312662
versions: + Python 3.8, - Python 3.4, Python 3.5
2018-02-23 09:57:53llecarozsetnosy: + llecaroz

messages: + msg312624
versions: + Python 3.4, Python 3.5
2018-01-29 14:31:34eric.araujosetmessages: + msg311141
2018-01-28 23:20:08eric.araujosetassignee: eric.araujo
resolution: fixed
stage: patch review -> backport needed
2018-01-27 02:19:00eric.araujosetmessages: + msg310832
2018-01-26 02:02:58bbaylessetpull_requests: + pull_request5176
2018-01-26 02:01:41bbaylessetpull_requests: + pull_request5175
2018-01-26 00:02:05eric.araujosetmessages: + msg310720
2018-01-23 03:01:12bbaylessetnosy: + bbayles
2018-01-21 22:42:57bbaylessetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request5110
2017-12-19 23:29:59pitrousetnosy: + pitrou, - llecaroz
versions: + Python 3.6, - Python 3.5, Python 3.8
messages: + msg308691
stage: needs patch
2017-12-13 15:48:29llecarozcreate