classification
Title: Accept arbitrary files for distutils' upload command
Type: enhancement Stage: needs patch
Components: Distutils Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Aaron.Meurer, alexis, asvetlov, eric.araujo, exarkun, frougon, glyph, illume, tarek
Priority: normal Keywords: easy

Created on 2011-09-09 06:32 by illume, last changed 2014-03-12 08:33 by eric.araujo.

Messages (9)
msg143754 - (view) Author: Rene Dudfield (illume) Date: 2011-09-09 06:32
We need to specify files to upload to pypi.  Otherwise we have to use the web interface.

'Regarding the "setup.py upload" command, this isn't very helpful because it will not upload a package that was already built - for example, a Windows package built on another host while I am trying to release from a Linux desktop. Since automatic package building across a build farm is the only way I can actually build packages for most platforms, this rules out use of the "upload" command.'

See this post for more details:
http://as.ynchrono.us/2011/09/releasing-python-software-is-tedious.html
msg144303 - (view) Author: Glyph Lefkowitz (glyph) Date: 2011-09-19 17:45
This has been a serious annoyance in the Twisted and pyOpenSSL release processes; it effectively prevents us from timely uploads of win32 binaries of any kind.
msg144382 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-21 15:09
This was discussed some time ago in a thread starting here: http://mail.python.org/pipermail/catalog-sig/2011-May/003736.html  I’ll summarize the important parts below.

Currently, upload will only push files that are products of a command run from the same command line, e.g. “sdist upload”.
If you run “sdist”, check the result, and then do “sdist upload”, the second sdist should be exactly the same, unless your setup.py is buggy, you made changes to your files or you have a computer problem.  (I agree that’s a lot of unless).  So for distutils, having to run “sdist upload” is not inefficient, and makes you be explicit about the files you want to push, which is IMO good.

A trick can be used to avoid the second sdist to redo all its work: Fist you run “python setup.py sdist --keep-temp”, then you check the sdist, and to upload you call “python setup.py sdist --dry-run upload”.  I’m not in favor of adding that trick to the doc, as for normal usage, running sdist twice is okay.

As you may not be aware, distutils is under a feature freeze.  To cut a long story short, let’s just say that it’s impossible to gradually improve it without breaking a lot of third-party code, so we only fix bugs and we add features in distutils2, the next gen (included in Python 3.3 and available on PyPI for 2.4-3.2 soon).

I think there are good arguments to improve upload for distutils2, mainly:
- you never know what can go wrong, so you want to upload files you have tested
- PyPI will reject bad files anyway, no sense in being more restrictive on the client side

I propose a new option, say -f/--file, that takes a filename as argument:

  pysetup run upload -f dist/spam-0.2.tar.gz -f dist/spam-0.2.exe

I’d prefer to avoid globs (upload -i 'dist.spam-0.1.*' e.g.) because I want to retain the original design choice of the upload command: the person doing the upload needs to be explicit about what kind of file for what versions and with what Python version to upload.

Does that sounds good?
msg144471 - (view) Author: Jean-Paul Calderone (exarkun) * (Python committer) Date: 2011-09-23 19:25
> pysetup run upload -f dist/spam-0.2.tar.gz -f dist/spam-0.2.exe

I'm not sure why it's "run upload" instead of just "upload", but maybe that's the convention in pysetup.  Apart from that, this looks like a vast improvement to me.  As far as not supporting globs goes, I guess I don't really care either way.  If globs aren't supported, then I'll probably end up globbing somewhere else (or constructing the full upload list in some other automated way).  The motivation for the complaint is that too many things require manual interaction; I'm not going to use a new, easily automated utility manually. :)
msg144516 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-25 01:59
> I'm not sure why it's "run upload" instead of just "upload"
pysetup has actions and commands.  Actions do stuff on the Python installation: “pysetup metadata foobar” prints metadata, “pysetup remove foobar” uninstalls, “pysetup install foobar” fetches from PyPI and installs, etc.  Commands are like distutils commands, they work on a project in a source directory; for namespacing reasons, they are subitems of the run command.  (I’m working on documenting that.)

> Apart from that, this looks like a vast improvement to me.
Good!  Adding the easy keyword, as it’s not hard to implement if one understands how upload works, or someone could add tests without much difficulty.  I may get to it myself shortly, it will make a nice change from bug fixes and improve my karma :)
msg148874 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-12-05 15:01
Someone from the mentoring project has volunteered to work on a patch.
msg178249 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-12-26 20:14
Close as out of date since packaging has been removed from stdlib.
msg179343 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-01-08 12:18
distutils2, while not actively developed any more for inclusion in the stdlib in 3.4, will mutate into one or more projects where these bug reports will be useful.  Bugs will certainly be migrated (and closed in this tracker), but please keep them open for easy query.
msg213221 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-12 08:33
With the lift of the distutils feature freeze decided at PyCon 2013, this feature request could be done for 3.5.
History
Date User Action Args
2014-03-12 08:33:42eric.araujosettitle: Accept arbitrary files for packaging's upload command -> Accept arbitrary files for distutils' upload command
messages: + msg213221
versions: + Python 3.5, - Python 3.4
2014-03-12 08:32:39eric.araujosetmessages: - msg193103
2013-09-19 12:39:47frougonsetnosy: + frougon
2013-07-15 14:53:00Aaron.Meurersetnosy: + Aaron.Meurer
2013-07-15 14:12:11eric.araujosetmessages: + msg193103
components: + Distutils, - Distutils2
versions: + Python 3.4, - 3rd party
2013-01-08 12:18:45eric.araujosetstatus: closed -> open
resolution: out of date ->
messages: + msg179343

versions: - Python 3.3
2012-12-26 20:14:40asvetlovsetstatus: open -> closed

nosy: + asvetlov
messages: + msg178249

resolution: out of date
2012-10-21 20:32:20mikehoysetnosy: - mikehoy
2011-12-05 15:01:05eric.araujosetmessages: + msg148874
2011-10-17 06:42:52mikehoysetnosy: + mikehoy
2011-09-25 01:59:31eric.araujosetkeywords: + easy

messages: + msg144516
stage: needs patch
2011-09-23 19:25:34exarkunsetnosy: + exarkun
messages: + msg144471
2011-09-21 15:09:27eric.araujosetassignee: tarek -> eric.araujo

components: + Distutils2, - Distutils
title: setup.py upload to pypi needs to work with specified files -> Accept arbitrary files for packaging's upload command
nosy: + alexis
versions: + 3rd party, Python 3.3
messages: + msg144382
2011-09-19 17:45:30glyphsetnosy: + glyph
messages: + msg144303
2011-09-09 06:32:48illumecreate