classification
Title: Packaging should provide support for extensible categories
Type: enhancement Stage: committed/rejected
Components: Distutils2, Library (Lib) Versions: Python 3.3, 3rd party
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: alexis, eric.araujo, tarek, vinay.sajip
Priority: normal Keywords:

Created on 2011-06-24 08:53 by vinay.sajip, last changed 2014-03-13 02:45 by eric.araujo. This issue is now closed.

Messages (13)
msg138895 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 08:53
Some installation locations are platform-dependent and cannot be categorised into a small but fixed number of categories. This is particularly true for the Windows ecosystem - PowerShell, Office, SharePoint all have specific locations where files need to be installed for interoperability with them.

This can be catered for by a pre-hook for install_data, but some very small core changes are needed:

1. In the install_data constructor, initialise self.categories to an empty dictionary.
2. In install_data.expand_categories, add a  "local_vars.update(self.categories)" after the "local_vars = get_paths()" statement.

Just these small changes are sufficient to allow sufficient control over custom installation locations. For projects that need custom categories, they just need to do the necessary setup in an install_data pre-hook:

def pre_install_data(cmd):
    cmd.categories['customcategory'] = '/path/for/my/custom/category'

I have this working in the pythonv branch, and if the feature request is accepted I can work up a patch including changes to docs, tests etc.
msg138907 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 10:57
I think this should be part of the setup.cfg specification, not an install_data hook.
msg138910 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 11:09
@Éric: I agree that some support in setup.cfg would be good, but the problem is that you can't be 100% declarative - sometime the locations can only be determined at runtime. 

I assume you are thinking of something like:

[install_data]
categories = 
    cat1 = path1
    cat2 = path2

In my specific use case, declarative paths will not work in all scenarios - you have to use ctypes to call a Windows API (SHGetKnownFolderPath) to find the specific place to put things.

The two approaches can work together - the categories dictionary can be populated from setup.cfg where this makes sense, and of course for static paths it makes perfect sense. The Windows world is a lot more chaotic than static paths can cater for, unfortunately :-(
msg138939 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 13:13
> I assume you are thinking of something like:
Yep.

> In my specific use case, declarative paths will not work in all
> scenarios - you have to use ctypes to call a Windows API
> (SHGetKnownFolderPath) to find the specific place to put things.
Ah, too bad.

> The two approaches can work together
Yes.  Let’s implement this in setup.cfg and document how to write a hook to call Windows APIs to get app-specific paths.  See also #7175.
msg138949 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 14:36
> Yes.  Let’s implement this in setup.cfg and document how to  write a hook to 
>call Windows APIs to get app-specific paths.  See also  #7175.

When you say "let's", do you mean me, or you? :-) Just so we don't *both* do it 
...

Re. #7175, the discussion is somewhat orthogonal - the other issue ISTM is 
really about where to map appdata.* categories.
msg138965 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 17:16
Basic support for categories now added to install_data:

https://bitbucket.org/vinay.sajip/pythonv/changeset/a4822be62d90

I couldn't see a packaging.util function for matching lines of type "key = value", but if there's one I missed, I can change the implementation to suit.
msg144296 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-19 16:16
> Basic support for categories now added to install_data:
Can you add tests?

> I couldn't see a packaging.util function for matching lines of type
> "key = value",
If there are many places that could benefit from such a function, we can add it.
msg145309 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-10 17:43
> Can you add tests?

https://bitbucket.org/vinay.sajip/pythonv/changeset/f7c5098e3c3b
msg145417 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-12 17:00
Great.  Do you want to update the docs (packaging/setupscript and maybe packaging/commandref too) or shall I do it?
msg145431 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-12 21:06
> Great.  Do you want to update the docs (packaging/setupscript and maybe 

> packaging/commandref too) or shall I do it?

Feel free to do it, if that's OK with you. I've got a couple of other pythonv issues which are taking my time at the moment, plus I am also looking at Windows binary packaging as discussed recently on python-dev.
msg153671 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-19 06:20
I’ve had a look at your implementation.  It is an interesting use of properties, but doesn’t quite fit into how packaging works.  Most of the options parsing and validation (from config files and command-line alike) is done in each command’s finalize_options method, some parsing and validation is done in the config module (for the metadata, files and soon extension sections), and the section named “global” is validated by the Distribution (or maybe Dispatcher now) class.  So I’ll certainly reuse your tests, but will have to redo the implementation.

I’m also wondering if install_data’s options are the right place for this.  For example, does it make sense to allow “pysetup run install_data --categories spam=blah”?  (Any command option can be given on the command line.)  Maybe the global section of setup.cfg would be better.
msg153700 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-02-19 11:56
> So I’ll certainly reuse your tests, but will have to redo the implementation.

That's fine, but please bear in mind what I said about a 100% declarative approach being insufficient - see the Windows example I gave, where Windows APIs have to be called at installation time to determine how the categories are set up. My tests don't cater for that, but in my view it's an important requirement that shouldn't be lost. That's why I put the code into a hook, install_data being the one which required minimal changes to the code.

I'm not wedded to the specific implementation, simple though it is - I was aiming for a solution that worked and would be easy to review. I agree that install_data may not be the best section for this, and the global section seems more reasonable. But the key thing, apart from the parsing logic etc., is to allow changes / additions to the categories to be made at installation time by user-defined code (=> hook).
msg153732 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-19 23:16
> That's fine, but please bear in mind what I said about a 100% declarative approach being insufficient
Well, the config file approach supports simple cases, and for people needing Windows API calls, they will be able to use a pre-command hook.  In that case though, there will be no validation; this is a general problem with how commands and hooks work.
History
Date User Action Args
2014-03-13 02:45:45eric.araujosetstatus: open -> closed
resolution: out of date
stage: committed/rejected
2012-02-19 23:16:38eric.araujosetmessages: + msg153732
2012-02-19 11:56:29vinay.sajipsetmessages: + msg153700
2012-02-19 06:20:33eric.araujosetmessages: + msg153671
2011-10-12 21:06:54vinay.sajipsetmessages: + msg145431
2011-10-12 17:00:46eric.araujosetmessages: + msg145417
2011-10-10 17:43:20vinay.sajipsetmessages: + msg145309
2011-09-19 16:16:34eric.araujosetassignee: tarek -> eric.araujo
messages: + msg144296
versions: + 3rd party
2011-06-24 17:16:47vinay.sajipsetmessages: + msg138965
2011-06-24 14:36:03vinay.sajipsetmessages: + msg138949
2011-06-24 13:13:28eric.araujosetmessages: + msg138939
2011-06-24 11:09:15vinay.sajipsetmessages: + msg138910
2011-06-24 10:57:45eric.araujosetmessages: + msg138907
2011-06-24 08:53:25vinay.sajipcreate