classification
Title: packaging fails in install_distinfo when writing RESOURCES
Type: behavior Stage: resolved
Components: Distutils2, Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: alexis, eric.araujo, paul.moore, python-dev, tarek, trevor, vinay.sajip
Priority: normal Keywords:

Created on 2011-06-22 11:22 by vinay.sajip, last changed 2011-11-14 14:24 by python-dev. This issue is now closed.

Messages (20)
msg138821 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-22 11:22
This part of install_distinf.run():

    if install_data.get_resources_out() != []:
        resources_path = os.path.join(self.distinfo_dir,
                                      'RESOURCES')
        logger.info('creating %s', resources_path)
        with open(resources_path, 'wb') as f:
            writer = csv.writer(f, delimiter=',',
                                lineterminator='\n',
                                quotechar='"')
            for tuple in install_data.get_resources_out():
                writer.writerow(tuple)

fails at the writerow line:

creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/METADATA
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/INSTALLER
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/REQUESTED
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/RESOURCES
Traceback (most recent call last):
  File "/tmp/venv/bin/pysetup3", line 5, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.3/packaging/run.py", line 678, in main
    return dispatcher()
  File "/usr/local/lib/python3.3/packaging/run.py", line 667, in __call__
    return func(self, self.args)
  File "/usr/local/lib/python3.3/packaging/run.py", line 204, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.3/packaging/run.py", line 247, in _install
    if install_local_project(target):
  File "/usr/local/lib/python3.3/packaging/install.py", line 125, in install_local_project
    return _run_install_from_dir(path)
  File "/usr/local/lib/python3.3/packaging/install.py", line 160, in _run_install_from_dir
    func(source_dir)
  File "/usr/local/lib/python3.3/packaging/install.py", line 90, in _run_packaging_install
    dist.run_command('install_dist')
  File "/usr/local/lib/python3.3/packaging/dist.py", line 761, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.3/packaging/command/install_dist.py", line 526, in run
    self.run_command(cmd_name)
  File "/usr/local/lib/python3.3/packaging/command/cmd.py", line 329, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python3.3/packaging/dist.py", line 761, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.3/packaging/command/install_distinfo.py", line 116, in run
    writer.writerow(tuple)
TypeError: 'str' does not support the buffer interface


I think the open(resources_path) should use 'w' rather than 'wb' as the open mode.

Relevant part of setup.cfg:

resources =
    virtualenvwrapper.sh = {scripts}

I know I can put it in the scripts = section, but I'm testing having something in resources, which ought to work ...
msg138913 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 11:24
It is strange that neither the tests for install_distinfo and resources did not catch this.  I agree about the nonsense of opening in binary mode.
msg138921 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 11:52
I suppose it's because I'm writing a script as if it were data? (i.e. to the {scripts} location.
msg142977 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-25 16:20
Vinay: I don’t understand your message.

I wasn’t part of the effort that added the resources subsystem, so I will work on a patch for this bug to learn more about it.
msg142983 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-25 17:30
Éric: I only meant that when writing data resources, one might reasonably use 'wb', but when writing scripts, which are text, 'w' is more appropriate.
msg142984 - (view) Author: trevor (trevor) Date: 2011-08-25 18:17
i see the same behavior - the error occurs leaving an empty RESOURCES file
msg142986 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-25 18:42
BTW, IIRC I have fixed it in the pythonv branch.

https://bitbucket.org/vinay.sajip/pythonv
msg143264 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-31 15:52
> I only meant that when writing data resources, one might reasonably
> use 'wb', but when writing scripts, which are text, 'w' is more
> appropriate.

I don’t see why.  All text is bytes, so we can do all I/O in bytes when writing resources and avoid special-casing.

> BTW, IIRC I have fixed it in the pythonv branch.
> https://bitbucket.org/vinay.sajip/pythonv

A link to a specific changeset or file would be great.

[trevor]
> i see the same behavior - the error occurs leaving an empty RESOURCES file

Do you see that when running a test, a command or some other code?
msg143265 - (view) Author: trevor (trevor) Date: 2011-08-31 16:00
[eric.araujo]
> Do you see that when running a test, a command or some other code?

when attempting to install a self-created package.

tools from 3.3a ~72060:1696e2789d91
msg143282 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-31 21:53
> I don’t see why.  All text is bytes, so we can do all I/O in bytes
> when writing resources and avoid special-casing.

I was only commenting on how the bug might have come about.

> A link to a specific changeset or file would be great.

Lib/packaging/command/install_distinfo.py">https://bitbucket.org/vinay.sajip/pythonv/changeset/3e3a07a94f69#chg-Lib/packaging/command/install_distinfo.py
msg143321 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-09-01 12:35
@Éric: I also noticed that your latest comment 

"All text is bytes, so we can do all I/O in bytes when writing resources and avoid special-casing."

contradicts your earlier position:

"I agree about the nonsense of opening in binary mode."

If using text mode, we might lose any specific encoding used for the source script unless we do encoding-detection as the interpreter does, and preserve that across the change-shebang-and-save. Using binary does avoid this, and should be doable with simpler code, so perhaps it is preferable - as long as we do it correctly so as to avoid the TypeError :-)
msg143465 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-03 16:11
> I also noticed that your latest comment 
> "All text is bytes, so we can do all I/O in bytes when writing
> resources and avoid special-casing."
> contradicts your earlier position:
> "I agree about the nonsense of opening in binary mode."

It does :)  My earlier position was wrong.
msg144015 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-09-14 08:31
I looked at this bug again - I was getting a little confused about it ;-) The problem is happening not when writing out a resource, but the RESOURCES file listing the resources installed. This is a text file, of course, so my suggested fix of using open(resources_path, 'w', encoding='utf-8') would be reasonable.
msg145237 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-09 08:07
I’d like to fix this.  How can I reproduce the bug?
msg145247 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-09 09:21
I got the problem when installing a package with resources using pysetup3. Here's the relevant part of the console session:

(venv) vinay@eta-natty:~/projects$ pysetup3 install nemo
Installing from source directory: /home/vinay/projects/nemo
running install_dist
running build
running build_py
running build_scripts
running install_lib
creating /tmp/venv/lib/python3.3/site-packages
creating /tmp/venv/lib/python3.3/site-packages/virtualenvwrapper
byte-compiling /tmp/venv/lib/python3.3/site-packages/virtualenvwrapper/hook_loader.py to hook_loader.pyc
byte-compiling /tmp/venv/lib/python3.3/site-packages/virtualenvwrapper/user_scripts.py to user_scripts.pyc
byte-compiling /tmp/venv/lib/python3.3/site-packages/virtualenvwrapper/__init__.py to __init__.pyc
byte-compiling /tmp/venv/lib/python3.3/site-packages/nemo.py to nemo.pyc
running install_scripts
changing mode of /tmp/venv/bin/nemo to 755
running pre_hook hooks.pre_install_data for command install_data
running install_data
running install_distinfo
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/METADATA
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/INSTALLER
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/REQUESTED
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/RESOURCES
Traceback (most recent call last):
  File "/tmp/venv/bin/pysetup3", line 6, in <module>
    rc = packaging.run.main() # None interpreted as 0
  File "/usr/local/lib/python3.3/packaging/run.py", line 653, in main
    return dispatcher()
  File "/usr/local/lib/python3.3/packaging/run.py", line 642, in __call__
    return func(self, self.args)
  File "/usr/local/lib/python3.3/packaging/run.py", line 91, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.3/packaging/run.py", line 164, in _install
    return not install_local_project(target)
  File "/usr/local/lib/python3.3/packaging/install.py", line 122, in install_local_project
    return _run_install_from_dir(path)
  File "/usr/local/lib/python3.3/packaging/install.py", line 160, in _run_install_from_dir
    func(source_dir)
  File "/usr/local/lib/python3.3/packaging/install.py", line 87, in _run_packaging_install
    dist.run_command('install_dist')
  File "/usr/local/lib/python3.3/packaging/dist.py", line 709, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.3/packaging/command/install_dist.py", line 508, in run
    self.run_command(cmd_name)
  File "/usr/local/lib/python3.3/packaging/command/cmd.py", line 330, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python3.3/packaging/dist.py", line 709, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.3/packaging/command/install_distinfo.py", line 113, in run
    writer.writerow(row)
TypeError: 'str' does not support the buffer interface

This is the relevant code in install_distinfo.run():

                resources_path = os.path.join(self.distinfo_dir,
                                              'RESOURCES')
                logger.info('creating %s', resources_path)
                if not self.dry_run:
                    #with open(resources_path, 'w', encoding='utf-8') as f:
                    with open(resources_path, 'wb') as f:
                        writer = csv.writer(f, delimiter=',',
                                            lineterminator='\n',
                                            quotechar='"')
                        for row in install_data.get_resources_out():
                            writer.writerow(row)

If I substitute the commented out line above which replaces 'wb' with 'w' and encoding, I get this result:

(venv) vinay@eta-natty:~/projects$ pysetup3 install nemo
Installing from source directory: /home/vinay/projects/nemo
running install_dist
running build
running build_py
running build_scripts
running install_lib
byte-compiling /tmp/venv/lib/python3.3/site-packages/virtualenvwrapper/hook_loader.py to hook_loader.pyc
byte-compiling /tmp/venv/lib/python3.3/site-packages/virtualenvwrapper/user_scripts.py to user_scripts.pyc
byte-compiling /tmp/venv/lib/python3.3/site-packages/virtualenvwrapper/__init__.py to __init__.pyc
byte-compiling /tmp/venv/lib/python3.3/site-packages/nemo.py to nemo.pyc
running install_scripts
changing mode of /tmp/venv/bin/nemo to 755
running pre_hook hooks.pre_install_data for command install_data
running install_data
running install_distinfo
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/METADATA
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/INSTALLER
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/REQUESTED
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/RESOURCES
creating /tmp/venv/lib/python3.3/site-packages/nemo-0.1.dist-info/RECORD
msg145460 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-13 15:43
I started to look at this a few days ago and found out that there are no tests at all for writing RESOURCES.  I need to look again at the documentation and code and add many tests.
msg145515 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-14 13:41
I have a test.
msg145527 - (view) Author: Roundup Robot (python-dev) Date: 2011-10-14 15:06
New changeset 1f3459b08298 by Éric Araujo in branch 'default':
Fix writing of the RESOURCES file by packaging (#12386)
http://hg.python.org/cpython/rev/1f3459b08298
msg145528 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-14 15:10
I’ve reproduced the bug and fixed it by opening in text mode, following all other calls to open in the same file.  This should now work for ASCII paths and non-ASCII paths that use the same encoding as open’s default, but it’ll probably break with non-ASCII paths under a C locale, so I’ve opened #13178.
msg147598 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-14 14:24
New changeset cb49bc384957 by Éric Araujo in branch 'default':
Fix writing of the RESOURCES file (#12386).
http://hg.python.org/distutils2/rev/cb49bc384957

New changeset 2d469ccfe30e by Éric Araujo in branch 'python3':
Merge fixes for #13170 and #12386 and other misc. changes from default
http://hg.python.org/distutils2/rev/2d469ccfe30e
History
Date User Action Args
2011-11-14 14:24:20python-devsetmessages: + msg147598
2011-10-14 15:10:44eric.araujosetstatus: open -> closed

nosy: + paul.moore
messages: + msg145528

resolution: fixed
stage: needs patch -> resolved
2011-10-14 15:06:10python-devsetnosy: + python-dev
messages: + msg145527
2011-10-14 13:41:33eric.araujosetmessages: + msg145515
2011-10-13 15:43:29eric.araujosetmessages: + msg145460
2011-10-13 15:42:06eric.araujolinkissue13162 superseder
2011-10-09 09:21:19vinay.sajipsetmessages: + msg145247
2011-10-09 08:07:57eric.araujosetmessages: + msg145237
2011-09-14 08:31:18vinay.sajipsetmessages: + msg144015
2011-09-03 16:11:56eric.araujosetmessages: + msg143465
2011-09-01 12:35:13vinay.sajipsetmessages: + msg143321
2011-08-31 21:53:12vinay.sajipsetmessages: + msg143282
2011-08-31 16:00:10trevorsetmessages: + msg143265
2011-08-31 15:52:41eric.araujosetmessages: + msg143264
2011-08-25 18:42:45vinay.sajipsetmessages: + msg142986
2011-08-25 18:17:50trevorsetmessages: + msg142984
2011-08-25 17:30:11vinay.sajipsetmessages: + msg142983
2011-08-25 16:20:22eric.araujosetmessages: + msg142977
2011-08-24 09:53:59trevorsetnosy: + trevor
2011-06-24 11:52:57vinay.sajipsetmessages: + msg138921
2011-06-24 11:24:29eric.araujosetassignee: tarek -> eric.araujo
messages: + msg138913
stage: needs patch
2011-06-23 07:51:11vinay.sajipsettype: behavior
2011-06-22 11:22:24vinay.sajipcreate