On Windows, packaging seems to create RECORD files with an additional CR at the end of the line. (So the line end is CR CR LF). This does not seem to be consistent, but it is likely to be because a file is being opened in text mode rather than binary.

I am trying to develop a reproducible test case, but am having difficulty at the moment. I have opened this bug in any case as a place holder and in case someone else can reproduce the issue.

packaging.util creates a RECORD file with OS-specific line endings and csv.writerow; packaging.command.install_distinfo uses LF everywhere and csv.writerow.  I’m not sure this is related to binary vs. text I/O; maybe it’s some other code like the RESOURCES system that’s giving lines with endings to install_distinfo, which then adds CRLF.  Do you have a RECORD file to upload for inspection?

Unfortunately, no. I have been unable to get this in a reproducible form - but I have seen it a few times now. I will keep trying to reproduce.

The worst thing is that packaging fails to recognise the data in RECORD and won't uninstall the package. It would be helpful if packaging were a little more permissive in how it handles this sort of problem in RECORD.

I'm trying to reproduce this too, but have failed so far. Which version(s) of Windows did the problems appear on?

Windows 7, 32 bit.

I found the problem - it's in packaging.util.write_record_file. The
file passed to csv.writer should be opened with newline=''.

Here's a patch. It includes a test, but I don't expect the test will catch the issue except on Windows...

> I found the problem - it's in packaging.util.write_record_file.
We have two functions that write RECORD files; I’ve opened a report to kill one.

> The file passed to csv.writer should be opened with newline=''.
How will we port this to 2.x?

> I don't expect the test will catch the issue except on Windows...
Do you mean that the test will fail or be a no-op on other OSes?  We can mark it as Windows-specific (@unittest.skipIf(sys.platform != 'win32', 'test only relevant on win32')) or just let it run if it’s harmless.  The important point is: does it fail before the fix, does it pass after?

On 17 October 2011 14:15, Éric Araujo <report@bugs.python.org> wrote:
>> The file passed to csv.writer should be opened with newline=''.
> How will we port this to 2.x?

No idea :-( The 2.7 documentation says use the 'b' flag, but that
probably doesn't allow an encoding parameter (it doesn't on 3.x).

>> I don't expect the test will catch the issue except on Windows...
> Do you mean that the test will fail or be a no-op on other OSes?  We can mark it as Windows-specific (@unittest.skipIf(sys.platform != 'win32', 'test only relevant on win32')) or just let it run if it’s harmless.  The important point is: does it fail before the fix, does it pass after?

The test fails before the fix, passes after. It's a no-op on platforms
where text and binary files are the same, (i.e., non-Windows systems).
So it's harmless.

>>> The file passed to csv.writer should be opened with newline=''.
>> How will we port this to 2.x?
> No idea :-( The 2.7 documentation says use the 'b' flag, but that
> probably doesn't allow an encoding parameter (it doesn't on 3.x).

Ah, I see that newline controls newline translation.  We use codecs.open in distutils2; I think it uses universal newlines mode.  We’ll figure it out, the important part is to have good tests.

> The test fails before the fix, passes after. It's a no-op on platforms
> where text and binary files are the same, (i.e., non-Windows systems).
> So it's harmless.

Great.  I’ll look at your patch and try to port it to distutils2.

Any progress on this one?

I tried to reproduce using distutils2, but don't know how distutils2 works - there's no pysetup and python -m distutils2.run doesn't work the same as python -m packaging.run.

Can we at least get packaging working in time for 3.3, even if it leaves an issue with distutils2 to sort out?

> I tried to reproduce using distutils2, but don't know how distutils2
> works - there's no pysetup and python -m distutils2.run doesn't work
> the same as python -m packaging.run.

Maybe you’re using the old PyPI release or Tarek’s outdated Bitbucket repository.  http://hg.python.org/distutils is current, contains pysetup and supports -m distutils2.run.

I’ll port your patch to d2 this week.

> Maybe you’re using the old PyPI release or Tarek’s outdated Bitbucket repository.  http://hg.python.org/distutils is current, contains pysetup and supports -m distutils2.run.

I just did pip install distutils2 in a virtualenv. If that doesn't
work then yes, I got the wrong one :-(

Do I need to hg clone then do some magic pip incantation to get it
installed? I'm new to pip/viirtualenv. I'll have a look.

> I just did pip install distutils2 in a virtualenv. If that doesn't work then yes,
> I got the wrong one :-(
You need to get a development repo to make patches, not a released version, just like for CPython.

> Do I need to hg clone then do some magic pip incantation to get it installed?
Why would you want to install it?  Clone it, hack on it in place, run tests in place.  :)

FTR, “bin/pip install -e .” is the command to “install” a project in develop (or editable) mode with pip in a virtualenv.

>> Do I need to hg clone then do some magic pip incantation to get it installed?
> Why would you want to install it?  Clone it, hack on it in place, run tests in place.  :)
>
> FTR, “bin/pip install -e .” is the command to “install” a project in develop (or editable) mode with pip in a virtualenv.

Thanks, that's what I needed. I'm new to pip and know little more than
"pip install" :-)

I should be able to run a VM to test and commit this soon.  Keep faith!

This issue is now obsolete for distutils2.  Other projects that write RECORD files are hopefully more robust in their code and tests.