modulefinder.py does not open source files in "rb" which
prevents compile() from applying the encoding rules.

This first showed up for me on Windows with Python 3.8.

Here is my test case and the results on Windows with 3.8.

import modulefinder

with open('import_functools.py', 'w') as f:
   f.write('import functools\n')

mf = modulefinder.ModuleFinder()
mf.run_script('import_functools.py')
print('Test passed')

py -3.8-32 modulefinder_test.py
Traceback (most recent call last):
 File "modulefinder_test.py", line 7, in <module>
   mf.run_script('import_functools.py')
 File "C:\Python38.win32\lib\modulefinder.py", line 165, in run_script
   self.load_module('__main__', fp, pathname, stuff)
 File "C:\Python38.win32\lib\modulefinder.py", line 360, in load_module
   self.scan_code(co, m)
 File "C:\Python38.win32\lib\modulefinder.py", line 433, in scan_code
   self._safe_import_hook(name, m, fromlist, level=0)
 File "C:\Python38.win32\lib\modulefinder.py", line 378, in _safe_import_hook
   self.import_hook(name, caller, level=level)
 File "C:\Python38.win32\lib\modulefinder.py", line 177, in import_hook
   q, tail = self.find_head_package(parent, name)
 File "C:\Python38.win32\lib\modulefinder.py", line 233, in find_head_package
   q = self.import_module(head, qname, parent)
 File "C:\Python38.win32\lib\modulefinder.py", line 326, in import_module
   m = self.load_module(fqname, fp, pathname, stuff)
 File "C:\Python38.win32\lib\modulefinder.py", line 343, in load_module
   co = compile(fp.read()+'\n', pathname, 'exec')
 File "C:\Python38.win32\lib\encodings\cp1252.py", line 23, in decode
   return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 308: character maps to <undefined>

Barry

(+Windows tag)

These should use io.open_code() these days anyway, which always opens in 'rb'.

Could you make that change?

io.open_code() used in the patch.

Thanks, Barry!

I tweaked your NEWS entry a little, so once CI completes I'll merge and backport.

New changeset d42e582 by Barry in branch 'master':
bpo-40260: Update modulefinder to use io.open_code() and respect coding comments (GH-19488)
d42e582

New changeset 59047fa by Miss Islington (bot) in branch '3.8':
bpo-40260: Update modulefinder to use io.open_code() and respect coding comments (GH-19488)
59047fa

This patch has broken debian's builds due to use of modulefinder -- notably the type of file_info changed as a side-effect of this patch and is now causing:

../debian/pymindeps.py:29: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Traceback (most recent call last):
  File "../debian/pymindeps.py", line 185, in <module>
    main(sys.argv[1:])
  File "../debian/pymindeps.py", line 178, in main
    mf.run_script(arg)
  File "/tmp/code/Lib/modulefinder.py", line 163, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "../debian/pymindeps.py", line 91, in load_module
    suffix, mode, type = file_info
ValueError: not enough values to unpack (expected 3, got 2)

This is breaking python3.8 and python3.9 (should I open a separate issue or is it ok to continue discussion here?)

Go ahead and fix it against this one.

Would be ideal to add a test that hits this case as well.

I need to see the code of pymindeps to understand what you are doing and how to fix this. Can you post a URL to the source please?

Are you aware that load_module() changed in other ways that are required to fix the bug?

You may have to change yout pymindeps code anyway even with the mode restored to the tuple.

Steve: Do I need a new PR or just commit a fix to the same branch?

Regarding test case. I will need to know what pymindeps is doing to be able to design a suitable test case.

I'm admittedly a little unfamiliar with what it does as well -- I'm mostly repackaging debian sources for deadsnakes. If I'm correct in my assumption, it is attempting to find a minimal set of modules to package into python3-minimal and ensure that that set is correct

here's my current version of the source: https://github.com/deadsnakes/python3.9-nightly/blob/3eb45671e2f1910a6f117580f1733e431cce8a6e/debian/pymindeps.py

(additionally, I'm not sure this should be backported to python3.8, especially with changed behaviour)

The 3.8 behaviour is clearly broken, so backporting is fine. Also, io.open_code() is a security feature.

The core of the issue is that apparently load_module() is a public API, and the file_info parameter used to be a 3-tuple and is now a 2-tuple.

AFAICT, the fix should restore the 3-tuple everywhere and just ignore the mode parameter. The test should pass in a 3-tuple and make sure that it doesn't crash like this.

Sorry, misready. They're monkeypatching the API. It isn't documented, but it's also not clearly internal.

We don't have a strong need to change the meaning of that argument though, so best to leave it as it was and not break anyone.

I have the fix coded and tested.

I run out of git knowledge to update the PR branch so that I can push the fix.

I'll work on it more later in the day.

I have pushed the fix onto #19595
with an API test case and the changes to keep the debain subclassing
working.

I'm new the the work flow. Let me know if I need to change anything.

Thanks! I deleted the NEWS file (we've already got one for this issue in this release) and skipped the check instead. Once CI passes, I'll merge it.

New changeset 9b0b5d2 by Barry in branch 'master':
bpo-40260: Revert breaking changes made in modulefinder (GH-19595)
9b0b5d2

New changeset 81de3c2 by Miss Islington (bot) in branch '3.8':
bpo-40260: Revert breaking changes made in modulefinder (GH-19595)
81de3c2

This is still failing, but now in a different way:

Traceback (most recent call last):
  File "../debian/pymindeps.py", line 185, in <module>
    main(sys.argv[1:])
  File "../debian/pymindeps.py", line 178, in main
    mf.run_script(arg)
  File "/tmp/code/Lib/modulefinder.py", line 163, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "../debian/pymindeps.py", line 92, in load_module
    m = modulefinder.ModuleFinder.load_module(self, fqname,
  File "/tmp/code/Lib/modulefinder.py", line 359, in load_module
    self.scan_code(co, m)
  File "/tmp/code/Lib/modulefinder.py", line 432, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/tmp/code/Lib/modulefinder.py", line 377, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "../debian/pymindeps.py", line 42, in import_hook
    return modulefinder.ModuleFinder.import_hook(self, name, caller,
  File "/tmp/code/Lib/modulefinder.py", line 175, in import_hook
    q, tail = self.find_head_package(parent, name)
  File "/tmp/code/Lib/modulefinder.py", line 231, in find_head_package
    q = self.import_module(head, qname, parent)
  File "../debian/pymindeps.py", line 48, in import_module
    m = modulefinder.ModuleFinder.import_module(self,
  File "/tmp/code/Lib/modulefinder.py", line 325, in import_module
    m = self.load_module(fqname, fp, pathname, stuff)
  File "../debian/pymindeps.py", line 92, in load_module
    m = modulefinder.ModuleFinder.load_module(self, fqname,
  File "/tmp/code/Lib/modulefinder.py", line 342, in load_module
    co = compile(fp.read()+b'\n', pathname, 'exec')
TypeError: can only concatenate str (not "bytes") to str

The failure above is fp is now a text file but before it was a binary file. this causes fp.read() + b'\n' to fail

I can send a patch -- I don't think the b'\n' is necessary any more (as far as I can tell it used to be there for an old version of python where compile(...) errored if the file did not end in a newline, that's now handled in the parser internally though)

actually I said that backwards in the previous message -- it was a text file and now it's a binary file

Anthony,

Now that everything is opened using open_code that returns bytes its
not clear to me why this breaks for you.

Further the data must be bytes for the codings to be figured out.

Removing the b'\n' may be reasonable, but not for the reason given.
I kept it as the original code did this and assumed it was needed.
Maybe for Windows users that forget to put a trailing NL?

How can I reproduce your user case?

debian has its own implementation of find_modules which still returns a text file for PY_SOURCE

changing the type of that io object is the "breaking change" and maybe shouldn't be backported to python3.8 in a patch release

If we want to be strictly correct here, the fix should check the "mode" variable to determine the type. And given the fragility of the code being used here, there's no doubt some reliance on a trailing newline somewhere too.

I haven't bumped the categorisation, but at some level this is a security fix. All executable code is expected to be loaded through io.open_code(), and the fact that this didn't was a breach of that guarantee. There's a limit to how much we will accommodate people using it outside of what's documented.

Anthony, since you can apparently validate this, could you run through all the fixes necessary to keep that working? I want to stop this game of chasing, either by fixing everything in one go or by apologising for the breakage (the io.open_code() fix is staying).

The trailing newline was added in 1.5.1 78fc363 -- back then compile(...) could not take strings which did not end in newlines

now it can, so it is safe to remove -- I did the same triage on another PR here: #17817

Okay, no problems then.

We'll have to close that other PR now. Too bad nobody noticed it (or
knew where to send it - I doubt modulefinder has an obvious owner).

New changeset 39652cd by Anthony Sottile in branch 'master':
bpo-40260: Remove unnecessary newline in compile() call (GH-19641)
39652cd

New changeset fc45cb4 by Miss Islington (bot) in branch '3.8':
bpo-40260: Remove unnecessary newline in compile() call (GH-19641)
fc45cb4