classification
Title: Intra-package References Documentation Incomplete
Type: behavior Stage:
Components: Documentation Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: ADataGman, brett.cannon, docs@python, eric.snow, ncoghlan, terry.reedy
Priority: normal Keywords:

Created on 2019-02-07 04:38 by ADataGman, last changed 2019-02-14 00:38 by ADataGman.

Files
File name Uploaded Description Edit
sound.zip ADataGman, 2019-02-07 04:38 zip of code sample copied from documentation
Messages (7)
msg335002 - (view) Author: (ADataGman) Date: 2019-02-07 04:38
Attempting to follow https://docs.python.org/3.6/tutorial/modules.html#intra-package-references I was unable to recreate the intra-package reference as described. 

"For example, if the module sound.filters.vocoder needs to use the echo module in the sound.effects package, it can use from sound.effects import echo."

Creating the file structure described in https://docs.python.org/3.6/tutorial/modules.html#packages, with empty __init__.py files at all levels, or with __all__ defined as containing relevant file names, results in "No module named 'sound'". If I try to run this using "from ..effects import echo" then it results in "attempted relative import beyond top-level package".

At least one other user has run into this issue with this stack overflow post: https://stackoverflow.com/questions/53109627/python-intra-package-reference-doesnt-work-at-all
msg335104 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-02-08 19:14
3.6 only gets security patches.

I cannot comment on relative imports as I never use them, but the absolute imports should work and do for me.  I just added to idlelib.idle_test a file a.py containing 'from idlelib.idle_test import htest' and it works find.  The idlelib and idle_test __init__.py files are not empty, but that should not make a difference.  In fact, many of the test_xyz modules already import from the non-test utility modules in the same directory.

The error messsage suggests that you did not put the top level package directory, 'sound', in a directory in sys.path, as you must.  What directory contains your sound package and what is your sys.path?  I suspect that this should be closed as 'not a bug'.
msg335105 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-02-08 19:31
When reporting problems, copy and paste into the report the complete exception traceback and error message.  For import problems, also copy and paste sys.path.  If you disagree that this should be closed, please do so.
msg335126 - (view) Author: (ADataGman) Date: 2019-02-09 06:16
Here's the full error log I receive when using the code I submitted with
both ..effects and sound.effects. It also includes a print out of the sys
path as run from vocoder.

I tried running this in 3.7.2 and still get the same errors. It seems like
it's already on the sys path? I'm not saying there is a bug in python
itself, it's just not at all clear from the documentation that what is
described as being a way to access sibling packages actually works.

Given the attached code sample, as a newb I'd greatly appreciate an example
of how to take that sample and turn it into functioning code with whatever
modifiers is necessary. Ultimately whatever changes are made I'd like to
see reflected in the documentation for the intra-reference section.

PS C:\sound> &
C:/Users/user/AppData/Local/Programs/Python/Python37/python.exe
c:/sound/filters/vocoder.py
Traceback (most recent call last):
  File "c:/sound/filters/vocoder.py", line 7, in <module>
    vocoder.pp()
  File "c:/sound/filters/vocoder.py", line 4, in pp
    from ..effects import echo
ValueError: attempted relative import beyond top-level package
PS C:\sound> &
C:/Users/user/AppData/Local/Programs/Python/Python37/python.exe
c:/sound/filters/vocoder.py
Traceback (most recent call last):
  File "c:/sound/filters/vocoder.py", line 7, in <module>
    vocoder.pp()
  File "c:/sound/filters/vocoder.py", line 4, in pp
    from sound.effects import echo
ModuleNotFoundError: No module named 'sound'
PS C:\sound> &
C:/Users/user/AppData/Local/Programs/Python/Python37/python.exe
c:/sound/filters/vocoder.py
['c:\\sound\\filters',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\python37.zip',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\DLLs',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\lib',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37',
'C:\\Users\\user\\AppData\\Roaming\\Python\\Python37\\site-packages',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages']

On Fri, Feb 8, 2019 at 2:31 PM Terry J. Reedy <report@bugs.python.org>
wrote:

>
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
>
> When reporting problems, copy and paste into the report the complete
> exception traceback and error message.  For import problems, also copy and
> paste sys.path.  If you disagree that this should be closed, please do so.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35927>
> _______________________________________
>
msg335303 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-02-12 13:17
As discussed in http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#executing-the-main-module-twice and https://www.python.org/dev/peps/pep-0395/#why-are-my-imports-broken the cryptic error message here is due to the relative import being resolved based on the module name "__main__", since the submodule was executed directly, rather than the intended "sound.filter.vocoder" name that it would have when imported or executed via the "-m" switch.

It's spectacularly unobvious what's going on if you're not deeply familiar with the intricacies of main module execution, but I haven't had any great ideas for practical ways to improve the error message.

I'm not sure we currently have access to the required info at the point this error is raised, but perhaps it would be enough to add the name of the module doing the import to the error message as a reminder that directly executing scripts inside packages does weird things to import resolution?

Something like:

  "Attempted relative import beyond top-level package '__main__'"
msg335385 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-02-13 04:35
AData> You log and output show that you problem is as described: package 'sound' is in directory "C:/" and "C:/" is not is sys.path.  Possible remedies:
1. Add "C:/" to sys.path at the top of each module that imports sound.  This is a nuisance.
2. Move the sound package to 'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages'.  This can also a nuisance.
3. Add a file named 'C.pth' and containing "C:/" to the site-packages.  ('pth' abbreviates 'path'.)  This makes C:/ an extension of site-packages.  But all the non-python stuff in C: likely makes this a bad idea.
4 (3 improved). Add a directory such as "C:/pyprojects", make the contents of the .pth file "C:/pyprojects", and move sound to pyprojects.  
This is what I have done.  When I install a new version of Python, I copy the .pth file to the new site-packeges directory.  Works great.
msg335486 - (view) Author: (ADataGman) Date: 2019-02-14 00:38
Terry, thank you for the detailed list of alternatives on how to approach
this, that's helpful in how to tackle this issue in my current development.

Nick, for the relative imports of relative imports I think the error
message is fine. It makes sense. It doesn't necessarily give you a lot of
hints on where to go next unfortunately, but it does explain why it doesn't
work.

For the absolute imports, a hint like, the module must be discoverable in
sys.path or the like if you wanted to make this clearer.

My biggest concern was just that the way the documentation is written for
https://docs.python.org/3.6/tutorial/modules.html#intra-package-references
. It implies that the code will run without any hint of needing to be
either run from a higher level, or a work around used like Terry suggested.
Now that I've spent a few days researching and reading more on my own, and
with help from you and Terry, I understand what's happening and have
options. But for others just reading the docs and not already familiar,
it's not clear why the current example wouldn't work when run from the
sibling level.

I'd like to see the documentation for intra-package references updated to
clearly state that the code is being run from a level above. "For example,
if the module sound.filters.vocoder needs to use the echo module in the
sound.effects package, it can use from sound.effects import echo." only
works if you run it from the sound module. Which is fine! It's just not
clear to a newb that it's being executed from the sound package rather than
the filters package. Perhaps modify
https://docs.python.org/3.6/tutorial/modules.html#packages to include a
"sound.py" at the top level folder, and in the explanation for the
intra-package reference, just mention that the example quoted above runs
from that new file.

If there is a optimal way to modify the path for adding in a sibling level
package, please include that. Otherwise just clarify that the example
relies on the code being run from something from another level up.

On Tue, Feb 12, 2019 at 11:35 PM Terry J. Reedy <report@bugs.python.org>
wrote:

>
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
>
> AData> You log and output show that you problem is as described: package
> 'sound' is in directory "C:/" and "C:/" is not is sys.path.  Possible
> remedies:
> 1. Add "C:/" to sys.path at the top of each module that imports sound.
> This is a nuisance.
> 2. Move the sound package to
> 'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages'.
> This can also a nuisance.
> 3. Add a file named 'C.pth' and containing "C:/" to the site-packages.
> ('pth' abbreviates 'path'.)  This makes C:/ an extension of site-packages.
> But all the non-python stuff in C: likely makes this a bad idea.
> 4 (3 improved). Add a directory such as "C:/pyprojects", make the contents
> of the .pth file "C:/pyprojects", and move sound to pyprojects.
> This is what I have done.  When I install a new version of Python, I copy
> the .pth file to the new site-packeges directory.  Works great.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35927>
> _______________________________________
>
History
Date User Action Args
2019-02-14 00:38:19ADataGmansetmessages: + msg335486
2019-02-13 04:35:36terry.reedysetmessages: + msg335385
2019-02-12 13:17:57ncoghlansetmessages: + msg335303
2019-02-09 06:16:22ADataGmansetmessages: + msg335126
2019-02-08 19:31:01terry.reedysetmessages: + msg335105
2019-02-08 19:14:21terry.reedysetnosy: + terry.reedy, eric.snow, brett.cannon, ncoghlan

messages: + msg335104
versions: + Python 3.7, Python 3.8, - Python 3.6
2019-02-07 04:38:30ADataGmancreate