Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make IDLE recognize .pyi stub files as source for open, save, and edit #89610

Closed
AlexWaygood opened this issue Oct 12, 2021 · 37 comments
Closed
Assignees
Labels
3.11 only security fixes topic-IDLE type-feature A feature request or enhancement

Comments

@AlexWaygood
Copy link
Member

AlexWaygood commented Oct 12, 2021

BPO 45447
Nosy @terryjreedy, @ronaldoussoren, @zooba, @miss-islington, @E-Paine, @AlexWaygood
PRs
  • gh-89610: Add syntax highlighting for .pyi files in IDLE #28950
  • [3.10] bpo-45447: Add syntax highlighting for .pyi files in IDLE (GH-28950) #31303
  • bpo-45447: Add entry to What's new 3.10 #31304
  • bpo-45447: Add entry to What's new 3.9 #31305
  • [3.9] bpo-45447: Add syntax highlighting for .pyi files in IDLE (GH-28950) #31306
  • bpo-45447: Fix entry in What's New 3.11 #31307
  • [3.10] bpo-45447: Add entry to What's new 3.10 (GH-31304) #31308
  • [3.10] bpo-45447: Add entry to What's new 3.9 (GH-31305) #31309
  • [3.9] bpo-45447: Add entry to What's new 3.9 (GH-31305) #31310
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/terryjreedy'
    closed_at = None
    created_at = <Date 2021-10-12.15:24:36.540>
    labels = ['expert-IDLE', 'type-feature', '3.11']
    title = 'Make IDLE recognize .pyi stub files as source for open, save, and edit'
    updated_at = <Date 2022-02-14.03:35:10.254>
    user = 'https://github.com/AlexWaygood'

    bugs.python.org fields:

    activity = <Date 2022-02-14.03:35:10.254>
    actor = 'terry.reedy'
    assignee = 'terry.reedy'
    closed = False
    closed_date = None
    closer = None
    components = ['IDLE']
    creation = <Date 2021-10-12.15:24:36.540>
    creator = 'AlexWaygood'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 45447
    keywords = ['patch']
    message_count = 28.0
    messages = ['403738', '403741', '403755', '403772', '403884', '403889', '403915', '403998', '404021', '404023', '404030', '404100', '404102', '404110', '410637', '410669', '412561', '412617', '413147', '413149', '413150', '413156', '413157', '413158', '413168', '413169', '413170', '413211']
    nosy_count = 6.0
    nosy_names = ['terry.reedy', 'ronaldoussoren', 'steve.dower', 'miss-islington', 'epaine', 'AlexWaygood']
    pr_nums = ['28950', '31303', '31304', '31305', '31306', '31307', '31308', '31309', '31310']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue45447'
    versions = ['Python 3.11']

    Linked PRs

    @AlexWaygood
    Copy link
    Member Author

    IDLE currently does not do any syntax highlighting if you use it to open a .pyi stub file, even though everything in a .pyi file is a valid Python expression. It would be great if syntax highlighting for .pyi files could be supported.

    @AlexWaygood AlexWaygood added the 3.11 only security fixes label Oct 12, 2021
    @AlexWaygood AlexWaygood added topic-IDLE type-feature A feature request or enhancement labels Oct 12, 2021
    @AlexWaygood AlexWaygood added topic-IDLE type-feature A feature request or enhancement labels Oct 12, 2021
    @AlexWaygood AlexWaygood changed the title Support syntax highlighting for .pyi stub files IDLE: Support syntax highlighting for .pyi stub files Oct 12, 2021
    @AlexWaygood AlexWaygood changed the title Support syntax highlighting for .pyi stub files IDLE: Support syntax highlighting for .pyi stub files Oct 12, 2021
    @terryjreedy
    Copy link
    Member

    It should be fairly easy to recognize x.pyi as a python file. Just have to find the place. Something like
    def f(i:int) -> int: ...
    would have 'def' and 'int' highlighted.

    @AlexWaygood
    Copy link
    Member Author

    It looks like support for .py files is hardcoded into IDLE's source code in the following places:

    • browser.ModuleBrowserTreeItem.OnDoubleClick
    • browser.ModuleBrowserTreeItem.IsExpandable
    • browser.ModuleBrowserTreeItem.listchildren
    • editor.ispythonsource << This is the one that controls syntax highlighting
    • iomenu.IOBinding.filetypes

    If I were to file a PR adding support for .pyi files, would you prefer that I hardcoded ".pyi" in, or would it be better to add a SUPPORTED_FILES constant somewhere?

    @terryjreedy
    Copy link
    Member

    Thank you for doing the research.

    I have been thinking about adding a file containing idlelib 'leaf' objects, those with no idlelib dependencies, which are needed in more than one file. would be to reduce the complexity of the idlelib dependency graph, which has enough cyclic dependencies to make startup tricky. Time to do it. idlelib/common.py?, leaves.py?, or ??? Draft:
    ---
    """Idlelib objects with no external idlelib dependencies and which are needed in more than one idlelib module. They are included here because a) they don't particularly belong elsewhere or b) because inclusion here simplifies the idlelib dependency graph.

    TODO: Python versions (editor and help_about), tk version and patchlevel (pyshell, help_about, maxos?, editor?), std streams (pyshell, run), warning stuff (ditto).
    """

    # python_extensions is used in editor, browser, and iomenu.
    # .pyw is for Windows; .pyi is for stub files.
    python_extensions = ('.py', '.pyw', '.pyi')  
    extension_string = "*" + " *".join(python_extensions)

    editor.EditorWindow.ispythonsource could be moved into the new file if a 'firstline=None' argument were added.

    OnDoubleClick should use IsExpandable. IsExpandable should use the logic in ispythonsource so Linux no-extension python source can be browsed as well as edited.

    After manual testing, add a new test_common.py file in idlelib.idle_test. After imports, it could start with one test:

    class ExtensionTest(TestCase):
        def test_stub(self):
            self.assertIn('.pyi', common.python_extensions)
            self.assertIn('.pyi', common.extension_string)

    For IDLE, I can and will backport.

    @terryjreedy
    Copy link
    Member

    I am going to call the file util.py and will work on a PR.

    @AlexWaygood
    Copy link
    Member Author

    Oh — I've already nearly finished a PR. Do you mind if I submit mine? Happy to make any changes you like, obviously.

    @AlexWaygood
    Copy link
    Member Author

    Have posted my PR. I've tested it manually and run the IDLE test suite on it with the -u all setting; it seems fine. The new tests I've written are a bit thin on the ground right now, though. I may need some guidance/help if more extensive tests are needed -- I'm a little new to unittest.

    One other thing I'm not sure on: the docstring at the top of browser.py includes the line "- finish removing limitation to x.py files (ModuleBrowserTreeItem)" in the TODO items. Can this be removed now?

    @E-Paine
    Copy link
    Mannequin

    E-Paine mannequin commented Oct 15, 2021

    Would it make sense, since this issue pretty much makes IDLE officially support .pyi files, to add the "Edit with IDLE" menu when right-clicking on a .pyi file in Windows explorer?

    @terryjreedy
    Copy link
    Member

    Yes. After this issue is done, we should open a new issue and nosy Steve Dower, who is in charge of that for Windows, and Ned Deily, in case anything should be done on mac.

    @ronaldoussoren
    Copy link
    Contributor

    The most straightforward change for macOS is to add “pyi” to the list of extensions in “Mac/IDLE/IDLE.app/Contents/Info.plist”. That way IDLE will be seen as a valid editor for .pyi files.

    @terryjreedy
    Copy link
    Member

    Steve, how do we make it so that "Edit with IDLE" appears when right-clicking on a .pyi file in Windows explorer? As easy as on mac (see Ronald's answer above)?

    @zooba
    Copy link
    Member

    zooba commented Oct 16, 2021

    Bit more complicated. It'll need changes to Tools/msi/launcher/launcher_reg.wxs, Tools/msi/tcltk/tcltk_reg.wxs, and PC/layout/support/appxmanifest.py, but I don't think it's that obvious where (we don't want these files to open with py.exe by default, so they'll need a whole new definition).

    @AlexWaygood
    Copy link
    Member Author

    Shall I add the change to the mac file to my PR, or would it be better to leave it for a separate PR? (I'm not confident enough to make the changes to the Windows files.)

    @terryjreedy
    Copy link
    Member

    We can add the mac change once it is manually tested on some system. I may try on my macbook. Windows will be another PR or even issue.

    @AlexWaygood
    Copy link
    Member Author

    @terry, have you had a chance to glance at my PR at all? I'd really appreciate your review! :)

    @terryjreedy
    Copy link
    Member

    A minimal version of util.py just for this issue is fine. What is blocking the issue deciding exactly what the minimum should be. Are both the extension list and function needed?

    Alex says somewhere in PR comments that ispythonsource passes directories. That seems wrong. Perhaps moving and revising it and making no-extension files browsable and removing the x.py(?) limitation should be a separate issue. IDLE users on *nix don't mind using extensions. I have not seen a complaint, though perhaps people who do mind silently use something else. I need to check again how the editor function is used and decide if we can leave it alone for now. So don't remove anything just now.

    I need to experiment with Ron's suggestion on my Mac.

    The Windows context menu is definitely a separate issue.

    Please add idle_test/example.pyi with the line in msg403741 for manual testing now and possibly future automatic testing.

    @terryjreedy
    Copy link
    Member

    For the open dialog, we only need the expanded extension list. People with no-extension python code file must select 'all files'. Once the file is open, we need the issource function to decide whether to turn on the colorizer. I intend to review the patch itself tomorrow.

    @terryjreedy
    Copy link
    Member

    I broaden the issue title scope and will correspondingly restrict the PR scope. 'Recognition' applies to open and save dialogs, syntax marking, and module browsing. (The latter needs fixing for .pyw also.)

    Improving the handling of no extension source with initial #! line will be a different issue.

    @terryjreedy terryjreedy changed the title IDLE: Support syntax highlighting for .pyi stub files Make IDLE recognize .pyi stub files (and .pyw) as python source Feb 6, 2022
    @terryjreedy terryjreedy changed the title IDLE: Support syntax highlighting for .pyi stub files Make IDLE recognize .pyi stub files (and .pyw) as python source Feb 6, 2022
    @terryjreedy
    Copy link
    Member

    New changeset 50cf499 by Alex Waygood in branch 'main':
    bpo-45447: Add syntax highlighting for .pyi files in IDLE (GH-28950)
    50cf499

    @erlend-aasland
    Copy link
    Contributor

    Are there further work to be done here, or can we close this issue?

    @terryjreedy
    Copy link
    Member

    #89610 (comment) indicates other places that need revision.

    @erlend-aasland
    Copy link
    Contributor

    It is not obvious to me what needs revision. Could you elaborate?

    @terryjreedy
    Copy link
    Member

    All the other places indicated that require python file be .py, thereby excluding .pyi.

    @ronaldoussoren
    Copy link
    Contributor

    See also my comment about adding pyi to the list of CFBundleTypeExtensions in IDLE's Info.plist. That will allow opening stub files by double clicking on them in the Finder, and is not necessary for the File/open menu.

    Otherwise syntax coloring for stub files appears to work fine in the current Python 3.11 beta.

    @erlend-aasland
    Copy link
    Contributor

    erlend-aasland commented Jul 28, 2022

    See also my comment about adding pyi to the list of CFBundleTypeExtensions in IDLE's Info.plist.

    👉🏻 #95393

    @erlend-aasland
    Copy link
    Contributor

    It looks like support for .py files is hardcoded into IDLE's source code in the following places:

    • browser.ModuleBrowserTreeItem.OnDoubleClick
    • browser.ModuleBrowserTreeItem.IsExpandable
    • browser.ModuleBrowserTreeItem.listchildren

    👉🏻 #95397

    • iomenu.IOBinding.filetypes

    Seems irrelevant for this issue to me; I might be wrong, I only took a quick glance at the code.

    @terryjreedy terryjreedy reopened this Jul 28, 2022
    @terryjreedy
    Copy link
    Member

    I listed iomenu filetypes before we added the util.py_extensions tuple, which is used in several places. Before, the same information was duplicated in multiple places. That item is now obsolete.

    miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 28, 2022
    …ythonGH-95393)
    
    This allows opening stub files by double clicking on them in the Finder.
    
    Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
    (cherry picked from commit 06fc249)
    
    Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
    miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 28, 2022
    …ythonGH-95393)
    
    This allows opening stub files by double clicking on them in the Finder.
    
    Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
    (cherry picked from commit 06fc249)
    
    Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
    terryjreedy added a commit that referenced this issue Jul 28, 2022
    This allows opening stub files by double clicking on them in the Finder.
    
    Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
    miss-islington added a commit that referenced this issue Jul 28, 2022
    )
    
    This allows opening stub files by double clicking on them in the Finder.
    
    Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
    (cherry picked from commit 06fc249)
    
    Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
    miss-islington added a commit that referenced this issue Jul 28, 2022
    )
    
    This allows opening stub files by double clicking on them in the Finder.
    
    Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
    (cherry picked from commit 06fc249)
    
    Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
    @terryjreedy
    Copy link
    Member

    I open task issue #95410 with two of the items completed by this issue. I opened a browser issue #95411 and moved the browser PR there.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 only security fixes topic-IDLE type-feature A feature request or enhancement
    Projects
    Status: Done
    Development

    No branches or pull requests

    6 participants