Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove asyncore, asynchat and smtpd modules #72719

Closed
Mariatta opened this issue Oct 25, 2016 · 49 comments
Closed

Remove asyncore, asynchat and smtpd modules #72719

Mariatta opened this issue Oct 25, 2016 · 49 comments
Labels
3.12 bugs and security fixes tests Tests in the Lib/test dir

Comments

@Mariatta
Copy link
Member

Mariatta commented Oct 25, 2016

BPO 28533
Nosy @warsaw, @orsenthil, @vstinner, @asvetlov, @1st1, @grzgrzgrz3, @Mariatta, @nanjekyejoannah, @aeros, @jacksonriley, @OhSoDank, @iritkatriel
PRs
  • bpo-28533: Remove asyncore, asynchat, smtpd modules #29521
  • Revert "bpo-28533: Remove asyncore, asynchat, smtpd modules (GH-29521)" #29951
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2016-10-25.18:32:04.420>
    labels = ['3.11', 'tests', 'expert-asyncio']
    title = 'Remove asyncore, asynchat and smtpd modules'
    updated_at = <Date 2022-02-24.13:39:30.092>
    user = 'https://github.com/Mariatta'

    bugs.python.org fields:

    activity = <Date 2022-02-24.13:39:30.092>
    actor = 'vstinner'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Tests', 'asyncio']
    creation = <Date 2016-10-25.18:32:04.420>
    creator = 'Mariatta'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 28533
    keywords = ['patch']
    message_count = 37.0
    messages = ['279439', '291907', '291908', '294710', '294712', '294771', '294773', '356355', '356364', '356369', '357088', '379342', '396295', '396297', '396319', '396323', '404575', '406159', '406160', '406161', '406162', '406170', '406173', '406208', '406210', '406213', '406214', '406215', '406372', '406377', '406434', '407559', '407560', '407897', '407900', '407915', '413909']
    nosy_count = 12.0
    nosy_names = ['barry', 'orsenthil', 'vstinner', 'asvetlov', 'yselivanov', 'grzgrzgrz3', 'Mariatta', 'nanjekyejoannah', 'aeros', 'jacksonriley', 'dankreso', 'iritkatriel']
    pr_nums = ['29521', '29951']
    priority = 'normal'
    resolution = None
    stage = 'resolved'
    status = 'open'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue28533'
    versions = ['Python 3.11']

    @Mariatta
    Copy link
    Member Author

    Deprecation warning was added to asyncore in https://bugs.python.org/issue25002

    asyncore is still used in several tests and should be replaced.

    @vstinner
    Copy link
    Member

    asyncore is still used in several tests and should be replaced.

    Here is the list of Python files using asyncore:

    haypo@selma$ grep -l asyncore $(find -name "*.py")
    ./Lib/asyncore.py
    ./Lib/asynchat.py
    ./Lib/test/test_smtpd.py
    ./Lib/test/test_asyncore.py
    ./Lib/test/test_smtplib.py
    ./Lib/test/test_os.py
    ./Lib/test/test_ssl.py
    ./Lib/test/test_logging.py
    ./Lib/test/libregrtest/save_env.py
    ./Lib/test/test_asynchat.py
    ./Lib/test/test_pyclbr.py
    ./Lib/test/test_support.py
    ./Lib/test/test_poplib.py
    ./Lib/test/test_ftplib.py
    ./Lib/smtpd.py

    smtpd has been rewritten with asyncio:
    http://aiosmtpd.readthedocs.io/

    The documentation of the smtpd module already points to aiosmtpd.

    libregrtest/save_env.py only checks that asyncore.socket_map is not modified, we can keep it until asyncore is really removed.

    test_support.py only tests that it's possible to have a "clean" import of asyncore, it can be easily be replaced later.

    The list can reduced to:

    ./Lib/test/test_smtplib.py
    ./Lib/test/test_os.py
    ./Lib/test/test_ssl.py
    ./Lib/test/test_logging.py
    ./Lib/test/test_pyclbr.py
    ./Lib/test/test_poplib.py
    ./Lib/test/test_ftplib.py

    @vstinner
    Copy link
    Member

    asyncore was modified to emit a DeprecationWarning:
    http://bugs.python.org/issue25002#msg279417

    But then a lot of code starts with to fail -Werror.

    Copy of Martin Panter's msg279469:

    I normally run the tests with -Werror, and the failures I get are:

    • test_ssl, test_smtplib, test_poplib, test_logging, test_ftplib, test_support: tests use asyncore
    • test_os: test uses asynchat (When you import asynchat, the first error complains about importing asyncore, there are actually two warnings)
    • test_all: This seems to ignore DeprecationWarning; perhaps an exception for PendingDeprecationWarning should also be added?
    • test_asyncore and test_asynchat: Obviously these have to still test the modules, so they should anticipate the warnings, perhaps using Serhiy’s code
    • test_smtpd: smtpd module itself uses asyncore; see bpo-25008

    @grzgrzgrz3
    Copy link
    Mannequin

    grzgrzgrz3 mannequin commented May 29, 2017

    I would like to work on this issue.

    I think it's a good idea to split this task into few parts/PR.

    Let me start from ./Lib/test/test_poplib.py.

    What about rewriting pop3 server stub using asyncio, i think requests could be handled synchronously, there will be no benefits from asynchronous.

    Please let me know what you think.

    @vstinner
    Copy link
    Member

    I think it's a good idea to split this task into few parts/PR. Let me start from ./Lib/test/test_poplib.py.

    Once you have a patch, open a new issue and mention it here.

    @Mariatta Mariatta added the 3.7 (EOL) end of life label May 29, 2017
    @grzgrzgrz3
    Copy link
    Mannequin

    grzgrzgrz3 mannequin commented May 30, 2017

    ./Lib/test/test_poplib.py
    sub-issue bpo-28533

    @grzgrzgrz3
    Copy link
    Mannequin

    grzgrzgrz3 mannequin commented May 30, 2017

    ./Lib/test/test_poplib.py
    sub-issue bpo-30514

    Fixed issue number from previous comment

    @OhSoDank
    Copy link
    Mannequin

    OhSoDank mannequin commented Nov 11, 2019

    Hi, is this still open?

    I'm happy to work on replacing asyncore usage in one of the other test files.

    @nanjekyejoannah
    Copy link
    Member

    @dankreso, the issue is still open and no one has claimed it yet. So feel free to open a pull request.

    @aeros
    Copy link
    Contributor

    aeros commented Nov 11, 2019

    I'm happy to work on replacing asyncore usage in one of the other test files.

    Sounds good, just let us know which one(s) you're working on. (:

    @jacksonriley
    Copy link
    Mannequin

    jacksonriley mannequin commented Nov 20, 2019

    Lib/test/test_pyclbr.py - subissue bpo-38866
    A trivial one!

    @aeros
    Copy link
    Contributor

    aeros commented Oct 22, 2020

    Since this issue is now a significant blocker for PEP-594 (remove stdlib dead batteries, which includes asyncore and asynchat), I'm going to prioritize working on this and assign it to myself.

    @aeros aeros self-assigned this Oct 22, 2020
    @iritkatriel
    Copy link
    Member

    If the tests are the only thing blocking removal, does it make sense to move asyncore and asynchat to test.support and get on with removing them from the stdlib?

    @vstinner
    Copy link
    Member

    Irit:

    If the tests are the only thing blocking removal, does it make sense to move asyncore and asynchat to test.support and get on with removing them from the stdlib?

    Yes, it makes sense and it can be done right now, since asynchat/asyncore is deprecated since Python 3.6.

    @iritkatriel
    Copy link
    Member

    There is actually one usage outside of the tests: Lib/smtpd.py

    There is a note in the doc saying " The aiosmtpd package is a recommended replacement for this module. It is based on asyncio and provides a more straightforward API. smtpd should be considered deprecated."

    https://docs.python.org/3/library/smtpd.html

    @vstinner
    Copy link
    Member

    So the first step is to decide if it's ok to remove smtpd right now.

    @iritkatriel
    Copy link
    Member

    asyncore, asynchat and smtpd are all deprecated now, for removal in 3.12.

    While the tests are not blocking their removal (we can move them to test.support) it would still be better to rewrite these tests.

    @iritkatriel iritkatriel added 3.11 bug and security fixes tests Tests in the Lib/test dir and removed 3.7 (EOL) end of life labels Oct 21, 2021
    @iritkatriel iritkatriel changed the title Replace asyncore Replace asyncore/asynchat/smptd in tests Oct 21, 2021
    @vstinner
    Copy link
    Member

    I forgot about this issue and I created a duplicate: bpo-45785. Copy of my messages.

    I propose to remove asyncore, asynchat and smtpd modules from the Python standard library to remove the Python maintenance burden. These modules are deprecated since Python 3.6.

    --

    The asyncore module is a very old module of the Python stdlib for asynchronous programming, usually to handle network sockets concurrently. It's a common event loop, but its design has many flaws.

    The asyncio module was added to Python 3.4 with a well designed architecture. Twisted developers who have like 10 to 20 years of experience in asynchronous programming helped to design the asyncio API.

    By design, asyncio doesn't have flaws which would be really hard to fix in asyncore and asynchat.

    It was decided to start deprecating the asyncore and asynchat module in Python 3.6 released in 2016, 5 years ago. Open issues in asyncore and asynchat have been closed as "wont fix" because the module is deprecated. The two modules are basically no longer maintained.

    Documentation:

    I propose to remove the two modules right now in the Python stdlib. They were removed for 4 Python releases (3.6-3.10), it's long enough to respect the PEP-387. The PEP requires 2 Python releases at least before considering removing code.

    Since there are still 7 tests of the Python test suite still uses asyncore and asynchat, I propose to move these modules in Lib/test/ directory and make them private:

    Projects using asyncore and asynchat should use asyncio. If someone really wants to continue using asyncore and asynchat, it's trivial to copy asyncore.py and asynchat.py in their project, and maintain these files on their side.

    --

    About the smtpd module, it is also deprecated since Python 3.6 (deprecated for 4 Python releases). It's used by a single test (test_logging). I also propose to remove it from the stdlib.

    I proposed to rename Lib/smtpd.py to Lib/test/support/_smtpd.py.

    Projects using smtpd can consider using aiosmtpd which is based on asyncio:
    https://aiosmtpd.readthedocs.io/

    Or again, they can copy Python 3.10 smtpd.py in their project and maintain this file on their side.

    --

    Ah, a DeprecationWarning warning is only emitted at runtime since Python 3.10. What's New in Python 3.10:

    "asynchat, asyncore, smtpd: These modules have been marked as deprecated in their module documentation since Python 3.6. An import-time DeprecationWarning has now been added to all three of these modules."

    https://docs.python.org/dev/whatsnew/3.10.html#asynchat-asyncore-smtpd

    @vstinner
    Copy link
    Member

    I wrote PR 29521 to remove the asyncore, asynchat and smtpd modules.

    libregrtest/save_env.py only checks that asyncore.socket_map is not modified, we can keep it until asyncore is really removed.

    I chose to remove it instead of my PR.

    test_support.py only tests that it's possible to have a "clean" import of asyncore, it can be easily be replaced later.

    This code has been removed in the meanwhile.

    But then a lot of code starts with to fail -Werror.

    While tests using asyncore, asynchat and smtpd have been modified to catch the DeprecationWarning warning, I chose to remove this warning in my PR, since I made the 3 modules private. I don't think that test.support private modules must emit deprecation warnings.

    So the first step is to decide if it's ok to remove smtpd right now.

    IMO it's ok to remove it since it is deprecated since Python 3.6.

    asyncore, asynchat and smtpd are all deprecated now, for removal in 3.12.

    Did you see an explicit mention of Python 3.12? I only saw mention of "Python 3.6" which is the version when the 3 modules were deprecated.

    @vstinner
    Copy link
    Member

    Since Christian Heimes decided to mark my issue bpo-45785 "Remove asyncore, asynchat and smtpd modules" as a duplicate of this issue, I changed this issue title to "Remove asyncore, asynchat and smtpd modules".

    @vstinner vstinner changed the title Replace asyncore/asynchat/smptd in tests Remove asyncore, asynchat and smtpd modules Nov 11, 2021
    @vstinner
    Copy link
    Member

    I announced my plan to remove the 3 modules on python-dev:
    https://mail.python.org/archives/list/python-dev@python.org/thread/LZOOLX5EKOITW55TW7JQYKLXJUPCAJB4/

    @vstinner
    Copy link
    Member

    Follow-up: PEP-594 schedules the removal of these 4 modules in Python 3.12, but the PEP remains a draft.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @hugovk
    Copy link
    Member

    hugovk commented Apr 11, 2022

    @vstinner
    Copy link
    Member

    vstinner commented May 9, 2022

    Reported upstream (it's a single file, used for tests): ansible/ansible#76452

    Oh, I reported the issue to the wrong GitHub project. I reported it again at: ansible-collections/community.general#4656

    @vstinner
    Copy link
    Member

    vstinner commented May 9, 2022

    mercurial-6.0

    I notified the project upstream: https://bz.mercurial-scm.org/show_bug.cgi?id=6700

    @AA-Turner AA-Turner added 3.12 bugs and security fixes and removed 3.11 bug and security fixes labels Jun 7, 2022
    @Karlson2k
    Copy link

    The suggested stmpd replacement supports Python versions up to 3.9. Python 3.10 and later is not supported.
    There is no single commit in the past 14 months in the repo.

    @warsaw
    Copy link
    Member

    warsaw commented Jun 13, 2022

    True, but it's still the most viable replacement for smtpd. It needs a full-time steward though.

    @iritkatriel
    Copy link
    Member

    Follow-up: PEP 594 has been accepted and asyncore, asynchat and smtpd are set for removal in 3.12.

    Merged documentation PRs:

    Pending removal PR:

    Newer removal PR that was merged...

    And reverted:

    We should be able to un-revert this now.

    @gvanrossum
    Copy link
    Member

    I propose to remove the expert-asyncio label from this issue. Any objections? This doesn't require an asyncio expert at all, it's just documentation and removals (or deprecations?).

    @iritkatriel
    Copy link
    Member

    I think that was referring to tests that need to be rewritten with asyncio. The alternative is to move these modules (or parts of them) to test.support so the tests continue to work.

    @gvanrossum
    Copy link
    Member

    I think that was referring to tests that need to be rewritten with asyncio. The alternative is to move these modules (or parts of them) to test.support so the tests continue to work.

    That still doesn't require pinging the asyncio experts, IMO (and my opinion counts since I'm the only active one :-). I'll remove the label.

    @vstinner
    Copy link
    Member

    The alternative is to move these modules (or parts of them) to test.support so the tests continue to work.

    IMO that's a convenient way to unblock the issue. Moving away from asyncore/asynchat would be nice but nobody managed to update tests in 10 years, so maybe it's ok to not changed them until it becomes a major (blocking) issue :-)

    sobolevn added a commit to sobolevn/cpython that referenced this issue Sep 5, 2022
    sobolevn added a commit to sobolevn/cpython that referenced this issue Sep 5, 2022
    sobolevn added a commit to sobolevn/cpython that referenced this issue Sep 5, 2022
    sobolevn added a commit to sobolevn/cpython that referenced this issue Sep 5, 2022
    @sobolevn
    Copy link
    Member

    I've moved these modules to test/support as @iritkatriel and @vstinner proposed.
    For now, they were coppied as-is (probably some parts are unused, but this is a subject to another piece of work).

    Tests do continue to pass! 🎉

    vstinner pushed a commit that referenced this issue Nov 8, 2022
    Remove modules asyncore and asynchat, which were deprecated by PEP 594.
    
    Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
    @vstinner
    Copy link
    Member

    vstinner commented Nov 8, 2022

    asyncore, asynchat, smtpd are removed again in the main branch (future Python 3.12). Let's see how it goes this time.

    Summary of this issue.

    The asyncore/asynchat deprecation has a long history starting in 2015: https://bugs.python.org/issue25002

    In 2020, Kyle Stanley @aeros worked on the removal in a branch: https://github.com/aeros/cpython/tree/remove-asycore-asynchat-smtpd

    In 2021, I already removed the 3 modules (asyncore, asynchat, smtpd) in Python 3.11, but the Steering Council asked me to revert it since these modules only emitted a DeprecationWarning in 1 Python release (Python 3.10):

    After my revert, I notified Mercurial and Ansible that these 3 projects are deprecated since 2015.

    In 2022, these modules are removed again in Python 3.12:

    The removal is now covered by PEP 594 which has been approved the SC.

    You can now help the remaining projects still using the removed projects to be upgrade to asyncio and aiosmptd. People can also collaborate to maintain the 3 removed modules outside Python (on PyPI) if they want. Just copy code, tests and doc from Python 3.11.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.12 bugs and security fixes tests Tests in the Lib/test dir
    Projects
    None yet
    Development

    No branches or pull requests