classification
Title: Deprecate asyncore/asynchat
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: 25008 Superseder:
Assigned To: Nosy List: Mariatta, barry, brett.cannon, dstufft, eric.smith, giampaolo.rodola, gvanrossum, haypo, martin.panter, python-dev, r.david.murray, rhettinger, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2015-09-04 20:48 by gvanrossum, last changed 2016-11-17 19:48 by giampaolo.rodola.

Files
File name Uploaded Description Edit
issue25002.patch Mariatta, 2016-10-25 05:07 review
Messages (34)
msg249829 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 20:48
Now that we've got asyncio in two releases (3.4 and 3.5) we should start deprecating asyncore and asynchat. There isn't much use of these in the stdlib (smtpd.py uses them, and a bunch of tests) and we should probably rewrite those to use asyncio. Maybe smtpd.py can be deprecated itself.
msg249830 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2015-09-04 20:50
I'm all for deprecating asyncore/asynchat but should deprecating them wait until asyncio is no longer provisional?
msg249831 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 20:53
Yes, in 3.6 asyncio will no longer be provisional and we can start deprecating async{ore,hat}. Which is why I marked this bug with 3.6.
msg249833 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-09-04 20:55
The documentation of both modules begin with the following note:

"This module exists for backwards compatibility only. For new code we recommend using asyncio."

What do you mean by deprecating the module? Emit a PendingDeprecationWarning in Python 3.6 and emit a DeprecationWarning in Python 3.7?

> Maybe smtpd.py can be deprecated itself.

Maybe it can be fun to rewrite the module using asyncio, but I'm not convinced that a SMTP server in the Python stdlib is super useful.

(A HTTP server *is* very useful, it saved my life multiple times when I worked on embedded devices without any server available to transfer files.)
msg249855 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 22:54
Ideally these modules should emit a deprecation warning starting in 3.6 (when asyncio is no longer provisional) and we should strive to delete them per 3.8.

If nobody rewrites smptd.py using asyncio it should be deleted at the same time.
msg249873 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2015-09-05 00:33
I use smtpd.py for testing my code. But it's not such a big deal that I couldn't live without it. If I have some time to burn, I might convert it to asyncio so I can continue to use it.
msg249882 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-05 01:37
smtpd is used for testing smtplib.  It is also used in test_logging.  I would object to having it removed, but I suspect we can manage to rewrite it by 3.8.  Maybe Eric and I can collaborate on it.
msg249912 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-09-05 17:08
I opened http://bugs.python.org/issue25008 to discuss deprecating smtpd since this issue is about asyncore/asynchat.
msg250109 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-09-07 14:10
On Sep 04, 2015, at 08:55 PM, STINNER Victor wrote:

>Maybe it can be fun to rewrite the module using asyncio, but I'm not
>convinced that a SMTP server in the Python stdlib is super useful.

As I mentioned in issue25008, removing smtpd would be a hardship for myself,
and I don't think I'm alone.  I would however love to have (and maybe even
work on <wink>) an asyncio-based version.  I do think it is useful to have an
SMTP server in the stdlib - and actually, I'd also like to have an LMTP server
there too.  But I think we can take baby steps toward this by some folks
getting together and writing a third party asyncio-based module, releasing
this on PyPI and eventually leading to its inclusion in Python down the road.
I think there's no hurry to remove smtpd, but it can be deprecated-in-place
like optparse, as someone else mentioned.
msg250150 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-09-08 03:33
> Now that we've got asyncio in two releases (3.4 and 3.5) 
> we should start deprecating asyncore and asynchat.

If we care about making it easier for organizations to move from Python 2 to Python 3, we should develop a strong aversion to deprecating modules that have been around for a long time and aren't broken.

In addition, we should give weight and support to the numerous projects that are trying to have a code base the runs in both Python 2 and Python 3 (hence the astonishing success of the six module). 

Already my tooling for testing Python 3.6 is broken because some core dev aggressively removed functions from the inspect module that were deprecated in favor of using Signature objects.  That broke the popular Hypothesis testing tool which was carefully written to work for both Python 2 and Python 3.  In order to work again that tool will have to be modified to copy in the old code that was just taken out.

don't-leave-your-users-behind-ly yours ...
msg250151 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-08 03:53
Then can we at least close any feature requests for asyncore/asynchat as wontfix? (And porting smtpd.py to asyncio is still a good idea.)
msg250214 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-08 15:20
Yes, stopping changing asyncore/asynchat also means it becomes a stable target for people who *are* using it for 2/3 code.  We may have effectively done this already (without closing the open issues): the last asyncore-specific change (as opposed to library-wide changes that also hit asyncore) was a ResourceWarning added by Victor in June of 2014.  Likewise for asynchat.
msg250223 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-09-08 15:40
If you care of smooth python 2 => python 3 migration, I suggest to
deprecate smtpd and remove it later, when asyncore & asynchat will
also be removed (ex: in Python 4?).

I suggest to write a *new* SMTP server module using asyncio. As I
explained, the API will be very different for the user of the API.
msg250232 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-09-08 16:11
Victor's suggestion also aligns with my thinking on the subject as well: deprecate the modules but simply leave them sitting there for compatibility reasons until Python 4 of whenever we feel like the Python 2 transition is done for those that will ever bother making the switch. That way people's expectations are in the proper alignment of where things will end up and code won't break if it straddles Python 2 & 3 short of having to silence a deprecation warning. I think this is a good use of PendingDeprecationWarning and the message both in the deprecation and the docs can make it clear that the removal date is not planned but you should avoid using the module in new code.
msg250233 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-09-08 16:14
> Ideally these modules should emit a deprecation warning starting in 3.6

Is it ok to add the PendingDeprecationWarning in Python 3.5.1?
msg250234 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-08 16:29
> Is it ok to add the PendingDeprecationWarning in Python 3.5.1?

I prefer to wait until 3.6.  A bugfix release should not rock the boat, it should not make your code emit new warnings.
msg250239 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-09-08 17:01
In addition to a PendingDeprecationWarning in 3.6, it might be nice to put a note in the asyncore/asynchat docs like we did for optparse.
msg250260 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-09-08 22:19
IIRC I flagged up oustanding asyncore/asynchat issues months or even years back and Victor closed a lot down, so there shouldn't be many left to deal with.
msg279352 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2016-10-25 05:07
Hi,

Attached is the patch that adds pending deprecation warnings to asyncore and asynchat. Please review. Thanks :)
msg279416 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-25 15:50
Applied:

remote: notified python-checkins@python.org of incoming changeset bb23770f82f1
remote: notified python-checkins@python.org of incoming changeset 3b8dfe6f5bcb
msg279417 - (view) Author: Roundup Robot (python-dev) Date: 2016-10-25 15:57
New changeset bb23770f82f1 by Guido van Rossum in branch '3.6':
Issue 25002: Deprecate asyncore/asynchat. Patch by Mariatta.
https://hg.python.org/cpython/rev/bb23770f82f1

New changeset 3b8dfe6f5bcb by Guido van Rossum in branch 'default':
Issue 25002: Deprecate asyncore/asynchat. Patch by Mariatta. (3.6->3.7)
https://hg.python.org/cpython/rev/3b8dfe6f5bcb
msg279424 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-25 17:30
This change broke buildbots.

$ ./python -We -m test.regrtest test_os 
Run tests sequentially
0:00:00 [1/1] test_os
test test_os crashed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 151, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/home/serhiy/py/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 677, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/home/serhiy/py/cpython/Lib/test/test_os.py", line 5, in <module>
    import asynchat
  File "/home/serhiy/py/cpython/Lib/asynchat.py", line 48, in <module>
    import asyncore
  File "/home/serhiy/py/cpython/Lib/asyncore.py", line 65, in <module>
    PendingDeprecationWarning, stacklevel=2)
PendingDeprecationWarning: asyncore module is deprecated in 3.6. Use asyncio instead.

test_os failed

1 test failed:
    test_os

Total duration: 59 ms
Tests result: FAILURE
msg279429 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-25 17:54
Sorry about that. Should I roll it back or is there a way to make the test pass (expect this deprecation)?
msg279436 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-25 18:11
I think replacing

    import asynchat
    import asyncore

with

    with warnings.catch_warnings():
        warnings.simplefilter('ignore', PendingDeprecationWarning)
        import asynchat
        import asyncore

can help.

asynchat and asyncore are used in several tests.

In long term asynchat and asyncore should be replaced with alternatives.
msg279437 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2016-10-25 18:17
Sorry about this Serhiy. I can work on another patch based on your code snippet later today.

Should I create a different ticket about replacing asyncore and asynchat?
msg279438 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-25 18:26
> Should I create a different ticket about replacing asyncore and asynchat?

This may require several tickets, one per case.
msg279440 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-25 18:32
Ah, asyncore/asynchat are used in smtpd!

It looks to me that we should first write asyncio alternative to smtpd, then deprecate smtpd, and only after this we can deprecate asyncore/asynchat.
msg279442 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2016-10-25 18:34
Thanks Serhiy, 

I created these two tickets:
https://bugs.python.org/issue28534
https://bugs.python.org/issue28533
msg279444 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-10-25 18:51
The alternative has already been written: aiosmtpd, in the aiolibs project.  The question is should it be added to the stdlib...
msg279450 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-25 19:32
Serhiy, so should I revert this patch for now? Or are the silent
deprecation warnings outside the test suite okay? (In that case, maybe
Mariatta can upload a patch?)
msg279464 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-10-25 23:20
R. David Murray added the comment:
> The alternative has already been written: aiosmtpd, in the aiolibs
project.  The question is should it be added to the stdlib...

I suggest to keep it on PyPI to keep fast releases.
msg279469 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-10-26 00:19
Perhaps it is okay to keep the documentation changes, but I think either the library changes should be reverted or worked around where the modules are still in use.

I normally run the tests with -Werror, and the failures I get are:

* test_ssl, test_smtplib, test_poplib, test_logging, test_ftplib, test_support: tests use asyncore
* test_os: test uses asynchat (When you import asynchat, the first error complains about importing asyncore, there are actually two warnings)
* test_all: This seems to ignore DeprecationWarning; perhaps an exception for PendingDeprecationWarning should also be added?
* test_asyncore and test_asynchat: Obviously these have to still test the modules, so they should anticipate the warnings, perhaps using Serhiy’s code
* test_smtpd: smtpd module itself uses asyncore; see Issue 25008
msg279477 - (view) Author: Roundup Robot (python-dev) Date: 2016-10-26 01:43
New changeset 6eb3312a9a16 by Guido van Rossum in branch '3.6':
Issue #25002: Back out asyncore/asynchat deprecation.
https://hg.python.org/cpython/rev/6eb3312a9a16

New changeset 2879185bc511 by Guido van Rossum in branch 'default':
Issue #25002: Back out asyncore/asynchat deprecation. (3.6->3.7)
https://hg.python.org/cpython/rev/2879185bc511
msg279478 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-26 01:44
OK, backed out the code changes, kept the docs.
History
Date User Action Args
2016-11-17 19:48:31giampaolo.rodolasetnosy: + giampaolo.rodola
2016-10-26 01:44:21gvanrossumsetmessages: + msg279478
2016-10-26 01:43:46python-devsetmessages: + msg279477
2016-10-26 00:19:26martin.pantersetnosy: + martin.panter
messages: + msg279469
2016-10-25 23:20:52hayposetmessages: + msg279464
2016-10-25 19:32:51gvanrossumsetmessages: + msg279450
2016-10-25 19:22:16BreamoreBoysetnosy: - BreamoreBoy
2016-10-25 18:51:17r.david.murraysetmessages: + msg279444
2016-10-25 18:34:16Mariattasetmessages: + msg279442
2016-10-25 18:32:33serhiy.storchakasetdependencies: + Deprecate smtpd (based on deprecated asyncore/asynchat)
messages: + msg279440
2016-10-25 18:26:20serhiy.storchakasetversions: + Python 3.7
resolution: fixed ->
messages: + msg279438

components: + Library (Lib)
type: enhancement
stage: needs patch
2016-10-25 18:17:35Mariattasetmessages: + msg279437
2016-10-25 18:11:25serhiy.storchakasetmessages: + msg279436
2016-10-25 17:54:28gvanrossumsetmessages: + msg279429
2016-10-25 17:31:35serhiy.storchakasetstatus: closed -> open
2016-10-25 17:30:35serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg279424
2016-10-25 15:57:15python-devsetnosy: + python-dev
messages: + msg279417
2016-10-25 15:50:54gvanrossumsetstatus: open -> closed
resolution: fixed
messages: + msg279416
2016-10-25 05:07:33Mariattasetfiles: + issue25002.patch

nosy: + Mariatta
messages: + msg279352

keywords: + patch
2015-09-08 22:19:57BreamoreBoysetnosy: + BreamoreBoy
messages: + msg250260
2015-09-08 17:01:01rhettingersetmessages: + msg250239
2015-09-08 16:29:28gvanrossumsetmessages: + msg250234
2015-09-08 16:14:52hayposetmessages: + msg250233
2015-09-08 16:11:46brett.cannonsetmessages: + msg250232
2015-09-08 15:40:49hayposetmessages: + msg250223
2015-09-08 15:20:18r.david.murraysetmessages: + msg250214
2015-09-08 03:53:14gvanrossumsetmessages: + msg250151
2015-09-08 03:33:10rhettingersetnosy: + rhettinger
messages: + msg250150
2015-09-07 14:10:45barrysetmessages: + msg250109
2015-09-05 17:08:52brett.cannonsetmessages: + msg249912
2015-09-05 01:37:13r.david.murraysetnosy: + r.david.murray
messages: + msg249882
2015-09-05 00:33:07eric.smithsetnosy: + eric.smith
messages: + msg249873
2015-09-04 23:08:42brett.cannonsetnosy: + brett.cannon
2015-09-04 22:54:41gvanrossumsetmessages: + msg249855
2015-09-04 20:55:41hayposetnosy: + haypo
messages: + msg249833
2015-09-04 20:53:09gvanrossumsetmessages: + msg249831
2015-09-04 20:52:18gvanrossumsetnosy: + barry
2015-09-04 20:50:04dstufftsetnosy: + dstufft
messages: + msg249830
2015-09-04 20:48:03gvanrossumcreate