classification
Title: Deprecate smtpd (based on deprecated asyncore/asynchat)
Type: Stage: resolved
Components: Documentation Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: barry Nosy List: barry, brett.cannon, eric.smith, giampaolo.rodola, haypo, neic, r.david.murray, rhettinger, terry.reedy
Priority: normal Keywords: patch

Created on 2015-09-05 17:08 by brett.cannon, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
aiosmtpd.patch r.david.murray, 2015-09-28 00:28 review
Pull Requests
URL Status Linked Edit
PR 274 merged barry, 2017-02-24 16:59
PR 278 merged barry, 2017-02-24 20:19
PR 279 merged barry, 2017-02-24 20:23
PR 280 merged barry, 2017-02-24 20:26
PR 703 larry, 2017-03-17 21:00
PR 552 closed dstufft, 2017-03-31 16:36
Messages (18)
msg249911 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-09-05 17:08
As mentioned in passing in issue #25002, perhaps smtpd should be deprecated or at least be updated to use asyncio.
msg249913 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-09-05 17:10
I should mention one of the motivations behind deprecating smtpd is that no one should be using that module to run an SMTP server and it doesn't provide utility like http.server does and so we probably should drop the code to prevent anyone accidentally using it and not have to maintain it.
msg249935 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-09-05 22:05
Bringing in Eric V Smith's comment in
http://bugs.python.org/issue25002#msg249873 :

'''
I use smtpd.py for testing my code. But it's not such a big deal that I couldn't live without it. If I have some time to burn, I might convert it to asyncio so I can continue to use it.

'''

Bringing in R David Murray's comment in http://bugs.python.org/issue25002#msg249882 :

'''smtpd is used for testing smtplib.  It is also used in test_logging.  I would object to having it removed, but I suspect we can manage to rewrite it by 3.8.  Maybe Eric and I can collaborate on it.
'''
msg249942 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-09-05 22:32
FWIW, I would like to see the module live on (after conversion to asyncio).  Doug Hellman's example at https://pymotw.com/2/smtpd/ makes for a nice demonstration of Python's capabilities.
msg249953 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-09-06 02:51
Removing smtpd would definitely be a hardship for me right now, probably for obvious reasons.  I use it in testing frameworks, and even wrote a library called lazr.smtptest that is built around smtpd.  In Mailman, we have an LMTP server built on smtpd that implements *the* official way of getting mail into the system from an upstream mail server.  Mailman 3 core FWIW is Python 3.4 and 3.5 compatible.

That all being said, I've also been thinking lately about an asyncio-based reimplementation of SMTP and LMTP.  asyncore/chat and smtpd are not the easiest modules to work with, extend, understand, or debug.  I'd be very happy to collaborate with folks on an asyncio-based version.  Maybe we can put that out as a third party module, with a long term plan on pulling it into the stdlib and promoting it instead of smtpd.
msg250019 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-06 20:38
I believe that smtpd is used for a *lot* of testing code (anyone who is writing smtp clients in python, pretty much, and probably some writing in other languages).  I really don't see the rationale for removing it, assuming we can rewrite it "in time".

On the other hand, are we really going to *remove* asyncore/asynchat?  Or is this a "deprecation in place" like optparse?  If the latter then I don't really care if the same is done to smtpd :)  If the former, why, if we've kept optparse?
msg250110 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-09-07 14:26
Even if it may be possible to support asyncore/asynchat and asyncio at the same time, I would prefer to drop asyncore/asynchat support to simplify the code.

The problem is that the API will be very different.

asyncio has a different design. You don't instanciate a class to open a network connection, but you have to call "yield from loop.create_connection()" in a coroutine.

We might provide helper blocking functions starting the server for us and then running the event loop until CTRL+c is pressed (or another event to stop to stop the loop).

Rewriting smtpd with asyncio would allow to use it in an asyncio application. In 2015, I expect to see more and more applications using asyncio than asyncore/asynchat.

Can we modify the issue title to "Rewrite smtpd with asyncio"?
msg250111 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-09-07 14:34
On Sep 07, 2015, at 02:26 PM, STINNER Victor wrote:

>Can we modify the issue title to "Rewrite smtpd with asyncio"?

Sure, although I'm currently thinking it's best to go third party until the
API and implementation settle down.
msg250149 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-09-08 03:25
Unless there is a drop-in replacement, I don't think this should be deprecated or removed.  Our biggest problem in the Python world is getting organizations using Python 2 to switch to Python 3.  Deprecating modules that have been a part of the firmament for a long time works against that goal.  

Side note: a quick search shows that people are still writing SMTPD modules in other languages (i.e. Go now has one)
msg250510 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-09-12 03:22
http://nullege.com/codes/search?cq=import+smtpd
shows 135 hits.  FWIW, import+asyncore get 972.
msg251736 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-28 00:28
Here is a proof of concept port of smtpd to asyncio.  Obviously I'm not suggesting we commit this patch (it isn't complete...the tests don't run).  Instead I'm posting it as an outgrowth the of the sprint that was conducted today.  The DC team took a different tack, so I'm not submitting this as a patch to their repo, and since it is useful to look at the diff against default, it seems appropriate to post it here.

The advantage of this port is that it uses the existing smtpd protocol code, which is the "field tested" code.  Of course, I did have to change how the data is turned from bytes into lines, so I may have introduced some bugs.

As discussed, *using* an asyncio based smtpd has a different calling API than using the asynchat one.  However, the *pattern* is similar, so it may not be crazy to replace smtpd with the cleaned up version of this patch, exactly because the number of places this is used is relatively small and mostly in test code.  IMO it's a better option than deleting smtpd from the stdlib.

The code as is can be run using its CLI.  I tested it via telnet using the DebuggingServer and hand coding an email...for some reason I can't currently import smtplib (Enum errors in socket), so I didn't test it with smtplib, but it should work.
msg252285 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-10-04 20:15
I'm liking RDM's port of smtpd.py.  I've done some clean ups, bug fixes, and added some debugging.  See this branch over on gitlab:

https://gitlab.com/python-smtpd-hackers/aiosmtpd/tree/issue25508

Try `python3 server.py` then run client.py in another terminal (it uses smtplib) or just telnet.

I plan to continue pursuing this approach; please comment over there.
msg253097 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-10-16 20:51
This has come along pretty well.  The project is managed here:

https://gitlab.com/python-smtpd-hackers/aiosmtpd

and documentation here:

http://aiosmtpd.readthedocs.org/en/latest/
msg288391 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-02-22 23:10
aiosmtpd is coming along nicely:

http://aiosmtpd.readthedocs.io/en/latest/

We'll soon have a 1.0 release.  Still, I don't think it's worth pulling this into the stdlib.  But we could silently deprecate it and point to aiosmtpd in the docs.
msg290408 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-03-24 23:47
New changeset 66b5092fac4264efdc9c508a7dd425fa9833e147 by Barry Warsaw in branch '3.5':
bpo-25008: Deprecate smtpd and point to aiosmtpd (#274) (#279)
https://github.com/python/cpython/commit/66b5092fac4264efdc9c508a7dd425fa9833e147
msg290409 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-03-24 23:47
New changeset 0a1b656d8ce3da14f8acf947477b8e998e68ef3b by Barry Warsaw in branch '3.6':
bpo-25008: Deprecate smtpd and point to aiosmtpd (#274) (#278)
https://github.com/python/cpython/commit/0a1b656d8ce3da14f8acf947477b8e998e68ef3b
msg290410 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-03-24 23:48
New changeset f37b0cb230069481609b0bb06891b5dd26320504 by Barry Warsaw in branch '3.4':
bpo-25008: Deprecate smtpd and point to aiosmtpd (#274) (#280)
https://github.com/python/cpython/commit/f37b0cb230069481609b0bb06891b5dd26320504
msg290411 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-03-24 23:48
New changeset fba79dc568030091f84110fccac4cb64c2fef020 by Victor Stinner (Barry Warsaw) in branch 'master':
bpo-25008: Deprecate smtpd and point to aiosmtpd (#274)
https://github.com/python/cpython/commit/fba79dc568030091f84110fccac4cb64c2fef020
History
Date User Action Args
2017-03-31 16:36:16dstufftsetpull_requests: + pull_request907
2017-03-24 23:48:17hayposetmessages: + msg290411
2017-03-24 23:48:02barrysetmessages: + msg290410
2017-03-24 23:47:53barrysetmessages: + msg290409
2017-03-24 23:47:46barrysetmessages: + msg290408
2017-03-17 21:00:34larrysetpull_requests: + pull_request603
2017-02-24 20:32:41barrysetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2017-02-24 20:26:49barrysetpull_requests: + pull_request250
2017-02-24 20:23:17barrysetpull_requests: + pull_request249
2017-02-24 20:19:31barrysetpull_requests: + pull_request248
2017-02-24 16:59:22barrysetpull_requests: + pull_request246
2017-02-22 23:10:44barrysetversions: + Python 3.7, - Python 3.6
title: Deprecate smtpd (based on deprecated asyncore/asynchat): write a new smtp server with asyncio -> Deprecate smtpd (based on deprecated asyncore/asynchat)
messages: + msg288391

assignee: barry
components: + Documentation, - Library (Lib)
2016-11-17 19:48:42giampaolo.rodolasetnosy: + giampaolo.rodola
2016-10-25 18:32:33serhiy.storchakalinkissue25002 dependencies
2016-09-07 18:11:15r.david.murrayunlinkissue26209 dependencies
2016-02-06 18:45:21terry.reedylinkissue26209 dependencies
2015-10-16 20:51:21barrysetmessages: + msg253097
2015-10-08 15:02:58neicsetnosy: + neic
2015-10-04 20:15:16barrysetmessages: + msg252285
2015-09-28 00:28:14r.david.murraysetfiles: + aiosmtpd.patch
keywords: + patch
messages: + msg251736
2015-09-18 13:51:48hayposettitle: Deprecate smtpd -> Deprecate smtpd (based on deprecated asyncore/asynchat): write a new smtp server with asyncio
2015-09-12 03:22:56terry.reedysetnosy: + terry.reedy
messages: + msg250510
2015-09-08 03:25:02rhettingersetmessages: + msg250149
2015-09-07 14:44:23eric.smithsetnosy: + eric.smith
2015-09-07 14:34:13barrysetmessages: + msg250111
2015-09-07 14:26:49hayposetnosy: + haypo
messages: + msg250110
2015-09-06 20:38:15r.david.murraysetnosy: + r.david.murray
messages: + msg250019
2015-09-06 02:51:48barrysetmessages: + msg249953
2015-09-05 22:32:55rhettingersetmessages: + msg249942
2015-09-05 22:05:25rhettingersetnosy: + barry, rhettinger
messages: + msg249935
2015-09-05 17:10:08brett.cannonsetmessages: + msg249913
2015-09-05 17:08:13brett.cannoncreate