classification
Title: Unable to copy ssl.SSLContext
Type: behavior Stage: patch review
Components: SSL Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, miss-islington, ned.deily, pitrou, serhiy.storchaka, vitaly.krug
Priority: normal Keywords: patch

Created on 2018-03-08 06:09 by vitaly.krug, last changed 2018-12-07 06:02 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 6099 open christian.heimes, 2018-03-13 08:50
PR 10934 merged serhiy.storchaka, 2018-12-05 19:28
PR 11000 merged miss-islington, 2018-12-06 20:37
PR 11001 merged serhiy.storchaka, 2018-12-06 20:43
PR 11008 merged serhiy.storchaka, 2018-12-06 21:17
Messages (24)
msg313422 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-08 06:09
```
import copy
import ssl

copy.copy(ssl.create_default_context())
```
results in 

`TypeError: can't pickle SSLContext objects`

This prevents me from being able to `copy.deepcopy()` an object that references `ssl.SSLContext`.

The apparent root cause is apparently that `ssl.SSLContext` passes an extra arg to its `__new__` method, but doesn't implement the method `__getnewargs__` that would let `copy` extract the extra arg.
msg313437 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-03-08 12:30
This is rather a feature than a bug. It is not possible to make a copy of a SSLContext object because OpenSSL doesn't support the operation. A context contains elements that can't be cloned easily, e.g. session resumption tickets.
msg313458 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-08 21:37
Hi Christian, thank you for following up. Here is my use case, and perhaps you can suggest something else that will work:

I am refactoring the transport layer in the Pika AMQP client library. The user provides an ssl.SSLContext instance for connecting to an AMQP broker (e.g., RabbitMQ). Pika will resolve the hostname via getaddrinfo and make attempts to establish TCP and AMQP connection to the candidate IP addresses until one succeeds in establishing an AMQP connection over SSL. Each connection attempt will require a fresh unadulterated clone of the ssl.SSLContext instance provided by user to avoid any side-effects from prior connection attempts.

How can I obtain this pristine clone cleanly for each new connection attempt?
msg313459 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-08 21:40
Also, updating ssl.SSLContext documentation about intentional inability to copy generically and suggestion how to go about it if you need to obtain a clone or similar would be terrific and save developers time so they won't run into this gotcha when designing and implementing solutions. Many thanks!
msg313621 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-11 21:23
> Each connection attempt will require a fresh unadulterated clone of the ssl.SSLContext instance provided by user to avoid any side-effects from prior connection attempts.

What would those side-effects be?
msg313703 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-12 21:26
> What would those side-effects be?

Christian Heimes suggested that 
> A context contains elements that can't be cloned easily, e.g. session resumption tickets.

My concern then would be potential side-effects from such session resumption tickets and anything else that one connection attempt might save/change within an SSL Context that might have an undesirable side-effect on the follow-on connection attempts.
msg313704 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-12 21:30
You won't have any session resumption tickets until a connection succeeds.  And even then, I don't think it would be a problem.  By design, SSL contexts are meant to be re-used accross multiple connections.
msg313731 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-03-13 08:54
Antoine is correct. Session resumption is fully transparent for the application layer. The context object is designed to be reused by multiple connections to same or different servers.
msg314322 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-03-23 20:25
PR 6099 changes error message "can't pickle SSLContext objects" to "cannot serialize SSLContext object", right? Wouldn't be better to change the standard error message instead?
msg314347 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-24 00:35
It would be very helpful to make a statement in SSLContext's documentation to the effect that it's not copyable. This is frankly the first time I run into a non-copyable object.I spend quite a bit of time researching this after implementing a copying strategy that failed. It would have saved me (and others...) so much time is there was a warning in SSLContext documentation about not being able to serialize/copy/deepcopy by design!

Also, making that exception message more generic (ha, I wasn't pickling anything?!) as Serhiy Storchaka suggested would be a welcome addition, but not replacement for documentation.
msg314367 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-03-24 14:28
Serhiy,
I don't understand what you are trying to tell me. "cannot serialize '%s' object" is used all over the interpreter, e.g. io, pickle, etree, and more. I feel it's the standard message.

Vitaly,
A lot of objects can't be copied. It's the general case for all kinds of objects that hold operating system resources (files, sockets) or wrap external C libraries (bz2, lzma, sqlite, ssl). We generally don't document that an object cannot be pickled, serialized, or copied. If documentation doesn't state that an object is copy-able or doesn't provide a dedicated copy method, than it can't be copied.
msg314375 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-24 17:37
Thank you, I'll consider myself having been warned :)

On Sat, Mar 24, 2018, 7:28 AM Christian Heimes <report@bugs.python.org>
wrote:

>
> Christian Heimes <lists@cheimes.de> added the comment:
>
> Serhiy,
> I don't understand what you are trying to tell me. "cannot serialize '%s'
> object" is used all over the interpreter, e.g. io, pickle, etree, and more.
> I feel it's the standard message.
>
> Vitaly,
> A lot of objects can't be copied. It's the general case for all kinds of
> objects that hold operating system resources (files, sockets) or wrap
> external C libraries (bz2, lzma, sqlite, ssl). We generally don't document
> that an object cannot be pickled, serialized, or copied. If documentation
> doesn't state that an object is copy-able or doesn't provide a dedicated
> copy method, than it can't be copied.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33023>
> _______________________________________
>
msg314382 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-03-24 19:24
Christian, I thought the reason of adding __getstate__ methods which raise an exception is that the existing error message doesn't satisfy your. What is the other reason?
msg314417 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-03-25 14:27
That's the only reason for PR 6099. The change makes it more obvious that SSL objects can't be serialized or copied.
msg314419 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-03-25 16:29
See issue33138.
msg318894 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-07 05:40
We have a languishing PR here.  What should be done with it?  If I understand correctly, Serhiy is proposing a more general cleanup in Issue33138 PR 6239.  If that is merged, can PR 6099 here be simplified?  What branches should it apply to? (retargeting for 3.8 pending comments)
msg318899 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-07 06:44
The change to the dup() method looks as a bug fix to me. It needs to be backported to all maintained versions. New tests can be backported to all versions except perhaps 2.7 (not tested). Changes to __getstate__() will be not needed if merge issue33138 PR 6239 (they just change the error message). But it may be needed to apply them to 2.7 only.
msg318903 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-07 07:06
If changes to __getstate__() be merged in 2.7, I suggest to unify them with issue33138.
msg328980 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-31 09:26
After discussing on Python-Dev all error messages were changed to "cannot pickle 'XXX' object".

https://mail.python.org/pipermail/python-dev/2018-October/155599.html

I suggest to not add __getstate__() methods with different error messages. But the dup() method should be fixed.
msg331151 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-05 19:54
PR 10934 fixes just a typo in the name of NotImplementedError. It fixes also a similar typo in IDLE. It adds new tests and fixes existing tests for NotImplementedError in SSLSocket.
msg331247 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-06 20:37
New changeset 42b1d6127bd8595522a78a75166ebb9fba74a6a2 by Serhiy Storchaka in branch 'master':
bpo-33023: Fix NotImplemented to NotImplementedError. (GH-10934)
https://github.com/python/cpython/commit/42b1d6127bd8595522a78a75166ebb9fba74a6a2
msg331248 - (view) Author: miss-islington (miss-islington) Date: 2018-12-06 20:52
New changeset 6485aa6eb1024672f08afdd577e2b5792eb6b03c by Miss Islington (bot) in branch '3.7':
bpo-33023: Fix NotImplemented to NotImplementedError. (GH-10934)
https://github.com/python/cpython/commit/6485aa6eb1024672f08afdd577e2b5792eb6b03c
msg331249 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-06 21:00
New changeset 7a2cf1e7d3bf300e98c702589d405734f4a8fcf8 by Serhiy Storchaka in branch '3.6':
bpo-33023: Fix NotImplemented to NotImplementedError. (GH-10934). (GH-11001)
https://github.com/python/cpython/commit/7a2cf1e7d3bf300e98c702589d405734f4a8fcf8
msg331270 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-07 06:02
New changeset 324e1790094708538acf2e7795f9c44e3732aaf7 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-33023: Fix NotImplemented to NotImplementedError. (GH-10934). (GH-11001) (GH-11008)
https://github.com/python/cpython/commit/324e1790094708538acf2e7795f9c44e3732aaf7
History
Date User Action Args
2018-12-07 06:02:36serhiy.storchakasetmessages: + msg331270
2018-12-06 21:17:35serhiy.storchakasetpull_requests: + pull_request10242
2018-12-06 21:00:41serhiy.storchakasetmessages: + msg331249
2018-12-06 20:52:46miss-islingtonsetnosy: + miss-islington
messages: + msg331248
2018-12-06 20:43:02serhiy.storchakasetpull_requests: + pull_request10240
2018-12-06 20:37:15miss-islingtonsetpull_requests: + pull_request10239
2018-12-06 20:37:03serhiy.storchakasetmessages: + msg331247
2018-12-05 19:54:08serhiy.storchakasetmessages: + msg331151
2018-12-05 19:28:54serhiy.storchakasetpull_requests: + pull_request10189
2018-10-31 09:26:14serhiy.storchakasetmessages: + msg328980
2018-06-07 07:06:18serhiy.storchakasetmessages: + msg318903
2018-06-07 06:44:46serhiy.storchakasetmessages: + msg318899
2018-06-07 05:40:25ned.deilysetnosy: + ned.deily

messages: + msg318894
versions: + Python 3.8, - Python 2.7, Python 3.6
2018-03-25 16:29:21serhiy.storchakasetmessages: + msg314419
2018-03-25 14:27:06christian.heimessetmessages: + msg314417
2018-03-24 19:24:08serhiy.storchakasetmessages: + msg314382
2018-03-24 17:37:41vitaly.krugsetmessages: + msg314375
2018-03-24 14:28:33christian.heimessetmessages: + msg314367
2018-03-24 00:35:22vitaly.krugsetmessages: + msg314347
2018-03-23 20:25:41serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg314322
2018-03-13 08:54:03christian.heimessetmessages: + msg313731
2018-03-13 08:50:12christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request5862
2018-03-12 21:30:14pitrousetmessages: + msg313704
2018-03-12 21:26:48vitaly.krugsetmessages: + msg313703
2018-03-11 21:24:04pitrousettype: crash -> behavior
2018-03-11 21:23:51pitrousetnosy: + pitrou
messages: + msg313621
2018-03-08 21:40:39vitaly.krugsetmessages: + msg313459
2018-03-08 21:37:09vitaly.krugsetmessages: + msg313458
2018-03-08 12:30:30christian.heimessetmessages: + msg313437
2018-03-08 06:10:00vitaly.krugsetassignee: christian.heimes

components: + SSL
nosy: + christian.heimes
2018-03-08 06:09:44vitaly.krugsettype: crash
2018-03-08 06:09:21vitaly.krugcreate