When using ssl module, I sometimes get unexpected error. The observed error varies in different situations. After the investigation, I found the reason was that ssl shutdown was not performed and sometimes RST was sent over the network instead of FIN.

I created a patch against 2.7 branch.

I don't think your patch is right:

- calling unwrap() already shuts down the SSL layer; this is the right way to do it and is documented as such: "Performs the SSL shutdown handshake, which removes the TLS layer from the underlying socket, and returns the underlying socket object"

- shutdown() right now isn't blocking; if you add a call to SSL shutdown, it can either block or fail with EAGAIN or similar, which is something people won't expect

- close() should simply close the file descriptor, like on a regular socket (if you call socket.close(), it won't shutdown the TCP connection, especially if there's another file descriptor referencing the same connection)

As for Modules/_ssl.c, the case where SSL_shutdown() returns 0 is already handled:

        if (err == 0) {
            /* Don't loop endlessly; instead preserve legacy
               behaviour of trying SSL_shutdown() only twice.
               This looks necessary for OpenSSL < 0.9.8m */
            if (++zeros > 1)
                break;
            /* Shutdown was sent, now try receiving */
            self->shutdown_seen_zero = 1;
            continue;
        }

... so I don't think anything more is necessary.

So I think things are fine right now and your patch shouldn't be applied.

Please run the test so that you'll see the problem.

2013/4/9 Antoine Pitrou <report@bugs.python.org>

>
> Antoine Pitrou added the comment:
>
> I don't think your patch is right:
>
> - calling unwrap() already shuts down the SSL layer; this is the right way
> to do it and is documented as such: "Performs the SSL shutdown handshake,
> which removes the TLS layer from the underlying socket, and returns the
> underlying socket object"
>
> - shutdown() right now isn't blocking; if you add a call to SSL shutdown,
> it can either block or fail with EAGAIN or similar, which is something
> people won't expect
>
> - close() should simply close the file descriptor, like on a regular
> socket (if you call socket.close(), it won't shutdown the TCP connection,
> especially if there's another file descriptor referencing the same
> connection)
>
> As for Modules/_ssl.c, the case where SSL_shutdown() returns 0 is already
> handled:
>
>         if (err == 0) {
>             /* Don't loop endlessly; instead preserve legacy
>                behaviour of trying SSL_shutdown() only twice.
>                This looks necessary for OpenSSL < 0.9.8m */
>             if (++zeros > 1)
>                 break;
>             /* Shutdown was sent, now try receiving */
>             self->shutdown_seen_zero = 1;
>             continue;
>         }
>
> ... so I don't think anything more is necessary.
>
> So I think things are fine right now and your patch shouldn't be applied.
>
> ----------
> nosy: +pitrou
> stage:  -> patch review
> versions:  -Python 2.6, Python 3.1, Python 3.2, Python 3.5
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17672>
> _______________________________________
>

> sometimes RST was sent over the network instead of FIN

Your client sends data, but the server never reads it: when a TCP socket is closed while there's still data in the input socket buffer, a RST is sent instead of a FIN. That's normal behaviour.

Client gets an exception in reading the socket, not in writing. Please run
the test code and see what happens.

2013/4/9 Charles-François Natali <report@bugs.python.org>

>
> Charles-François Natali added the comment:
>
> > sometimes RST was sent over the network instead of FIN
>
> Your client sends data, but the server never reads it: when a TCP socket
> is closed while there's still data in the input socket buffer, a RST is
> sent instead of a FIN. That's normal behaviour.
>
> ----------
> nosy: +neologix
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17672>
> _______________________________________
>

> Client gets an exception in reading the socket, not in writing. Please run
> the test code and see what happens.

Of course it gets ECONNRESET on subsequent recv(), that's how TCP works.

Just make your handler read from the socket and it won't happen anymore.

> Client gets an exception in reading the socket, not in writing.

Yes, it does, and the exception bears the error code SSL_ERROR_EOF (8), which is expected here.

The question is: why would you expect reading *not* to raise an exception while the remote end of the connection has been closed? TCPStreamHandler will only keep the connection alive as long as the handle() method is running, the connection is disposed of afterwards.

As an interface of ssl socket, server does not have to read, just write
some data.
The client side should be able to read the bytes that ther server sent. The
problem is that client will sometimes raise an unexpected SSLError in
reading the ssl socket because server side does not shutdown the ssl
session cleanly.

2013/4/9 Charles-François Natali <report@bugs.python.org>

>
> Charles-François Natali added the comment:
>
> > Client gets an exception in reading the socket, not in writing. Please
> run
> > the test code and see what happens.
>
> Of course it gets ECONNRESET on subsequent recv(), that's how TCP works.
>
> Just make your handler read from the socket and it won't happen anymore.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17672>
> _______________________________________
>

> As an interface of ssl socket, server does not have to read, just write
> some data.
> The client side should be able to read the bytes that ther server sent. The
> problem is that client will sometimes raise an unexpected SSLError in
> reading the ssl socket because server side does not shutdown the ssl
> session cleanly.

Once again, that's not how TCP works.
If your server doesn't read data that the client sent it, a RST will
be sent when the server closes its end.

Please do read the TCP spec, or use google:
http://cs.baylor.edu/~donahoo/practical/CSockets/TCPRST.pdf

I suggest closing this issue...

> As an interface of ssl socket, server does not have to read, just
> write
> some data.
> The client side should be able to read the bytes that ther server
> sent.

Please re-read your own code. The server does:

+            def handle(self):
+                self.wfile.write("123")

and the client does:

+                        s.recv(3)

So, yes, the client is able to read the 3 bytes that the server sent.
It's only when you are trying to read an additional byte that an
exception is raised, because the TCP connection has been closed
by the server.

The error looks like : SSLError(8, '_ssl.c:1363: EOF occurred in violation
of protocol')
But why we see "in violation of protocol" here?

2013/4/10 Antoine Pitrou <report@bugs.python.org>

>
> Antoine Pitrou added the comment:
>
> > Client gets an exception in reading the socket, not in writing.
>
> Yes, it does, and the exception bears the error code SSL_ERROR_EOF (8),
> which is expected here.
>
> The question is: why would you expect reading *not* to raise an exception
> while the remote end of the connection has been closed? TCPStreamHandler
> will only keep the connection alive as long as the handle() method is
> running, the connection is disposed of afterwards.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17672>
> _______________________________________
>

Ah, sorry I understood now.

> The error looks like : SSLError(8, '_ssl.c:1363: EOF occurred in
> violation
> of protocol')
> But why we see "in violation of protocol" here?

Because the SSL layer wasn't shutdown cleanly: the TCP connection was
closed while the SSL layer was still active. You have three solutions
around this:

- you can call unwrap() for a clean SSL shutdown (the server has to call
unwrap() too).

- you can use suppress_ragged_eofs=True with wrap_socket()

- you can simply avoid reading past the server's data, which will
solve the problem altogether

I think creating an ssl socket from existing socket from an instance
generated by library routine, and replace that socket with ssl socket is
very common usage. Injecting wrap_socket is very easy. But injecting unwrap
call is not easy.

In python 2.6, I got a plain socket.error of "connection reset" (not
SSLError) in client side in such situation without unwrap in server side.
The same code does not raise exception in python 2.7, which I don't know
why...

Any way, reading the data in server side will solve the problem, thanks.

2013/4/10 Antoine Pitrou <report@bugs.python.org>

>
> Antoine Pitrou added the comment:
>
> > The error looks like : SSLError(8, '_ssl.c:1363: EOF occurred in
> > violation
> > of protocol')
> > But why we see "in violation of protocol" here?
>
> Because the SSL layer wasn't shutdown cleanly: the TCP connection was
> closed while the SSL layer was still active. You have three solutions
> around this:
>
> - you can call unwrap() for a clean SSL shutdown (the server has to call
> unwrap() too).
>
> - you can use suppress_ragged_eofs=True with wrap_socket()
>
> - you can simply avoid reading past the server's data, which will
> solve the problem altogether
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17672>
> _______________________________________
>

Ok, closing the issue then.