Title: Python 3.5.2 non-segfaulting bugs (from PyPy)
Type: behavior Stage:
Components: Versions: Python 3.5
Status: open Resolution:
Dependencies: 5322 18961 25937 28891 Superseder:
Assigned To: Nosy List: arigo, mattip, ncoghlan, serhiy.storchaka
Priority: normal Keywords:

Created on 2016-12-06 11:55 by arigo, last changed 2020-12-06 07:13 by mattip.

Messages (17)
msg282526 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:55
As discussed on python-dev, I am creating omnibus issues from the lists of crashers, of wrong-according-to-the-docs, and of strange-behavior-only issues that I found while developing Python 3.5.2 support for PyPy.  These occur with CPython 3.5.2 but most of them are likely still here in trunk.

This is the issue containing the bugs that are wrong according to documentation, or at least clearly (imo) unexpected behavior.
msg282527 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:55
(B1) on modern Linux: if the first call in the process to
  socketpair() ends in a EINVAL, then cpython will (possibly wrongly)
  assume it was caused by SOCK_CLOEXEC and not use SOCK_CLOEXEC at all
  in the future
msg282528 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:55
(B2) fcntl.ioctl(x, y, buf, mutate_flag): mutate_flag is there for the case
  of buf being a read-write buffer, which is then mutated in-place.
  But if we call with a read-only buffer, mutate_flag is ignored (instead
  of rejecting a True value)---ioctl(x, y, "foo", True) will not actually
  mutate the string "foo", but the True is completely ignored.  (I think
  this is a bug introduced during the Argument Clinic refactoring.)
msg282529 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:55
(B3) re.sub(b'y', bytearray(b'a'), bytearray(b'xyz')) -> b'xaz'

  re.sub(b'y', bytearray(b'\\n'), bytearray(b'xyz')) -> internal TypeError
msg282530 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:56
(B4) if you have a stack of generators where each is in 'yield from' from
  the next one, and you call '.next()' on the outermost, then it enters
  and leaves all intermediate frames.  This is costly but may be
  required to get the sys.settrace()/setprofile() hooks called.
  However, if you call '.throw()' or '.close()' instead, then it uses a
  much more efficient way to go from the outermost to the innermost
  frame---as a result, the enter/leave of the intermediate frames is not
  invoked.  This can confuse coverage tools and profilers.  For example,
  in a stack ``f1()->f2()->f3()``, vmprof would show f3() as usually
  called via f2() from f1() but occasionally called directly from f1().
msg282531 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:56
(B5) this is an old issue that was forgotten twice on the
  issue tracker: ``class C: __new__=int.__new__`` and ``class C(int):
  __new__=object.__new__`` can each be instantiated, even though they
  shouldn't.  This is because ``__new__`` is completely ignored if it is
  set to any built-in function that uses ``tp_new_wrapper`` as its C code
  (many of the built-in types' ``__new__`` are like that).,  In (at least) CPython 3.5,
  a few classes work only thanks to abuse of this bug: for example,
  ``io.UnsupportedOperation.__new__(io.UnsupportedOperation)`` doesn't
  work, but that was not noticed because ``io.UnsupportedOperation()``
  mistakenly works.
msg282532 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:57
(B6) this program fails the check for no sys.exc_info(), even though at
  the point this assert runs (called from the <== line) we are not in
  any except/finally block.  This is a generalization of

    import sys

    def g():
            raise ValueError
        except ValueError:
            yield 1
        assert sys.exc_info() == (None, None, None)
        yield 2

    gen = g()

        raise IndexError
    except IndexError:
        assert next(gen) is 1
    assert next(gen) is 2    # <==
msg282533 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:57
(B7) frame.clear() does not clear f_locals, unlike what a test says

    def test_locals_clear_locals(self):
        # Test f_locals before and after clear() (to exercise caching)
        f, outer, inner = self.make_frames()
        self.assertEqual(outer.f_locals, {})
        self.assertEqual(inner.f_locals, {})

  This test passes, but the C-level PyFrameObject has got a strong
  reference to f_locals, which is only updated (to be empty) if the
  Python code tries to read this attribute.  In the normal case,
  code that calls clear() but doesn't read f_locals afterwards will
  still leak everything contained in the C-level f_locals field.  This
  can be shown by this failing test::

    import sys

    def g():
        x = 42
        return sys._getframe()

    frame = g()
    d = frame.f_locals
    assert d == {}   # fails!  but 'assert d is frame.f_locals' passes,
                     # which shows that this dict is kept alive by
                     # 'frame'; and we've seen that it is non-empty
                     # as long as we don't read frame.f_locals.
msg282534 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:58
(B8) also discussed in connection with

  weak dicts (both kinds) and weak sets have an implementation of
  __len__ which doesn't give the "expected" result on PyPy, and in some
  cases on CPython too.  I'm not sure what is expected and what is not.
  Here is an example on CPython 3.5.2+ (using a thread to run the weakref
  callbacks only, not to explicitly inspect or modify 'd')::

    import weakref, _thread
    from queue import Queue

    queue = Queue()
    def subthread(queue):
        while True:
    _thread.start_new_thread(subthread, (queue,))

    class X:
    d = weakref.WeakValueDictionary()
    while True:
        x = X()
        d[52] = x
        del x
        while list(d) != []:
        assert len(d) == 0  # we've checked that list(d)==[], but this may fail

  On CPython I've seen the assert fail only after editing the function
  WeakValueDictionary.__init__.remove() to add ``time.sleep(0.01)`` as
  the first line.  Otherwise I guess the timings happen to make that test
msg282535 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:59
(B9) CPython 3.5.2: this ``nonlocal`` seems not to have a reasonable
  effect (note that if we use a different name instead of ``__class__``,
  this example correctly complain that there is no binding in the outer
  scope of ``Y``)::

    class Y:
        class X:
            nonlocal __class__
            __class__ = 42
        print(locals()['__class__'])     # 42
        print(__class__)                 # but this is a NameError
msg282536 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-12-06 11:59
(B10) Follow-up on issue #25388: running ``python`` if contains
  the following bytes...

  * ``b"#\xfd\n"`` => we get a SyntaxError: Non-UTF-8 code
  * ``b"# coding: utf-8\n#\xfd\n"`` => we get no error!
msg282544 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-06 12:14
About (B10) see issue18961 and issue25937.
msg282545 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-06 12:48
About (B3): Only str and bytes are supported. The support of bytearray templates is not documented and is not guarantied.
msg282586 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-12-07 01:27
I broke B9 for the "nonlocal __class__" misbehaviour out to a separate issue:

I love zero-argument super() as a user, but as an interpreter maintainer... ugh :P
msg282613 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-07 10:37
(B5) seems was just fixed in issue5322.

>>> class C: __new__ = int.__new__
>>> C()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: int.__new__(C): C is not a subtype of int
>>> class C(int): __new__ = object.__new__
>>> C()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object.__new__(C) is not safe, use int.__new__()
msg382587 - (view) Author: mattip (mattip) * Date: 2020-12-06 07:08
It seems (B6) is still failing (both on PyPy3.7 and on CPython3.8.6)
msg382588 - (view) Author: mattip (mattip) * Date: 2020-12-06 07:13
No, my bad, B6 passes now on CPython
Date User Action Args
2020-12-06 07:13:44mattipsetmessages: + msg382588
2020-12-06 07:08:33mattipsetnosy: + mattip
messages: + msg382587
2016-12-07 10:37:17serhiy.storchakasetdependencies: + object.__new__ argument calling autodetection faulty
messages: + msg282613
2016-12-07 09:32:35serhiy.storchakasetdependencies: + Standardise more behaviours for zero-argument super() __class__ and __classcell__
2016-12-07 01:27:44ncoghlansetnosy: + ncoghlan
messages: + msg282586
2016-12-06 12:48:18serhiy.storchakasetmessages: + msg282545
2016-12-06 12:15:00serhiy.storchakasetnosy: + serhiy.storchaka
dependencies: + Non-UTF8 encoding line, DIfference between utf8 and utf-8 when i define python source code encoding.
messages: + msg282544
2016-12-06 12:08:48arigosettype: behavior
2016-12-06 11:59:40arigosetmessages: + msg282536
2016-12-06 11:59:25arigosetmessages: + msg282535
2016-12-06 11:58:44arigosetmessages: + msg282534
2016-12-06 11:57:39arigosetmessages: + msg282533
2016-12-06 11:57:17arigosetmessages: + msg282532
2016-12-06 11:56:58arigosetmessages: + msg282531
2016-12-06 11:56:16arigosetmessages: + msg282530
2016-12-06 11:55:49arigosetmessages: + msg282529
2016-12-06 11:55:34arigosetmessages: + msg282528
2016-12-06 11:55:23arigosetmessages: + msg282527
2016-12-06 11:55:03arigocreate