This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: dis does not work with the new optimized ops
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, corona10, kj, martmists
Priority: normal Keywords:

Created on 2021-10-04 09:56 by martmists, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (7)
msg403126 - (view) Author: Martmists (martmists) * Date: 2021-10-04 09:56
Python 3.11.0a0 (heads/main:e6d1aa1, Oct  4 2021, 10:33:36) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> c = bytes([127, 1, 0, 2])  # LOAD_FAST__LOAD_FAST (1, 2)
>>> dis.dis(c)
          0 <127>                    1
          2 <0>
>>> c = bytes([48, 0, 161, 0])  # LOAD_GLOBAL_BUILTIN (0) CALL_FUNCTION (0)
>>> dis.dis(c)
          0 <48>
          2 CALL_METHOD              0

---

The reason for this seems to be the lack of definition in `opcode.py` aside from being mentioned in _specialized_instructions. Additionally, it seems dis._unpack_opargs is not yet prepared to support opcodes like LOAD_FAST__LOAD_FAST which take two arguments.

While I don't expect this to be a first priority in this big change, it's important to not forget it.
msg403127 - (view) Author: Martmists (martmists) * Date: 2021-10-04 10:03
Another thing I found to be quite odd is JUMP_ABSOLUTE_QUICK, which is set as 40, but as it's derived from JUMP_ABSOLUTE it still requires an argument. This makes HAVE_ARGUMENT useless as there's an op that doesn't follow the convention.
msg403136 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-10-04 13:42
Thanks Martmists for the bug report, I'll try to address some of your concerns:

> The reason for this seems to be the lack of definition in `opcode.py` aside from being mentioned in _specialized_instructions.

If I understood Mark's intentions in PEP 659 correctly, when you call dis.dis on any normally generated code object, specialized instructions will never show up. Specialized instructions are a hidden implementation detail, and instead the normal generic instruction will appear. So there isn't really a need to support it (any Python hand-crafted specialized instruction will likely segfault anyways).

> Additionally, it seems dis._unpack_opargs is not yet prepared to support opcodes like LOAD_FAST__LOAD_FAST which take two arguments

Sorry, I don't quite understand what you mean here. The superinstructions do not use different bytecode format: ie they take only one oparg. LOAD_FAST__LOAD_FAST works by reading the *next* instruction's oparg. So:

LOAD_FAST   0
LOAD_FAST   1
LOAD_CONST  0

Becomes:

LOAD_FAST__LOAD_FAST 0
LOAD_FAST            1
LOAD_CONST           0

LOAD_FAST__LOAD_FAST will increment program counter inline [1] to read the oparg of the instruction after it without going through eval loop, then on dispatch it will effectively "skip" the next instruction and go straight to LOAD_CONST. The speedup comes from skipping eval-loop and dispatch overhead of one instruction. The bytecode format is unchanged.

[1] https://github.com/python/cpython/blob/07cf10bafc8f6e1fcc82c10d97d3452325fc7c04/Python/ceval.c#L1755

I hope this helps you.
msg403137 - (view) Author: Martmists (martmists) * Date: 2021-10-04 13:55
> Specialized instructions are a hidden implementation detail, and instead the normal generic instruction will appear.

Will this change be reflected in co_code? If yes, then I'm aware of cases where this may be used by bytecode modification libraries. If not, how would it deal with co_code being modified?

> Sorry, I don't quite understand what you mean here. The superinstructions do not use different bytecode format: ie they take only one oparg.
I see, so it'll just replace the first of the two ops to streamline the process rather than replacing the second one with a dummy instruction
msg403139 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-10-04 14:36
> Will this change be reflected in co_code?
No. As per PEP 659: 

"Once any instruction in a code object has executed a few times, that code object will be "quickened" by allocating a new array for the bytecode that can be modified at runtime, and is not constrained as the code.co_code object is."

Example:

x = lambda a: len(a)
print(x.__code__.co_code)

#quickened
for _ in range(10):
 x('')

print(x.__code__.co_code)

There should be no change.

> If not, how would it deal with co_code being modified?
Disclaimer: I am not a contributor to such libraries, so my knowledge in this area is limited.

co_code is readonly and the bytes object it points is also immutable. So I'd imagine that they'd have to create a brand new code object, in which case it shouldn't matter. But if there is some way to modify co_code in-place using ctypes: I don't know. Sorry! The best way to find out is to test those libraries with 3.11 and see if such functionality breaks, then file a bug report here please. Thank you.
msg403140 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-10-04 14:46
127 and 48 aren't instructions.
>>> opcode.stack_effect(48)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid opcode or oparg

So the output from dis seems correct.


>If not, how would it deal with co_code being modified?

The co_code attribute is immutable and read only. It cannot be modified.

Specialized instructions and super instructions are private to the interpreter.
If they are visible to Python code (not counting ctypes hacks), then that is a bug.
msg403241 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-10-05 13:31
Closing this issue as the behavior in dis module is as intended. For any potential issues to 3rd party packages discovered in the future, consider opening an issue in that package's issue tracker, or submitting a brand new issue to bugs.python.org.
History
Date User Action Args
2022-04-11 14:59:50adminsetgithub: 89525
2021-10-05 13:31:06kjsetstatus: open -> closed
resolution: not a bug
messages: + msg403241

stage: resolved
2021-10-04 14:46:12Mark.Shannonsetmessages: + msg403140
2021-10-04 14:36:21kjsetmessages: + msg403139
2021-10-04 13:55:25martmistssetmessages: + msg403137
2021-10-04 13:42:46kjsetmessages: + msg403136
2021-10-04 12:35:50corona10setnosy: + Mark.Shannon, kj
2021-10-04 11:57:19corona10setnosy: + corona10
2021-10-04 10:03:20martmistssetmessages: + msg403127
2021-10-04 09:56:52martmistscreate