classification
Title: Build libraries with "/OPT:REF" linker optimization on Windows
Type: enhancement Stage: resolved
Components: C API, Extension Modules, IO, Windows Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: Austin-Lamb, eryksun, paul.moore, steve.dower, terry.reedy, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2021-01-04 17:42 by Austin-Lamb, last changed 2021-02-19 23:51 by Austin-Lamb. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24098 merged Austin-Lamb, 2021-01-04 17:59
Messages (10)
msg384333 - (view) Author: Austin Lamb (Austin-Lamb) * Date: 2021-01-04 17:42
I noticed that CPython and the various libraries built out of the repo aren't using the "/OPT:REF" linker optimization on Windows.  This optimization allows the linker to throw away dead/unreachable code, resulting in a substantial decrease in binary size.  That in turn also reduces the amount of Disk I/O that must be done to get these binaries in memory (which can be meaningful on spinning hard drives that are still rather common), and the reference set of applications using Python.

I'll send a PR for this shortly, just filing the issue to be able to reference it in the PR.
msg384695 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-08 23:27
The PR says that this deletes "uncalled code".  I imagine that this is safe for a .exe.  But for a .dll?  As i understand it, one can, from python code, access any function via ctypes, so it seems to me that there is no way for the linker to know which callable functions in a .dll can be safely deleted.  Am I missing something?
msg385729 - (view) Author: Austin Lamb (Austin-Lamb) * Date: 2021-01-26 18:28
I don't know much about how ctypes exposes stuff - are these exposed as DLL exports?  Or if not, how do they get exposed?

The linker will remove things it can't find a callsite for, but it leaves in exported functions as of course those are by definition externally callable.  If ctypes is called another way, I can help confirm whether anything is accidentally being stripped out.
msg386093 - (view) Author: Austin Lamb (Austin-Lamb) * Date: 2021-02-01 19:35
Ping - can anyone suggest either how ctypes does its exporting, or suggest tests I could run to confirm if this change affects python code?  The entire test suite in the repo passes.
msg386097 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-01 19:51
On Windows it should only affect non-exported functions. Comparing "dumpbin /exports" output will tell you whether anything has changed there.

The bigger concern is if it merges identical functions that are compared by function pointer. I know we've had issues with that in the past, and I forget how we've dealt with that. I thought OPT:REF would do this optimisation too (though ISTR it's known as COMDAT folding and may have its own option).
msg386490 - (view) Author: Austin Lamb (Austin-Lamb) * Date: 2021-02-04 19:10
You're right to think that folding is a problem Steve, but thankfully my PR does not enable COMDAT folding for that very reason :).

/OPT:ICF enables identical COMDAT folding, but surprisingly-to-me /OPT:REF seems to still enable a small amount of  folding (I confirmed with the linker dev at Microsoft that this is true).  Thus, just /OPT:REF alone caused test failures as there are test cases verifying COMDAT folding didn't occur.

Thus, in my PR I am using /OPT:REF,NOICF to explicitly disable any ICF'ing at all and I see no folding occurring anywhere, and those COMDAT-folding-sensitive tests pass with that flag.

The dumpbin /exports output is identical after this change.
msg387358 - (view) Author: Austin Lamb (Austin-Lamb) * Date: 2021-02-19 21:42
What are the next steps for this - anything else I can provide, or is someone able to take a look at the Pull Request?

Thanks!
msg387366 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-19 23:27
New changeset b4af629f4d4868ef74ee298d66259fae78c7fd89 by Austin Lamb in branch 'master':
bpo-42825: Enable /OPT:REF (GH-24098)
https://github.com/python/cpython/commit/b4af629f4d4868ef74ee298d66259fae78c7fd89
msg387367 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-19 23:27
The ping was enough. Thanks!
msg387370 - (view) Author: Austin Lamb (Austin-Lamb) * Date: 2021-02-19 23:51
Thanks Steve!
History
Date User Action Args
2021-02-19 23:51:05Austin-Lambsetmessages: + msg387370
2021-02-19 23:27:33steve.dowersetstatus: open -> closed
messages: + msg387367

assignee: steve.dower
resolution: fixed
stage: patch review -> resolved
2021-02-19 23:27:08steve.dowersetmessages: + msg387366
2021-02-19 21:42:46Austin-Lambsetmessages: + msg387358
2021-02-04 21:30:21orbit.hierarchysettype: performance -> enhancement
components: + Extension Modules, IO, C API
2021-02-04 19:10:36Austin-Lambsetmessages: + msg386490
2021-02-01 19:51:38steve.dowersetmessages: + msg386097
2021-02-01 19:35:15Austin-Lambsetmessages: + msg386093
2021-01-26 18:28:40Austin-Lambsetmessages: + msg385729
2021-01-08 23:27:30terry.reedysetnosy: + terry.reedy
messages: + msg384695
2021-01-06 06:29:26ammar2setnosy: + eryksun

title: Optimization opportunity on Windows -> Build libraries with "/OPT:REF" linker optimization on Windows
2021-01-04 17:59:34Austin-Lambsetkeywords: + patch
stage: patch review
pull_requests: + pull_request22928
2021-01-04 17:42:40Austin-Lambcreate