classification
Title: Tkinter/IDLE: preserve clipboard on closure
Type: behavior Stage: needs patch
Components: IDLE, Tkinter Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: epaine, eryksun, taleinat, terry.reedy
Priority: normal Keywords: patch

Created on 2020-04-30 17:35 by epaine, last changed 2020-11-02 09:09 by taleinat.

Files
File name Uploaded Description Edit
tkinter-clipboard-on-exit.patch taleinat, 2020-05-20 14:36
Pull Requests
URL Status Linked Edit
PR 19819 closed epaine, 2020-04-30 17:36
Messages (25)
msg367765 - (view) Author: E. Paine (epaine) * Date: 2020-04-30 17:35
Currently, on a Windows machine, the clipboard contents is lost when the root is closed. This, therefore requires you to keep the IDLE instance open until after the copy has been complete (particularly annoying when copying between different IDLE instances). The solution is to pipe the tkinter clipboard contents to "clip.exe" (built-in to Windows) which will preserve it after root closure.
msg369002 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-16 02:50
Please write out a manual test example (steps 1, 2, ..., N) that fails now and passes with the patch.
msg369135 - (view) Author: E. Paine (epaine) * Date: 2020-05-17 16:09
Test example:

1) Open IDLE (shell or editor)
2) Copy some text in the IDLE window
3) Close the IDLE window (instance should end)
4) Paste into application of choice
Without the patch, the clipboard is cleared when the instance ends so nothing is pasted. With the patch, the content remains in the clipboard as expected.

Note: This is not a problem if clipboard history is turned on or the content is pasted before closure, however the order of events above is fairly common for me.


Encoding:

In the latest commit to the pull, I have changed the encoding from UTF-8 to UTF-16 and my reasoning is as follows:

1) Most importantly, using UTF-8, characters with a unicode value >=2^8 are incorrectly copied as multiple characters. UTF-8 does support these characters but uses a different number of bytes per character (which is presumably what is causing these issues - for example, "AĀ" is encoded as "\x41\xc4\x80", which pastes as "A─Ç") UTF-16, however, correctly works for all characters supported by Tcl (see next point).

2) "Strings in Tcl are encoded using 16-bit Unicode characters" (https://www.tcl.tk/man/tcl8.2.3/TclCmd/encoding.htm). Therefore, the encoding we choose should have at least 16 bits allocated per character (meaning either UTF-16 or UTF-32).

3) Windows' "Unicode-enabled functions [...] use UTF-16 (wide character) encoding, which is [...] used for native Unicode encoding on Windows operating systems" (https://docs.microsoft.com/en-us/windows/win32/intl/unicode). For me, this was what decided the new encoding (between the two given in the previous point).
msg369156 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-17 23:37
Now that I think about it, I have run into enough problems with ^V not pasting something copied with ^C that I always leave source windows open until successful.  I had not noticed that there is only a problem between windows (but this may be true) or after closing IDLE or that it only occurs when copying *from* IDLE.  In fact, the last might not be true if other apps have the same bug.  (I will start paying more attention after this is fixed.)
msg369165 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-18 03:25
eryksun, Piping to clip.exe is not working well.  On the patch, I asked if you know what Windows system call it uses, but I cannot request your review there.
msg369199 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-18 09:35
I can reproduce this behavior on macOS as well, so this doesn't seem to be Windows-specific.

This does not happen with other apps in general, so it is not normal behavior for apps.

Testing with a minimal tkinter app (see code below) gives similar behavior, so this appears to be an issue with tkinter and/or tcl/tk.


import tkinter
text = tkinter.Text()
text.pack()
tkinter.mainloop()
msg369287 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-18 18:48
I closed the PR but IMO this issue should remain open.

I am changing the title, though, since this is not actually Windows-specific.
msg369304 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-18 22:33
Thanks Tal.  I should have inquired about behavior on *nix before putting so much effort into a Windows-only workaround, and not try to rush something into the beta coming out today or tomorrow.  I will return to trying to do an invisible paste into a bare Text widget.
msg369322 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-19 06:44
Looking into this further, this seems to be an issue in tkinter rather than with tcl/tk. Running `wish a.tcl` with the following simple script does leave its text in the clipboard:

pack [text .t]
.t insert 1.0 "Test text"
clipboard clear
clipboard append -- [.t get 1.0 end]
exit

On the other hand, the following equivalent Python script leaves the clipboard empty:

import tkinter
text = tkinter.Text()
text.pack()
text.clipboard_clear()
text.clipboard_append("Testing again")
msg369334 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-19 11:45
I ran your tkinter code on Windows, without IDLE, and the clipboard was clear thereafter.  I added 'test.mainloop()' and observed the following.
1. Close immediately, clipboard is clear, as before.
2. Paste 'Testing again' into 'text' with ^V and close, and clipboard is still clear.
3. Paste elsewhere (here, Command Prompt, or IDLE), close, and 'Testing again' can be pasted anywhere.

4. When I run from an IDLE editor, paste into the editor (which in running in the IDLE process rather than the user process where the test code is running, and close, 'Testing again' remains on the clipboard.

It appears that either a. tkinter clear the clipboard unless pasted into another process, or b. tkinter (or tk) always clears the clipboard on exit and it is gone unless pulled out of the process first by pasting elsewhere.  The best option is for this to be fixed.

The example codes are not quite equivalent.  Tkinter clipboard_clear and _append call self.tk.call with the equivalent arguments *plus*, 
  self._options({'displayof':text._w}) == ('-displayof', '.!text')
However, commenting out the addition had no visible effect.
msg369338 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-19 12:20
Note that Tcl/Tk once had exactly the same issue on Windows, and they added specific code to "render" to the clipboard upon exit if the current app is the "clipboard owner".

https://core.tcl-lang.org/tk/tktview/732662

This was fixed here:

https://core.tcl-lang.org/tk/tktview/939389

Note that Tcl/Tk has separate OS-specific implementations of "TkClipCleanup" for Windows, macOS and "Unix".

https://github.com/tcltk/tk/search?q=TkClipCleanup

This only appears to be called by TkCloseDisplay, which is turn is never called elsewhere in the Tk codebase. tkinter never appears to call either of those functions. Perhaps this is the core issue?
msg369383 - (view) Author: E. Paine (epaine) * Date: 2020-05-19 19:49
I'm (sadly) not particularly familiar with C, though I have tried to trace the calls of TkClipCleanup. As @taleinat mentioned, it is called by the TkCloseDisplay, though, in turn, I believe this method is called by the DeleteWindowsExitProc method. The name "DeleteWindowsExitProc" is passed, in the Initialize method, to the TkCreateThreadExitHandler method (I assume method references work in a similar-enough way to Python?) along with the tsdPtr object.

I stopped tracing it at this point, and instead worked from the Python end. The Tkapp_New method calls the Tcl_CreateInterp method and I have three questions about this (each only applicable depending on the answer to the previous one):
1. Is the Tkapp_New method the correct one to be looking at for tcl/tk initialisation (I can't find where _tkinter.create is implemented)?
2. Is there a reason why the call is to tcl rather than tk?
3. Would changing this to tk cause the fix in tk to be applied?
msg369387 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-19 20:25
Indeed, you've got that pretty much correct. The call chain is:

Tk.__init__
_tkinter_create_impl (called as _tkinter.create() from Python)
Tkapp_New

Tkapp_New does a lot of things. In particular, it calls Tcl_CreateInterp and later Tcl_AppInit. Tcl_AppInit (apparently a local copy is usually  used!) calls Tcl_Init and later Tk_Init.

At this point the call chain enters tcl/tk code. Tk_Init calls Initialize (except on cygwin - I'm ignoring this). The inline comment for Initialize says "The core of the initialization code for Tk, called from Tk_Init and Tk_SafeInit." And at the very end of Initialize, we find the call TkCreateThreadExitHandler(DeleteWindowsExitProc, tsdPtr).
msg369389 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-19 20:52
During finalization, TkFinalizeThread would call DeleteWindowsExitProc (registered via TkCreateThreadExitHandler). This in turn is set as a thread-exit handler via Tcl_CreateThreadExitHandler upon the first call to TkCreateThreadExitHandler.

Now we're out of Tk and into Tcl itself. TkFinalizeThread would be called by Tcl's Tck_FinalizeThread. This, in turn, is called by Tcl_Finalize (for thread number zero?). And _tkinter sets an atexit handler to call Tcl_Finalize.

Ah, but no! That handler is actually not set, due to the "#if 0" before it! So the Tcl/Tk finalization is actually skipped, causing this issue.
msg369391 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-19 20:58
Indeed, adding a simple _tkinter.destroy() method which invokes Tcl_Finalize, and calling it in the Tk.destroy() method, makes copied text remain available after closing IDLE!

I did the above to test my hypothesis, but I'm not sure it would be a proper fix.
msg369392 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-19 21:03
epaine, if you'd like to create a new PR based on these findings, I'd be happy to review it!
msg369395 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-19 21:15
After reporting my experiments above, msg369334, I made further failing efforts to simulate pasting into another process, as in 3 and 4.  It might be that a concrete key event is needed.  So I strongly suspect that the solution for IDLE is indeed a tkinter solution, and it seems that patching _tkinter.c is needed.  (And a solution only for IDLE would not help other tkinter users.)  Please continue.
msg369406 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-19 23:47
The #if 0 was added by Guido in 43ff8683fe68424b9c179ee4970bb865e11036d6 in 1998, before the tcl/tk clip fix for Windows.

* Temporarily get rid of the registration of Tcl_Finalize() as a
low-level Python exit handler.  This can attempt to call Python code
at a point that the interpreter and thread state have already been
destroyed, causing a Bus Error.  Given the intended use of
Py_AtExit(), I'm not convinced that it's a good idea to call it
earlier during Python's finalization sequence...  (Although this is
the only use for it in the entire distribution.)

The code has a comment that was part of a multifile svn merge, so author unknown.

    /* This was not a good idea; through <Destroy> bindings,
       Tcl_Finalize() may invoke Python code but at that point the
       interpreter and thread state have already been destroyed! */

Calling Tcl_Finalize() within Tk.destroy avoids this as .destroy is called while python is running, either explicitly or when the associated window is closed.

However, it is possible to have more than 1 Tk instance, either accidentally or on purpose*, each with its own .tk, which I presume is the 'associated tcl interpreter' instance.  So Tk.destroy may be called more than once and each call must not disable other Tk instances.  To test: Create 2 roots and two Tk windows, destroy 1.  Does the other window disappear? Does root2.withdraw raise? Does adding widgets raise?

If yes, we would either need to count working Tk instances or try calling less shutdown stuff.
msg369440 - (view) Author: E. Paine (epaine) * Date: 2020-05-20 11:15
Multiple Tk instances are already recommended against, but what would be the implications of preventing multiple roots? 

A simple check could be added to the Tk class init which ensures _default_root is None (if it is not None, an error is raised). In this case, I think it would be much easier for the proposed changes to _tkinter, but also make future maintenance of tkinter easier.

I am currently investigating potential solutions based on what Tal has found, and will come back with details if I succeed (and thank you Tal for offering to review a PR).

Separately, should we change the issue from IDLE to tkinter, as that the fix we are looking at applying?
msg369444 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-20 11:39
Regarding multiple Tk instances, IMO we should try the straightforward solution first: Tcl/Tk has its own mechanisms for handling per-interpreter state, so we may not actually need to handle any of this ourselves.

Regarding the title of this issue, it is indeed a bug in Tkinter rather than IDLE, but it does affect IDLE significantly, and someone searching for this might search for IDLE rather than Tkinter. I suggest leaving the title as it is.
msg369453 - (view) Author: E. Paine (epaine) * Date: 2020-05-20 13:51
After some initial testing, I have found that while calling Tcl_Finalize  on window closure keeps the clipboard contents (as expected), it also finishes the Python interpreter.

The solution was to instead use the Tcl_FinalizeThread method, "which you can call if you just want to clean up per-thread state" (https://www.tcl.tk/man/tcl8.4/TclLib/Exit.htm).

Reading this, I was expecting it to stop further Tk instances from being created after the original one was closed, however some initial testing has found this to not be true.

I feel it is too early to create a PR for this yet (I need to do more research and properly understand the calls), but it is quite possible calling this method on root "destroy" is the only thing required to fix this issue.
msg369456 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-20 14:36
Attaching the changes I made while testing as a patch file.
msg369601 - (view) Author: E. Paine (epaine) * Date: 2020-05-22 13:13
Unfortunately, after lots of testing/experimenting, I cannot find a way to make the correct call/s at the correct time. The methods that call the exit handlers directly or through InvokeExitHandlers are Tcl_Exit, Tcl_Finalize & FinalizeThread (Tcl_FinalizeThread) – Tcl_ExitThread calls Tcl_FinalizeThread.

We want to call the exit handlers (to access TkClipCleanup) with as little else as possible and both Tcl_Exit & Tcl_Finalize call FinalizeThread, so I concluded the method we should call is Tcl_FinalizeThread. Tcl_Finalize also stops the Python interpreter (not quite sure why), so this should only be called when the interpreter is stopping/stopped.

The problem with Tcl_FinalizeThread, is that it also finalises some of the mutexs (most notably asyncMutex - part of the async system). Once the mutexs are finalised, I can't find a way of creating them again (I think they are a global variables created on load), meaning that a new tcl interpreter cannot be created in the same thread (and even if it can, any calls to it cause the Python interpreter to crash).

This means we cannot call the Tcl_FinalizeThread method either when the root is destroyed or when the Tkapp object is deleted, as the user could still request a new tcl interpreter. This leaves us with only one option: call it when the Python interpreter is closing. For this, which call is made doesn’t really matter, and so to test, I commented out the "#if 0", to see if this call fixed the clipboard issue (if it had worked, we could have deleted all bindings before calling Tcl_Finalize). Unfortunately, this did not fix the clipboard issue and so I did not deem it worth developing an "unbinder" (possibly because the GC had already destroyed the interpreter?).

I did not even get onto the issue of multiple, simultaneous interpreters but unless someone has an idea that could solve any of the issues above (or we ask the Tcl team to make TkClipCleanup a public method!), I can’t see how we can patch this issue.
msg376354 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-09-04 10:53
I closed duplicate issue #41709,  Linux Mint 19.3, python  3.6.8
msg380199 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-11-02 09:09
Perhaps we could get a Tcl/Tk dev to help with this?
History
Date User Action Args
2020-11-02 09:09:38taleinatsetmessages: + msg380199
2020-09-04 10:53:03terry.reedysetmessages: + msg376354
versions: - Python 3.7
2020-09-04 10:51:46terry.reedylinkissue41709 superseder
2020-05-22 13:13:23epainesetmessages: + msg369601
2020-05-20 14:36:01taleinatsetfiles: + tkinter-clipboard-on-exit.patch

messages: + msg369456
2020-05-20 13:51:12epainesetmessages: + msg369453
2020-05-20 11:39:08taleinatsetmessages: + msg369444
2020-05-20 11:15:40epainesetmessages: + msg369440
2020-05-19 23:48:04terry.reedysetversions: + Python 3.10
2020-05-19 23:47:54terry.reedysetmessages: + msg369406
2020-05-19 21:15:05terry.reedysetmessages: + msg369395
2020-05-19 21:03:21taleinatsetmessages: + msg369392
2020-05-19 20:58:46taleinatsetmessages: + msg369391
2020-05-19 20:52:04taleinatsetmessages: + msg369389
2020-05-19 20:25:56taleinatsetmessages: + msg369387
2020-05-19 19:49:43epainesetmessages: + msg369383
2020-05-19 12:20:40taleinatsetmessages: + msg369338
2020-05-19 11:45:45terry.reedysettitle: IDLE: preserve clipboard on closure -> Tkinter/IDLE: preserve clipboard on closure
messages: + msg369334
components: + Tkinter
stage: patch review -> needs patch
2020-05-19 06:44:39taleinatsetmessages: + msg369322
2020-05-18 22:33:36terry.reedysetmessages: + msg369304
stage: needs patch -> patch review
2020-05-18 18:48:35taleinatsetstatus: closed -> open
title: IDLE: preserve clipboard on closure on Windows -> IDLE: preserve clipboard on closure
stage: resolved -> needs patch
2020-05-18 18:48:19taleinatsetmessages: + msg369287
2020-05-18 16:32:37epainesetstatus: open -> closed
stage: patch review -> resolved
2020-05-18 09:35:58taleinatsetnosy: + taleinat
messages: + msg369199
2020-05-18 03:25:02terry.reedysetnosy: + eryksun
messages: + msg369165
2020-05-17 23:37:13terry.reedysettype: enhancement -> behavior
messages: + msg369156
2020-05-17 16:09:25epainesetmessages: + msg369135
2020-05-16 02:50:09terry.reedysettitle: IDLE preserve clipboard on closure (Windows) -> IDLE: preserve clipboard on closure on Windows
messages: + msg369002
versions: + Python 3.7, Python 3.8
2020-05-03 15:38:40epainesettype: enhancement
2020-04-30 17:36:42epainesetkeywords: + patch
stage: patch review
pull_requests: + pull_request19139
2020-04-30 17:35:47epainecreate