classification
Title: Unpickling Path objects
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Antony.Lee, pitrou, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-06-01 09:18 by Antony.Lee, last changed 2021-09-01 08:14 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 28083 open Antony.Lee, 2021-08-31 09:32
Messages (12)
msg266810 - (view) Author: Antony Lee (Antony.Lee) * Date: 2016-06-01 09:18
Currently, pickling Path objects lead to issues when working across platforms: Paths (and, thus, objects that contain Paths) created on a POSIX platform (PosixPaths) cannot be unpickled on Windows and vice versa.  There are a few possibilities around this issue.

- Don't do anything about it, you should use PurePaths if you care about cross-platform compatibility: this would be pretty awkward, as any call to the Path API would require converting back the PurePath to a Path first.

- Silently convert Paths to PurePaths during pickling (a solution that seems to have been adopted by http://docs.menpo.org/en/stable/api/menpo/io/export_pickle.html for example): it would be better if Paths at least roundtripped correctly within a single platform.

- Convert Paths to PurePaths at unpickling time, only if the platform is different (and possibly with a warning): this is the least bad solution I came up with so far.  Note that calls to the Path API on a "converted" PurePath object would be invalid anyways as the PurePath (being of a different platform) cannot be representing a valid path.

Thoughts?
msg400703 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-08-31 09:51
What does "converting" mean in this context?
msg400706 - (view) Author: Antony Lee (Antony.Lee) * Date: 2021-08-31 10:21
It means the Path/PurePath that would be constructed with the same single str parameter, or equivalently that has the same os.fspath().
msg400709 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-31 10:35
I think that you mean that PosixPath will be converted to PurePosixPath on Windows, and WindowsPath will be converted to PureWindowsPath on Posix.

In can be easily achieved by making alias PosixPath=PurePosixPath on Windows and WindowsPath=PureWindowsPath on Posix. It will have other effects, would they be positive or negative?
msg400711 - (view) Author: Antony Lee (Antony.Lee) * Date: 2021-08-31 11:34
You are correct as to the meaning of "convert".
The alternative approach you suggest would also work, but that seems to go much more against the design of pathlib to me.  OTOH I guess it is up to Antoine to rule on that.
msg400714 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-08-31 11:47
I guess the question is: why do people expect Paths to be picklable accross platforms? Is there a particular use case?
msg400716 - (view) Author: Antony Lee (Antony.Lee) * Date: 2021-08-31 12:01
Despite the now well-known security limitations of pickle, it is still used as a simple way (from the user PoV) to exchange arbitrary Python objects (see e.g. https://joblib.readthedocs.io/en/latest/persistence.html).  Such objects can sometimes include Paths as attributes, and it seems unfortunate that the presence of a Path makes the entire object (which may include many more things than just the Path) impossible to unpickle on a different OS (especially if unpickling into a PurePath keeps all the functionality that makes sense on that other OS).
msg400719 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-08-31 12:16
I understand.  Unpickling to PurePath might make enough sense indeed...
msg400721 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-31 12:16
Pickling Patch can be useful in multiprocessing, you can pass the Patch argument to function executed in other process. It can also be useful if you save the state of your program and restore it at the next start. In both cases you pickle and unpickle on the same computer.

If unpickle Path on other platform, the only meaningful operation on result are these which are implemented in PurePath. So Antony's proposition looks reasonable to me.
msg400817 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-01 07:08
It would be nice to have simple general mechanism for everloading names of modules, classes and functions, similar to _compat_pickle. Currently you can do this with creating a subclass of Unpickler and overloading find_class(), but it is not so convenient. There is no way to register standard replacements globally.
msg400820 - (view) Author: Antony Lee (Antony.Lee) * Date: 2021-09-01 07:57
I guess I could instead add some OS-dependent entries for Path classes into _compat_pickle (to do the remapping at load time) if you prefer?  I don't have a strong preference either-way.
msg400822 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-01 08:14
_compat_pickle is only used with protocol < 3.
History
Date User Action Args
2021-09-01 08:14:57serhiy.storchakasetmessages: + msg400822
2021-09-01 07:57:55Antony.Leesetmessages: + msg400820
2021-09-01 07:08:14serhiy.storchakasetmessages: + msg400817
2021-08-31 12:16:41pitrousettype: behavior -> enhancement
versions: + Python 3.11, - Python 3.6
2021-08-31 12:16:11serhiy.storchakasetmessages: + msg400721
2021-08-31 12:16:01pitrousetmessages: + msg400719
2021-08-31 12:01:27Antony.Leesetmessages: + msg400716
2021-08-31 11:47:13pitrousetmessages: + msg400714
2021-08-31 11:34:00Antony.Leesetmessages: + msg400711
2021-08-31 10:35:42serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg400709
2021-08-31 10:21:36Antony.Leesetmessages: + msg400706
2021-08-31 09:51:43pitrousetmessages: + msg400703
2021-08-31 09:32:33Antony.Leesetkeywords: + patch
stage: patch review
pull_requests: + pull_request26526
2016-06-01 09:21:42SilentGhostsetnosy: + pitrou

type: behavior
versions: - Python 3.5
2016-06-01 09:18:04Antony.Leecreate