Title: obsolete macpath module dangerously broken and should be removed
Type: Stage: needs patch
Components: macOS Versions: Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Chi Hsuan Yen, Mariatta, chortos, haypo, jesstess, macpherson.scott, merwok, ned.deily, python-dev, ronaldoussoren, serhiy.storchaka
Priority: normal Keywords: easy, patch

Created on 2010-09-14 05:14 by ned.deily, last changed 2016-10-05 05:41 by Mariatta.

File name Uploaded Description Edit
macpath_join_fix.patch chortos, 2011-08-18 12:08 fix for macpath.join('', *comps) review
macpath_join_fix_with_test.patch chortos, 2011-08-18 18:41 fix for macpath.join('', *comps) with a test review
3.6-deprecate-macpath.patch Chi Hsuan Yen, 2016-10-02 14:04 review
3.7-remove-macpath.patch Chi Hsuan Yen, 2016-10-02 14:07 review
3.7-remove-macpath_ver2.patch Chi Hsuan Yen, 2016-10-02 14:20 review
Messages (19)
msg116367 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-09-14 05:21
The macpath module in the standard library purports to supply "the Mac OS 9 (and earlier) implementation of the os.path module.  It can be used to manipulate old-style Macintosh pathnames on Mac OS X (or any other platform).  The following functions are available in this module: normcase(), normpath(), isabs(), join(), split(), isdir(), isfile(), walk(), exists(). For other functions available in os.path dummy counterparts are available."

Old-style Mac pathnames are not further documented - in fact, the above quote is the entire external documentation for the module - but they are ones using colon separators, like ("MacHD:Users:nad:macpath_test:file"). These style path names were initially supported on Mac OS X by compatibility APIs for programs ported from classic Mac OS but those interfaces have long been deprecated in OS X and in most cases are not available in 64-bit execution mode.

Even if one did have a need to use the obsolete old-style paths, the macpath module is currently practically unusable for anything other than simple character manipulations of the path.  Nearly all of the functions that actually call OS routines are broken in one or more ways.

Those that do make file system calls are calling APIs that are expecting normal posix-style ('/' separated paths) incorrectly with old-style (":) paths (ispath, isdir, lexists, etc) which means they only give correct results in the trivial case of unqualified file or directory names, i.e. those in the working directory.

The islink() function confusingly is testing whether a path points to a Finder Alias file (not a symlink).  Unfortunately, the Carbon API used for that test does not exist in a 64-bit version and all of the Python Carbon modules were removed in Python 3.

$ arch -i386 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
$ arch -x86_64 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
$ python3.2 -c 'import macpath; print(macpath.islink("alias_to_file"))'

The underlying import errors are being masked by "try"'s:

$ arch -i386 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
$ arch -x86_64 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'ResolveAliasFile'
$ python3.2 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named Carbon.File

The realpath function is also broken in 2.7, calling Carbon.File.FSResolveAliasFile with a ":" separated path when it expects a "/" path, and is totally broken in 3.x (no Carbon modules).

While macpath *could* be repaired by proper documentation, fixing the mishmash of path types internally, and supplying C wrappers and/or alternative 64-bit APIs, it does not seem worth the effort as these style paths are seldom encountered in modern code.  Considering how broken the module currently is and given that it probably was simply overlooked in the Python 2 to 3 transition, I think a case could be made for immediate removal prior to Python 3.2 release and even for a 3.1.x maintenance release.  Short of that, it should be cleared documented as deprecated in 3.2, 3.1, and 2.7 along with warnings about broken functionality along with added DeprecationWarnings to the module.

I can write up patches to do that depending on what the consensus is.
msg116387 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-09-14 11:53
I'm -0 on removing the module wholesale because parts of OSX still use MacOS9 style paths (in particular some Carbon APIs and AppleScript).

I'm +1 on removing all functions that aren't path manipulation code, that is 'macpath.join' would stay while 'macpath.islink' would go, and fixing the documentation.

And to be complete: I'm -1 on fixing macpath.islink and other functions that access a life filesystem because the macpath module is only meant to be used for that on MacOS9 and that platform is no longer supported.  Path manipulation code is still useful.

BTW. The module cannot be removed from 2.7 and 3.1 due to API stability requirements.
msg116388 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-09-14 11:56
The solution may be different depending on Python version. I propose to keep macpath in Python 2.7, just because it's too late to change such thing in Python2. But we may mark macpath as deprecated, eg. "macpath will be removed in Python 3.2" (see above). I suppose that the same treatement can be used on Python 3.1 (maybe with a different warning type/message).

But for Python 3.2, I suggest to drop macpath. Said differently: drop Mac OS 9 support. If you really need Mac OS 9 support, continue to use Python 2.7. If I understood correctly, macpath is not used for os.path in Python 3.2, and it is broken (at least on 64 bits build).

But well, I am not a Mac programmer, so I don't know if Mac OS 9 is still commonly installed or not.
msg116389 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-09-14 12:00
MacOS9 is already unsupported, except for macpath all traces of OS9 support are gone with the possible exception of distutils (I removed traces of OS9 support code in 3.2 and those got restored when Tarek replaced distutils by the version from the 3.1 branch, I don't remember if I redid my OS9-removal work).

As I wrote earlier the path manipulation code (macpath.join, macpath.split etc.) are still useful because OS9-style paths are still used in some parts of OSX.
msg116517 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-09-16 09:13
Patches in progress.
msg116973 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-09-20 21:12
I agree with Ronald’s proposal.

(Re “I don't remember if I redid my OS9-removal work”: Looks like you didn’t.)
msg142310 - (view) Author: Oleg Oshmyan (chortos) Date: 2011-08-18 12:08
I fully agree with Ronald’s proposal. And for a start, here is a trivial patch that fixes macpath.join('', ...) [at the moment it just returns its last argument]. By the way, this fix is probably eligible for inclusion in Python 2.7 too.

I also have ‘implement macpath.relpath’ in my to-do list, although in practice this means nothing.
msg142365 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-08-18 16:44
Thanks for the patch.  Is the function already tested or does someone need to add a test?
msg142397 - (view) Author: Oleg Oshmyan (chortos) Date: 2011-08-18 18:41
It is already tested but here is a new version of the patch that expands the existing test to cover the situation that was broken (plus another one).
msg217238 - (view) Author: Jessica McKellar (jesstess) * Date: 2014-04-27 04:14
Thanks for writing up this issue, ned.deily, and thanks for providing a patch, chortos.

I couldn't find documentation clearly specifying what the correct behavior of macpath.join should be (i.e. what are the exact rules for leading and trailing colons), but comparing the old and updated behavior against the new tests, this patch does make the function's behavior more consistent.

* The patch passes `make patchcheck`.
* The full test suite passes with this patch.

=> commit review
msg217901 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-05 00:43
The patch appears fine but it really doesn't have anything to do with the gist of this issue.  The problem remains that much of macpath is fundamentally broken.  In the intervening years since the issue was opened, I contend that any need for OS 9 style paths (aka HFS paths) has diminished even further so I would still support starting in Python 3.5 the deprecation and removal process for this module.  But, as long as it remains, to address the current problems at a minimum a patch would be needed to remove or raise exceptions for all of the macpath functions that make file system calls and to update the documentation.
msg227687 - (view) Author: Roundup Robot (python-dev) Date: 2014-09-27 15:58
New changeset 2ae2ca9d2b66 by Serhiy Storchaka in branch '2.7':
Issue #9850: Fixed macpath.join() for empty first component.  Patch by

New changeset 54987723de99 by Serhiy Storchaka in branch '3.4':
Issue #9850: Fixed macpath.join() for empty first component.  Patch by

New changeset e29866cb6b98 by Serhiy Storchaka in branch 'default':
Issue #9850: Fixed macpath.join() for empty first component.  Patch by
msg227689 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-09-27 16:01
Committed macpath_join_fix_with_test.patch with some additional tests. Thank you for your contribution Oleg.

Needed a patch for deprecating all module or some functions.
msg247128 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-22 15:08
I think it is by now safe to remove macpath, AFAIK there is no real use-case anymore for having classic MacOS9 paths on any recentish version of OSX.]

I'm setting the version to 3.6 because it is too late to do this for Python 3.5, but it can be done for 3.6.
msg247242 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-07-24 03:47
In third-party code the path manipulating functions can be used in unexpected manner than doesn't relate to paths on MacOS. Also "import macpath" can be left even with the use of macpath was eliminated or are dead code. There is a chance to break third-party code when just remove macpath without deprecation period.
msg277844 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-02 02:18
It's a bit late in the 3.6 cycle to be removing macpath, but it's not too late to mark it in 3.6 as deprecated and to be removed in 3.7.  If someone wants to write two patches, one for 3.6 to add a deprecation warning to the code and to the docs, the other for 3.7 to actually remove all traces of macpath, that would be great!
msg277892 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-10-02 14:07
Here are my tries. There are more MacOS-specific module names in Tools/freeze/ They are unrelated to macpath so I leave them to the next bug.
msg277893 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-10-02 14:20
Oops forgot to commit deleted files in the 3.7 patch.
msg278102 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2016-10-05 05:41
I'm just reading PEP-4 now for procedure of declaring a module deprecated.
Should PEP-4 page be updated with the proposal / info for macpath deprecation?

Date User Action Args
2016-10-05 05:41:35Mariattasetmessages: + msg278102
2016-10-02 14:20:48Chi Hsuan Yensetfiles: + 3.7-remove-macpath_ver2.patch

messages: + msg277893
2016-10-02 14:07:11Chi Hsuan Yensetfiles: + 3.7-remove-macpath.patch

messages: + msg277892
2016-10-02 14:04:41Chi Hsuan Yensetfiles: + 3.6-deprecate-macpath.patch
2016-10-02 03:09:51Mariattasetnosy: + Mariatta
2016-10-02 02:37:50berker.peksagsetkeywords: + easy, - needs review
2016-10-02 02:18:14ned.deilysetassignee: ronaldoussoren ->
type: enhancement ->
messages: + msg277844
versions: + Python 3.7
2016-10-01 10:34:57Chi Hsuan Yensetnosy: + Chi Hsuan Yen
2016-08-21 00:10:25macpherson.scottsetnosy: + macpherson.scott
2015-07-24 03:47:04serhiy.storchakasetmessages: + msg247242
2015-07-22 15:08:06ronaldoussorensetmessages: + msg247128
versions: + Python 3.6, - Python 3.5
2014-09-27 16:01:28serhiy.storchakasetversions: - Python 2.7
nosy: + serhiy.storchaka

messages: + msg227689

type: behavior -> enhancement
stage: commit review -> needs patch
2014-09-27 15:58:41python-devsetnosy: + python-dev
messages: + msg227687
2014-05-05 00:43:31ned.deilysetmessages: + msg217901
2014-04-27 04:14:44jesstesssetversions: + Python 2.7, Python 3.5, - Python 3.3
nosy: + jesstess

messages: + msg217238

keywords: + needs review
stage: needs patch -> commit review
2011-08-18 18:41:19chortossetfiles: + macpath_join_fix_with_test.patch

messages: + msg142397
2011-08-18 16:44:49merwoksetmessages: + msg142365
2011-08-18 12:08:50chortossetfiles: + macpath_join_fix.patch

nosy: + chortos
messages: + msg142310

keywords: + patch
2011-03-11 02:09:38merwoksetnosy: ronaldoussoren, haypo, ned.deily, merwok
versions: + Python 3.3, - Python 3.1, Python 2.7, Python 3.2
2010-09-20 21:12:27merwoksetnosy: + merwok
messages: + msg116973
2010-09-16 09:13:43ned.deilysetmessages: + msg116517
2010-09-14 12:00:38ronaldoussorensetmessages: + msg116389
2010-09-14 11:56:18hayposetnosy: + haypo
messages: + msg116388
2010-09-14 11:53:47ronaldoussorensetmessages: + msg116387
2010-09-14 05:21:56ned.deilysetmessages: - msg116365
2010-09-14 05:21:47ned.deilysetmessages: + msg116367
2010-09-14 05:14:10ned.deilycreate