classification
Title: obsolete macpath module dangerously broken and should be removed
Type: Stage: resolved
Components: macOS Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Chi Hsuan Yen, Jim Fasarakis-Hilliard, Mariatta, chortos, eric.araujo, jesstess, macpherson.scott, ned.deily, python-dev, ronaldoussoren, serhiy.storchaka, vstinner
Priority: normal Keywords: easy, patch

Created on 2010-09-14 05:14 by ned.deily, last changed 2017-05-16 20:49 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
macpath_join_fix.patch chortos, 2011-08-18 12:08 fix for macpath.join('', *comps) review
macpath_join_fix_with_test.patch chortos, 2011-08-18 18:41 fix for macpath.join('', *comps) with a test review
3.6-deprecate-macpath.patch Chi Hsuan Yen, 2016-10-02 14:04 review
3.7-remove-macpath.patch Chi Hsuan Yen, 2016-10-02 14:07 review
3.7-remove-macpath_ver2.patch Chi Hsuan Yen, 2016-10-02 14:20 review
Pull Requests
URL Status Linked Edit
PR 1540 merged vstinner, 2017-05-10 22:56
PR 1590 merged vstinner, 2017-05-15 09:27
Messages (32)
msg116367 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-09-14 05:21
The macpath module in the standard library purports to supply "the Mac OS 9 (and earlier) implementation of the os.path module.  It can be used to manipulate old-style Macintosh pathnames on Mac OS X (or any other platform).  The following functions are available in this module: normcase(), normpath(), isabs(), join(), split(), isdir(), isfile(), walk(), exists(). For other functions available in os.path dummy counterparts are available."

Old-style Mac pathnames are not further documented - in fact, the above quote is the entire external documentation for the module - but they are ones using colon separators, like ("MacHD:Users:nad:macpath_test:file"). These style path names were initially supported on Mac OS X by compatibility APIs for programs ported from classic Mac OS but those interfaces have long been deprecated in OS X and in most cases are not available in 64-bit execution mode.

Even if one did have a need to use the obsolete old-style paths, the macpath module is currently practically unusable for anything other than simple character manipulations of the path.  Nearly all of the functions that actually call OS routines are broken in one or more ways.

Those that do make file system calls are calling APIs that are expecting normal posix-style ('/' separated paths) incorrectly with old-style (":) paths (ispath, isdir, lexists, etc) which means they only give correct results in the trivial case of unqualified file or directory names, i.e. those in the working directory.

The islink() function confusingly is testing whether a path points to a Finder Alias file (not a symlink).  Unfortunately, the Carbon API used for that test does not exist in a 64-bit version and all of the Python Carbon modules were removed in Python 3.

$ arch -i386 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
1
$ arch -x86_64 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
False
$ python3.2 -c 'import macpath; print(macpath.islink("alias_to_file"))'
False

The underlying import errors are being masked by "try"'s:

$ arch -i386 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
$ arch -x86_64 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'ResolveAliasFile'
$ python3.2 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named Carbon.File

The realpath function is also broken in 2.7, calling Carbon.File.FSResolveAliasFile with a ":" separated path when it expects a "/" path, and is totally broken in 3.x (no Carbon modules).


While macpath *could* be repaired by proper documentation, fixing the mishmash of path types internally, and supplying C wrappers and/or alternative 64-bit APIs, it does not seem worth the effort as these style paths are seldom encountered in modern code.  Considering how broken the module currently is and given that it probably was simply overlooked in the Python 2 to 3 transition, I think a case could be made for immediate removal prior to Python 3.2 release and even for a 3.1.x maintenance release.  Short of that, it should be cleared documented as deprecated in 3.2, 3.1, and 2.7 along with warnings about broken functionality along with added DeprecationWarnings to the module.

I can write up patches to do that depending on what the consensus is.
msg116387 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-09-14 11:53
I'm -0 on removing the module wholesale because parts of OSX still use MacOS9 style paths (in particular some Carbon APIs and AppleScript).

I'm +1 on removing all functions that aren't path manipulation code, that is 'macpath.join' would stay while 'macpath.islink' would go, and fixing the documentation.

And to be complete: I'm -1 on fixing macpath.islink and other functions that access a life filesystem because the macpath module is only meant to be used for that on MacOS9 and that platform is no longer supported.  Path manipulation code is still useful.

BTW. The module cannot be removed from 2.7 and 3.1 due to API stability requirements.
msg116388 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-09-14 11:56
The solution may be different depending on Python version. I propose to keep macpath in Python 2.7, just because it's too late to change such thing in Python2. But we may mark macpath as deprecated, eg. "macpath will be removed in Python 3.2" (see above). I suppose that the same treatement can be used on Python 3.1 (maybe with a different warning type/message).

But for Python 3.2, I suggest to drop macpath. Said differently: drop Mac OS 9 support. If you really need Mac OS 9 support, continue to use Python 2.7. If I understood correctly, macpath is not used for os.path in Python 3.2, and it is broken (at least on 64 bits build).

But well, I am not a Mac programmer, so I don't know if Mac OS 9 is still commonly installed or not.
msg116389 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-09-14 12:00
MacOS9 is already unsupported, except for macpath all traces of OS9 support are gone with the possible exception of distutils (I removed traces of OS9 support code in 3.2 and those got restored when Tarek replaced distutils by the version from the 3.1 branch, I don't remember if I redid my OS9-removal work).

As I wrote earlier the path manipulation code (macpath.join, macpath.split etc.) are still useful because OS9-style paths are still used in some parts of OSX.
msg116517 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-09-16 09:13
Patches in progress.
msg116973 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-20 21:12
I agree with Ronald’s proposal.

(Re “I don't remember if I redid my OS9-removal work”: Looks like you didn’t.)
msg142310 - (view) Author: Oleg Oshmyan (chortos) Date: 2011-08-18 12:08
I fully agree with Ronald’s proposal. And for a start, here is a trivial patch that fixes macpath.join('', ...) [at the moment it just returns its last argument]. By the way, this fix is probably eligible for inclusion in Python 2.7 too.

I also have ‘implement macpath.relpath’ in my to-do list, although in practice this means nothing.
msg142365 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-18 16:44
Thanks for the patch.  Is the function already tested or does someone need to add a test?
msg142397 - (view) Author: Oleg Oshmyan (chortos) Date: 2011-08-18 18:41
It is already tested but here is a new version of the patch that expands the existing test to cover the situation that was broken (plus another one).
msg217238 - (view) Author: Jessica McKellar (jesstess) * Date: 2014-04-27 04:14
Thanks for writing up this issue, ned.deily, and thanks for providing a patch, chortos.

I couldn't find documentation clearly specifying what the correct behavior of macpath.join should be (i.e. what are the exact rules for leading and trailing colons), but comparing the old and updated behavior against the new tests, this patch does make the function's behavior more consistent.

* The patch passes `make patchcheck`.
* The full test suite passes with this patch.

=> commit review
msg217901 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-05 00:43
The patch appears fine but it really doesn't have anything to do with the gist of this issue.  The problem remains that much of macpath is fundamentally broken.  In the intervening years since the issue was opened, I contend that any need for OS 9 style paths (aka HFS paths) has diminished even further so I would still support starting in Python 3.5 the deprecation and removal process for this module.  But, as long as it remains, to address the current problems at a minimum a patch would be needed to remove or raise exceptions for all of the macpath functions that make file system calls and to update the documentation.
msg227687 - (view) Author: Roundup Robot (python-dev) Date: 2014-09-27 15:58
New changeset 2ae2ca9d2b66 by Serhiy Storchaka in branch '2.7':
Issue #9850: Fixed macpath.join() for empty first component.  Patch by
https://hg.python.org/cpython/rev/2ae2ca9d2b66

New changeset 54987723de99 by Serhiy Storchaka in branch '3.4':
Issue #9850: Fixed macpath.join() for empty first component.  Patch by
https://hg.python.org/cpython/rev/54987723de99

New changeset e29866cb6b98 by Serhiy Storchaka in branch 'default':
Issue #9850: Fixed macpath.join() for empty first component.  Patch by
https://hg.python.org/cpython/rev/e29866cb6b98
msg227689 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-09-27 16:01
Committed macpath_join_fix_with_test.patch with some additional tests. Thank you for your contribution Oleg.

Needed a patch for deprecating all module or some functions.
msg247128 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-22 15:08
I think it is by now safe to remove macpath, AFAIK there is no real use-case anymore for having classic MacOS9 paths on any recentish version of OSX.]

I'm setting the version to 3.6 because it is too late to do this for Python 3.5, but it can be done for 3.6.
msg247242 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-07-24 03:47
In third-party code the path manipulating functions can be used in unexpected manner than doesn't relate to paths on MacOS. Also "import macpath" can be left even with the use of macpath was eliminated or are dead code. There is a chance to break third-party code when just remove macpath without deprecation period.
msg277844 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-02 02:18
It's a bit late in the 3.6 cycle to be removing macpath, but it's not too late to mark it in 3.6 as deprecated and to be removed in 3.7.  If someone wants to write two patches, one for 3.6 to add a deprecation warning to the code and to the docs, the other for 3.7 to actually remove all traces of macpath, that would be great!
msg277892 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-10-02 14:07
Here are my tries. There are more MacOS-specific module names in Tools/freeze/freeze.py. They are unrelated to macpath so I leave them to the next bug.
msg277893 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-10-02 14:20
Oops forgot to commit deleted files in the 3.7 patch.
msg278102 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2016-10-05 05:41
I'm just reading PEP-4 now for procedure of declaring a module deprecated.
Should PEP-4 page be updated with the proposal / info for macpath deprecation?

https://www.python.org/dev/peps/pep-0004/

Thanks.
msg293443 - (view) Author: Jim Fasarakis-Hilliard (Jim Fasarakis-Hilliard) * Date: 2017-05-10 17:26
Bump, any update on what to do with this issue?
msg293458 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-10 22:59
> Bump, any update on what to do with this issue?

I updated and completed Chi Hsuan Yen's 3.6-deprecate-macpath.patch to create a pull request. I suggest to start by deprecating macpath in 3.7, and remove it from Python 3.8.

This module is here for years. macOS 9 is now dead, all macOS users migrated to OS X nowadays. I don't think that there is an urgency to *remove* macpath.

The title says "obsolete macpath module dangerously broken": is "dangerous" still up to date after the following fix?

New changeset 54987723de99 by Serhiy Storchaka in branch '3.4':
Issue #9850: Fixed macpath.join() for empty first component.  Patch by
https://hg.python.org/cpython/rev/54987723de99

If yes, we may also warn users in the macpath documentation that the module is dangerous, especially in the documentation of Python < 3.7.
msg293692 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-15 09:01
New changeset 89a1c93f046f9726310f8362227be7b8e50eea22 by Victor Stinner in branch 'master':
bpo-9850: Deprecate the macpath module (#1540)
https://github.com/python/cpython/commit/89a1c93f046f9726310f8362227be7b8e50eea22
msg293693 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-15 09:02
> Bump, any update on what to do with this issue?

I just deprecated the macpath module in Python 3.7. It will be removed from Python 3.8.
msg293694 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-15 09:04
You can use test.support.import_module(deprecated=True) for importing deprecated module in tests.
msg293695 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-15 09:11
And please add a Misc/NEWS entry.
msg293696 - (view) Author: Jim Fasarakis-Hilliard (Jim Fasarakis-Hilliard) * Date: 2017-05-15 09:19
Great! Also, as Mariatta mentioned in a previous message, shouldn't an entry in PEP-4 be made?
msg293697 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-15 09:38
Serhiy Storchaka: "And please add a Misc/NEWS entry."

I created https://github.com/python/cpython/pull/1590


Serhiy: "You can use test.support.import_module(deprecated=True) for importing deprecated module in tests."

I looked at this function, but I don't want to skip the test if the module cannot be imported.


Jim Fasarakis-Hilliard: "Great! Also, as Mariatta mentioned in a previous message, shouldn't an entry in PEP-4 be made?"

I created https://github.com/python/peps/pull/257

I didn't know the PEP 4.
msg293698 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-15 09:39
New changeset d812eb731d886065bdd9bc94a3f0e5dfdcd671a4 by Victor Stinner in branch 'master':
bpo-9850: Document macpath deprecation in Misc/NEWS (#1590)
https://github.com/python/cpython/commit/d812eb731d886065bdd9bc94a3f0e5dfdcd671a4
msg293700 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-15 09:53
> I looked at this function, but I don't want to skip the test if the module cannot be imported.

You can pass required_on=('',).
msg293704 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-15 10:36
>> I looked at this function, but I don't want to skip the test if the module cannot be imported.
>
> You can pass required_on=('',).

('',) looks mysterious to me. I prefer my current code.

If you consider that an helper is required, I would prefer a more explicit option on import_module(), like always_raise=True or something like that?
msg293706 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-15 10:40
No, the current code LGTM.
msg293785 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-16 20:49
Ok, I don't see any open task anymore, so I close this issue.

The macpath may or may not be removed in Python 3.8, so not before 2018-01-29. IMHO it's ok to remove the module, macpath was used for MacOS 9, but all macOS users are now running macOS X. Otherwise, you can simply use an older Python version (like the future Python 3.7 ;-)). We can discuss the effective macpath removal later, when master will become Python 3.8.
History
Date User Action Args
2017-05-16 20:49:57vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg293785

stage: needs patch -> resolved
2017-05-15 10:40:34serhiy.storchakasetmessages: + msg293706
2017-05-15 10:36:49vstinnersetmessages: + msg293704
2017-05-15 09:53:54serhiy.storchakasetmessages: + msg293700
2017-05-15 09:39:12vstinnersetmessages: + msg293698
2017-05-15 09:38:45vstinnersetmessages: + msg293697
2017-05-15 09:27:34vstinnersetpull_requests: + pull_request1683
2017-05-15 09:19:57Jim Fasarakis-Hilliardsetmessages: + msg293696
2017-05-15 09:11:34serhiy.storchakasetmessages: + msg293695
2017-05-15 09:04:50serhiy.storchakasetmessages: + msg293694
versions: - Python 3.6
2017-05-15 09:02:06vstinnersetmessages: + msg293693
2017-05-15 09:01:23vstinnersetmessages: + msg293692
2017-05-10 22:59:45vstinnersetmessages: + msg293458
2017-05-10 22:56:34vstinnersetpull_requests: + pull_request1639
2017-05-10 17:26:19Jim Fasarakis-Hilliardsetnosy: + Jim Fasarakis-Hilliard
messages: + msg293443
2016-10-05 05:41:35Mariattasetmessages: + msg278102
2016-10-02 14:20:48Chi Hsuan Yensetfiles: + 3.7-remove-macpath_ver2.patch

messages: + msg277893
2016-10-02 14:07:11Chi Hsuan Yensetfiles: + 3.7-remove-macpath.patch

messages: + msg277892
2016-10-02 14:04:41Chi Hsuan Yensetfiles: + 3.6-deprecate-macpath.patch
2016-10-02 03:09:51Mariattasetnosy: + Mariatta
2016-10-02 02:37:50berker.peksagsetkeywords: + easy, - needs review
2016-10-02 02:18:14ned.deilysetassignee: ronaldoussoren ->
type: enhancement ->
messages: + msg277844
versions: + Python 3.7
2016-10-01 10:34:57Chi Hsuan Yensetnosy: + Chi Hsuan Yen
2016-08-21 00:10:25macpherson.scottsetnosy: + macpherson.scott
2015-07-24 03:47:04serhiy.storchakasetmessages: + msg247242
2015-07-22 15:08:06ronaldoussorensetmessages: + msg247128
versions: + Python 3.6, - Python 3.5
2014-09-27 16:01:28serhiy.storchakasetversions: - Python 2.7
nosy: + serhiy.storchaka

messages: + msg227689

type: behavior -> enhancement
stage: commit review -> needs patch
2014-09-27 15:58:41python-devsetnosy: + python-dev
messages: + msg227687
2014-05-05 00:43:31ned.deilysetmessages: + msg217901
2014-04-27 04:14:44jesstesssetversions: + Python 2.7, Python 3.5, - Python 3.3
nosy: + jesstess

messages: + msg217238

keywords: + needs review
stage: needs patch -> commit review
2011-08-18 18:41:19chortossetfiles: + macpath_join_fix_with_test.patch

messages: + msg142397
2011-08-18 16:44:49eric.araujosetmessages: + msg142365
2011-08-18 12:08:50chortossetfiles: + macpath_join_fix.patch

nosy: + chortos
messages: + msg142310

keywords: + patch
2011-03-11 02:09:38eric.araujosetnosy: ronaldoussoren, vstinner, ned.deily, eric.araujo
versions: + Python 3.3, - Python 3.1, Python 2.7, Python 3.2
2010-09-20 21:12:27eric.araujosetnosy: + eric.araujo
messages: + msg116973
2010-09-16 09:13:43ned.deilysetmessages: + msg116517
2010-09-14 12:00:38ronaldoussorensetmessages: + msg116389
2010-09-14 11:56:18vstinnersetnosy: + vstinner
messages: + msg116388
2010-09-14 11:53:47ronaldoussorensetmessages: + msg116387
2010-09-14 05:21:56ned.deilysetmessages: - msg116365
2010-09-14 05:21:47ned.deilysetmessages: + msg116367
2010-09-14 05:14:10ned.deilycreate