Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ensurepip/venv broken on Windows if path includes unicode #69944

Closed
dimaqq mannequin opened this issue Nov 28, 2015 · 6 comments
Closed

ensurepip/venv broken on Windows if path includes unicode #69944

dimaqq mannequin opened this issue Nov 28, 2015 · 6 comments

Comments

@dimaqq
Copy link
Mannequin

dimaqq mannequin commented Nov 28, 2015

BPO 25758
Nosy @brettcannon, @pfmoore, @ncoghlan, @vstinner, @tjguk, @ezio-melotti, @ericsnowcurrently, @dimaqq, @zware, @eryksun, @zooba, @dstufft, @jayvdb
Files
  • issue25758_1.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2016-09-10.00:35:46.571>
    created_at = <Date 2015-11-28.09:56:52.995>
    labels = ['expert-unicode', 'OS-windows']
    title = 'ensurepip/venv broken on Windows if path includes unicode'
    updated_at = <Date 2016-09-10.00:35:46.570>
    user = 'https://github.com/dimaqq'

    bugs.python.org fields:

    activity = <Date 2016-09-10.00:35:46.570>
    actor = 'steve.dower'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2016-09-10.00:35:46.571>
    closer = 'steve.dower'
    components = ['Unicode', 'Windows']
    creation = <Date 2015-11-28.09:56:52.995>
    creator = 'Dima.Tisnek'
    dependencies = []
    files = ['41185']
    hgrepos = []
    issue_num = 25758
    keywords = ['patch']
    message_count = 6.0
    messages = ['255534', '255544', '274095', '274382', '275511', '275512']
    nosy_count = 15.0
    nosy_names = ['brett.cannon', 'paul.moore', 'ncoghlan', 'vstinner', 'tim.golden', 'ezio.melotti', 'python-dev', 'eric.snow', 'Dima.Tisnek', 'zach.ware', 'eryksun', 'steve.dower', 'dstufft', 'Marcus.Smith', 'jayvdb']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue25758'
    versions = ['Python 3.5', 'Python 3.6']

    @dimaqq
    Copy link
    Mannequin Author

    dimaqq mannequin commented Nov 28, 2015

    One of my students installed Python 3.5 on Windows 10 to default location where user name "Łukasz" contains unicode.

    Now "-m venv" and "-m ensurepip" do not work:

    C:\Users\Łukasz>C:\Users\Łukasz\AppData\Local\Programs\Python\Python35-32\python.exe -m venv workshops
    Error: Command '['C:\\Users\\\u0141ukasz\\workshops\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

    C:\Users\Łukasz>C:\Users\Łukasz\AppData\Local\Programs\Python\Python35-32\python.exe -m ensurepip
    Traceback (most recent call last):
      File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 170, in _run_module_as_main
        "__main__", mod_spec)
      File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\ensurepip\__main__.py", line 4, in <module>
        ensurepip._main()
      File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\ensurepip\__init__.py", line 209, in _main
        default_pip=args.default_pip,
      File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\ensurepip\__init__.py", line 116, in bootstrap
        _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
      File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\ensurepip\__init__.py", line 40, in _run_pip
        import pip
      File "<frozen importlib._bootstrap>", line 969, in _find_and_load
      File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 896, in _find_spec
      File "<frozen importlib._bootstrap_external>", line 1136, in find_spec
      File "<frozen importlib._bootstrap_external>", line 1112, in _get_spec
      File "<frozen importlib._bootstrap_external>", line 1093, in _legacy_get_spec
      File "<frozen importlib._bootstrap>", line 444, in spec_from_loader
      File "<frozen importlib._bootstrap_external>", line 530, in spec_from_file_location
    UnicodeEncodeError: 'mbcs' codec can't encode characters in position 0--1: invalid character

    @dimaqq dimaqq mannequin added the topic-unicode label Nov 28, 2015
    @SilentGhost SilentGhost mannequin added the OS-windows label Nov 28, 2015
    @eryksun
    Copy link
    Contributor

    eryksun commented Nov 28, 2015

    The problem is that the compile_source function in Modules/zipimport.c calls PyUnicode_EncodeFSDefault to get an encoded string to pass as the filename argument of Py_CompileString. On Windows this uses the ANSI codepage (i.e. 'mbcs'). Apparently your system's ANSI codepage doesn't map the "Ł" character.

    I reproduced the problem more simply by copying pip-7.1.2-py2.py3-none-any.whl to a subdirectory named "Łukasz"; adding the wheel path to sys.path; and attempting to execute "import pip".

    One solution is to replace Py_CompileString with Py_CompileStringObject. This way compile_source doesn't have to worry about encoding its pathname argument. A minimal patch is attached, but it needs a test.

    @ncoghlan
    Copy link
    Contributor

    ncoghlan commented Sep 1, 2016

    As a test case for handling non-ASCII characters in the name of the zipfile itself, I believe it should be sufficient to add 'Ł' to TEMP_DIR and TEMP_ZIP in https://hg.python.org/cpython/file/default/Lib/test/test_zipimport.py

    @zooba
    Copy link
    Member

    zooba commented Sep 5, 2016

    Given a quick read, it looks like bpo-27781 (PEP-529) will resolve this?

    Not encoding the path at all is obviously better, but maybe I'll add this as supporting evidence to the PEP...

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 10, 2016

    New changeset 663a62bcf9c9 by Steve Dower in branch '3.5':
    Issue bpo-25758: Prevents zipimport from unnecessarily encoding a filename (patch by Eryk Sun)
    https://hg.python.org/cpython/rev/663a62bcf9c9

    New changeset ead30e7262d5 by Steve Dower in branch 'default':
    Issue bpo-25758: Prevents zipimport from unnecessarily encoding a filename (patch by Eryk Sun)
    https://hg.python.org/cpython/rev/ead30e7262d5

    @zooba
    Copy link
    Member

    zooba commented Sep 10, 2016

    Applied Eryk's patch and updated the test to repro it (though it wouldn't repro on 3.6 with PEP-529 applied, but it definitely did on 3.5).

    @zooba zooba closed this as completed Sep 10, 2016
    @zooba zooba self-assigned this Sep 10, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants