classification
Title: python not correctly processing double quotes in command line args in windows
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.9
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, paul.moore, steve.dower, tim.golden, vikram.invincible.pal4, zach.ware
Priority: normal Keywords:

Created on 2020-11-29 05:33 by vikram.invincible.pal4, last changed 2020-11-29 07:28 by eryksun. This issue is now closed.

Messages (2)
msg382039 - (view) Author: Vikram Pal (vikram.invincible.pal4) Date: 2020-11-29 05:33
Using PowerShell in Windows, double quotes in argument given with "-c" flag to python does not work. E.g.

> Write-Host 's = "fn main() {"; s += "\n".join("let x = 0;" for _ in range(2)); s+= "}"; print(s)'
s = "fn main() {"; s += "\n".join("let x = 0;" for _ in range(6000)); s+= "}"; print(s)

> python -c 's = "fn main() {"; s += "\n".join("let x = 0;" for _ in range(2)); s+= "}"; print(s)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'fn' is not defined
msg382041 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-11-29 07:28
Python uses the Windows C runtime to parse the command line into an argument array. By the CRT rules [1], you need to escape the embedded double quotes using backslashes. For example:

    PS C:\> python -c 's = \"fn main() {\"; s += \"\n\".join(\"let x = 0;\" for _ in range(2)); s+= \"}\"; print(s)'
    fn main() {let x = 0;
    let x = 0;}

In case you don't know already, PowerShell magically converts the single quotes in your command line to double quotes. It recognizes that most Windows programs parse the command line using the C runtime or WinAPI CommandLineToArgvW, which doesn't have any special handling for single quotes.

You can use ctypes to see how your original command line gets parsed via CommandLineToArgvW, which is functionally the same as what the C runtime does. For example (note the -i option to enter interactive mode):

    PS C:\> python -ic 's = "fn main() {"; s += "\n".join("let x = 0;" for _ in range(2)); s+= "}"; print(s)'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    NameError: name 'fn' is not defined

    >>> import ctypes
    >>> kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
    >>> shell32 = ctypes.WinDLL('shell32', use_last_error=True)
    >>> kernel32.GetCommandLineW.restype = ctypes.c_wchar_p
    >>> shell32.CommandLineToArgvW.restype = ctypes.POINTER(ctypes.c_wchar_p)

    >>> cmd = kernel32.GetCommandLineW()
    >>> num = ctypes.c_int()
    >>> args = shell32.CommandLineToArgvW(cmd, ctypes.byref(num))

    >>> print(*zip(range(num.value), args[:num.value]), sep='\n')
    (0, 'C:\\Program Files\\Python39\\python.exe')
    (1, '-ic')
    (2, 's = fn')
    (3, 'main()')
    (4, '{; s += \\n.join(let')
    (5, 'x')
    (6, '=')
    (7, '0; for _ in range(2)); s+= }; print(s)')

As you can see, because the double quotes aren't escaped, they get consumed, combined with the rule for white space, to parse the command line after "-ic" into six arguments instead of just one argument.

---

[1] https://docs.microsoft.com/en-us/cpp/cpp/main-function-command-line-args?view=msvc-160#parsing-c-command-line-arguments
History
Date User Action Args
2020-11-29 07:28:08eryksunsetstatus: open -> closed

nosy: + eryksun
messages: + msg382041

resolution: not a bug
stage: resolved
2020-11-29 05:33:58vikram.invincible.pal4create