PowerShell 7.3.0 breaking command invocation-CodePudding

I use WinSCP within a Powershell script. It suddenly stopped working. After a while I could figure out that the problem appeared from a more recent version of PowerShell:

Reduced code:

& winscp `
    /log `
    /command `
        'echo Connecting...' `
        "open sftp://kjhgk:[email protected]/ -hostkey=`"`"ssh-ed25519 includes spaces`"`""

Error message using v7.2.7

Host "lkjhlk.com" does not exist.

Errror message using v7.3.0

Too many parameters for command 'open'.

As you can see with v7.3.0 WinSCP receives different input depending on the version of PS. I found out that the difference has something to do with the spaces in the hostkey. If they are omitted v7.3.0 outputs the same error.

What change to PowerShell caused this, and how can I fix it? (How can I debug such issues? I played a bit around with escaping, but the strings look the same no matter the version, no obvious breaking change that could be responsible)

CodePudding user response：

Version 7.3 of PowerShell (Core) introduced a breaking change with respect to how arguments with embedded " characters are passed to external programs, such as winscp:

While this change is mostly beneficial, because it fixes behavior that was fundamentally broken since v1 (this answer discusses the old, broken behavior), it also invariably breaks existing workarounds that build on the broken behavior, except those for calls to batch files and the WSH CLIs (wscript.exe and cscript.exe) and their associated script files (with file-name extensions such as .vbs and .js).

To make existing workarounds continue to work, set the $PSNativeCommandArgumentPassing preference variable (temporarily) to 'Legacy':

# Note: Enclosing the call in & { ... } makes it execute in a *child scope*
#       limiting the change to $PSNativeCommandArgumentPassing to that scope.
& {
  $PSNativeCommandArgumentPassing = 'Legacy'
  & winscp `
    /log `
    /command `
        'echo Connecting...' `
        "open sftp://kjhgk:[email protected]/ -hostkey=`"`"ssh-ed25519 includes spaces`"`"" 
}

Unfortunately, because wincp.exe only accepts
"open sftp://kjhgk:[email protected]/ -hostkey=""ssh-ed25519 includes spaces""" on its process command line (i.e., embedded " escaped as ""), and not also the most widely used form
"open sftp://kjhgk:[email protected]/ -hostkey=\"ssh-ed25519 includes spaces\"" (embedded " escaped as \"), which the fixed behavior now employs, for winscp.exe, specifically, a workaround will continue to be required.

If you don't want to rely on having to modify $PSNativeCommandArgumentPassing for the workaround, here are workarounds that function in both v7.2- and v7.3 :

Use --%, the stop-parsing token, which, however, comes with pitfalls and severe limitations, notably the inability to (directly) use PowerShell variables or subexpressions in the arguments that follow it - see this answer for details:

# Note: Must be single-line; note the --% and the 
#       unescaped use of "" in the argument that follows it.
#       Only "..." quoting must be used after --% 
#       and the only variable that can be used are cmd-style 
#       *environment variables* such as %OS%.
winscp /log /command 'echo Connecting...' --% "open sftp://kjhgk:[email protected]/ -hostkey=""ssh-ed25519 includes spaces"""

Preferably, call via cmd /c:

# Note: Pass-through command must be single-line,
#       Only "..." quoting supported, 
#       and the embedded command must obey cmd.exe's syntax rules.
cmd /c @"
  winscp /log /command "echo Connecting..." "open sftp://kjhgk:[email protected]/ -hostkey=""ssh-ed25519 includes spaces"""
"@

Note: You don't strictly need to use a here-string (@"<newline>...<newline>"@ or @'<newline>...<newline>'@), but it helps readability and simplifies using embedded quoting.

Both workarounds allow you to pass arguments directly as quoted, but unfortunately also require formulating the entire (pass-through) command on a single line.

Background information:

The v7.3 default $PSNativeCommandArgumentPassing value on Windows, 'Windows':

regrettably retains the old, broken behavior for calls to batch files and the WSH CLIs (wscript.exe and cscript.exe) and their associated script files (with file-name extensions such as .vbs and .js).
- While, for these programs only, this allows existing workarounds to continue to function, future code that only needs to run in v7.3 will continue to be burdened by the need for these obscure workarounds, which build on broken behavior.
  - The alternative, which was not implemented, would have been to build accommodations for these programs as well as some program-agnostic accommodations into PowerShell, so that in the vast majority of case there won't even be a need for workarounds in the future: see GitHub issue #15143.
- There are also troublesome signs that this list of exceptions will be appended to, piecemeal, which all but guarantees confusion for a given PowerShell version as to which programs require workarounds and which don't.
commendably, for all other programs, makes PowerShell encode the arguments when it - of necessity - rebuilds the command line behind the scenes as follows with respect to ":
- It encodes the arguments for programs that follow the C command-line parsing rules (as used by C / C / .NET applications) / the parsing rules of the CommandLineToArgv WinAPI function, which are the most widely observed convention for parsing a process' command line.
- In a nutshell, this means that embedded " characters embedded in an argument, to be seen as a verbatim part of it by the target program, are escaped as \", with \ itself requiring escaping only (as \\) if it precedes a " but is meant to be interpreted verbatim.
- Note that if you set $PSNativeCommandArgumentPassing value to 'Standard' (which is the default on Unix-like platforms, where this mode fixes all problems and makes v7.3 code never require workarounds), this behavior applies to all external programs, i.e. the above exceptions no longer apply).

For a summary of the impact of the breaking v7.3 change, see this comment on GitHub.

If you have / need to write cross-edition, cross-version PowerShell code: The Native module (Install-Module Native; authored by me), has an ie function (short for: Invoke Executable), is a polyfill that provides workaround-free cross-edition (v3 ), cross-platform, and cross-version behavior in the vast majority of cases - simply prepend ie to your external-program calls.
Caveat: In the specific case at hand it will not work, because it isn't aware that winscp.exe requires ""-escaping.