Problem with Quotes while using variables with "&" and "%"-CodePudding

I've been sitting here for 3 days now trying to solve the following problem. I need a call function that I can pass a url as a parameter to. However, this URL has an &, which no longer works as soon as I pass the variable as a call parameter. I can't set /& in the Url, because its a dynamic Url.

Every url begins with https:// and ends with &dl=1

@echo off
setlocal enabledelayedexpansion

set "url=!array[2]!"

call :do_download
   
:do_download 
    powershell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; 
       Invoke-WebRequest '%url%' -OutFile 'C:\Program Files (x86)\test.zip'"

But if i try in all versions of quotes, doublequotes, with trimm or other tricks, it dont work :´( I have rly no idea yet. But i need the Url as call-param.

@echo off
setlocal enabledelayedexpansion

set "url=!array[2]!"

call :do_download "%url%"

:do_download 
    powershell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; 
       Invoke-WebRequest '%1' -OutFile 'C:\Program Files (x86)\test.zip'"

If you try it self: you can save a packs.txt with:

Testlink
https://sync.luckycloud.de/d/76ffff3d76ea4e9ab15e/files/?p=/Test.txt&dl=1

The link is to download a .txt file with 200 words from lorem ipsum. and here is the complete code, that will work at this moment, but without call-param. you can set the paths whatever you want, it's only for example. This code runs by itself; the ! are used because later it will be inside an if, but that code is independent of this one.

@echo off
setlocal enabledelayedexpansion

call :readtest
pause
call :do_download
pause

:readtest
    set count=0
    for /f "usebackq tokens=*" %%A in ("C:\Program Files (x86)\Steam\steamapps\common\packs.txt") do (
        set /a count =1
        set url[!count!]=%%A
    )
    set "pack_sd=!url[2]!"
    goto :eof

:do_download 
powershell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest '%pack_sd%' -OutFile 'C:\Program Files (x86)\Steam\steamapps\common\test.txt'"
goto :eof

If i try the answer with use: :"=

call :do_download "%pack_sd%"
    
:do_download 
    echo TT %1 TT
    set "url=%1"
    set test=%url:"=%
    echo "TT %test% TT"

first echo:

TT "https://sync.luckycloud.de/d/76ffff3d76ea4e9ab15e/files/?p=FTest.txt&dl=1" TT

second echo:

error: command "dl" is wrong or unknown...
"TT "= TT"

If i try the answer with %~1

call :do_download "%pack_sd%"

:do_download 
    echo TT %1 TT
    set url=%~1
    echo "TT %url% TT%

first echo:

TT "https://sync.luckycloud.de/d/76ffff3d76ea4e9ab15e/files/?p=FTest.txt&dl=1" TT

second echo:

error: command "dl" is wrong or unknown...
"TT https://sync.luckycloud.de/d/76ffff3d76ea4e9ab15e/files/?p=FTest.txt TT"

if i use the answer with: '%1:"=' inside powershell command:

call :do_download "%pack_sd%"

:do_download 
        powershell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest '%1:"=' -OutFile 'C:\Program Files (x86)\Steam\steamapps\common\test.txt'"
    goto :eof 

error: "The string has no terminator: '."

CodePudding user response：

The problem boils down to this:

The requirement to support not just & but also % characters in your URLs significantly complicates the solution, unfortunately, because cmd.exe (inappropriately) subjects a parameter's verbatim value passed to a subroutine via call to string interpolation, which necessitates doubling the % in the value in order to preserve them.
- Thus, a URL read from a file with for /f must have its % programmatically doubled before passing it as a parameter to call :do_download, which is a nontrivial undertaking - see the code below and the source-code comments.
- Note that additional effort would be required if your URL also contained ! characters.
In your :do_download subroutine, use set "url=%~1". This stores the unquoted value of %1 (the first parameter) in variable %url%
- Do not use set "url=%1" or set url=%~1, both of which predictably break with a %1 value that has embedded surrounding double quotes and contains &.
- Do not use %1:"=, which doesn't do what you think it does. %1 is a parameter, whereas the string-substitution syntax you're trying to use only works with variables (e.g., %url:"=%)
Then you can use %url% as-is inside the "..." string that forms the powershell -c argument - or you could just use %~1 directly.
- With the intermediate %url% variable, however, you could alternatively take advantage of the fact that all cmd.exe variables (but not parameters such as %1) are also environment variables, which therefore allows PowerShell to access the URL via an environment-variable reference instead of embedding it directly in the command string (the equivalent of cmd.exe's %url% is $env:url in PowerShell). That is, you could replace the powershell call below with the following - note how enclosing '...' quoting as part of the PowerShell command must then not be used:
  - powershell -c "Write-Output $env:url"

Here's a minimal example that shows that a URL containing both & and % characters is correctly passed through to PowerShell; everything else in your code is incidental (use of for /f, specifics of the PowerShell command):

@echo off
setlocal enabledelayedexpansion

:: Use a sample URL that contains both & and % characters.
:: Note: The % must be DOUBLED here in order to escape it.
::       The resulting string will only have ONE %, as desired and will therefore
::       literally be the following, as it would be read from a file with `for /f`: 
::          http://example.org?p=/Test.txt&dl=1
set "url=http://example.org?p=%/Test.txt&dl=1"

:: Unfortunately, using `call` requires escaping of % even in variable values, 
:: as the embedded single % are then again subject to variable expansion
:: (even though they shouldn't be).
:: Doubling % *programmatically* is non-trivial and requires a helper variable
:: as well as mixing up-front and delayed expansion.
:: The following string substitution effectively doubles the embedded % chars.
:: so that the :do_download subroutine again sees them as verbatim single ones.

set "pct=%%"
call :do_download "!url:%pct%=%%%%!"

goto :eof

:do_download
  set "url=%~1"
  powershell -c "Write-Output '%url%'"
goto :eof

The above prints verbatim http://example.org?p=/Test.txt&dl=1, proving that the URL was correctly passed through to PowerShell.

CodePudding user response：

Does this work?

@echo off
setlocal enabledelayedexpansion

set "url=!array[2]!"

call :do_download
   
:do_download 
    powershell -c "$Url=$Env:url; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; 
       Invoke-WebRequest $Url -OutFile 'C:\Program Files (x86)\test.zip'"

You have Delayed Explansion enabled, so you should be able to just past the variable's name and expand it later.

Does this also work?

@echo off
setlocal enabledelayedexpansion

set "url=!array[2]!"

call :do_download url
GOTO :EOF

:do_download 
    powershell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; 
       Invoke-WebRequest $Env:!%~1! -OutFile 'C:\Program Files (x86)\test.zip'"

EDIT:

I'm not sure if the packs.txt file contains only URLs or other content. I placed 5 copies of the sync.luckycloud.de URL, the one mentioned in the question, in a mock packs.txt file and was able to get this code to work.

@ECHO OFF
SETLOCAL EnableDelayedExpansion
GOTO :Start

:ReadTest
    SET Count=0
    FOR /F "USEBACKQ EOL=` TOKENS=* DELIMS=`" %%U IN ("C:\Program Files (x86)\Steam\steamapps\common\packs.txt") DO (
        SET /A Count =1
        SET url[!count!]=%%U
    )
    SET "pack_sd=!url[2]!"
GOTO :EOF

:Do_Download
    PowerShell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest $Env:pack_sd -OutFile 'C:\Program Files (x86)\Steam\steamapps\common\test.txt'"
GOTO :EOF

:Start
    CALL :ReadTest
    CALL :Do_Download
GOTO :EOF

NOTE: I believe the characters below are not allowed as part of a URL. The FOR command first looks for DELIMS, but there should never be a back tick in the URL, so DELIMS should never be found. But if one is found, it is grabbed by DELIMS before EOL, this make EOL disabled. The default for EOL is a semicolon, which appears to be a valid character in URLs. So it is important to disable EOL, or give it a value that will never be in a URL.

"%\^`{|}<>

Final Generic Version:

ReadFile: Takes name of array for saving URLs and the path of a file containing URLs. Reads file, storing URLs in array.

Do_Downloads: Takes name of array of URLs and the path where files will be saved. URLs from array are downloaded and saved to files with name WebFile{N}.txt, where {N} is the line number the URL came from in the file containing URLs.

@ECHO OFF
SETLOCAL EnableDelayedExpansion
GOTO :Start

:ReadFile
    SET Count=0
    FOR /F "USEBACKQ EOL=` TOKENS=* DELIMS=`" %%U IN ("%~2") DO (
        SET /A Count =1
        SET %~1[!Count!]=%%U
    )
GOTO :EOF

:Do_Downloads
    FOR /L %%I IN (1, 1, %Count%) DO (
        SET Url=!%~1[%%I]!
        PowerShell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest $Env:Url -OutFile '%~2\WebFile%%I.txt'"
    )
GOTO :EOF

:Start
    CALL :ReadFile Urls "C:\Program Files (x86)\Steam\steamapps\common\packs.txt"
    CALL :Do_Downloads Urls "C:\Program Files (x86)\Steam\steamapps\common"
GOTO :EOF