Implementing Multi-threading/Parallel processing in script-CodePudding

So I have this simple script that allows me to copy and paste items in a different folder while also changing their name in Windows Powershell.

$destfolder = '.\destfolder\'
get-childitem -Recurse -File .\srcfolder\ | Foreach-Object {
    $newname = $_.name
$newname = $_.directory.parent.name   '-'   $_.directory.name   '-'   $_.name
Copy-Item -Path $_.FullName -Destination (Join-Path $destfolder $newname) -Verbose
}

I'm wondering how to add multi-threading/parallel processing to make it go faster. And maybe robocopy.

I tried to use the object parallel feature but it didn't work.

-I checked and it seems like the reason why Foreach-Object -Parallel didn't work is cause I'm running powershell version 5.1.22621.608

-I want to copy files from multiple subdirectories in a folder at once into a single folder without the directories, however robocopy doesn't allow for this

CodePudding user response：

First, you can speed up your command even in a single thread, by directly piping to Copy-Item in combination with a delay-bind script block:

$destfolder = '.\destfolder\'
Get-ChildItem -Recurse -File .\srcfolder\ | 
  Copy-Item -Destination { 
    Join-Path $destfolder ($_.directory.parent.name, $_.directory.name, $_.name -join '-') 
  } -Verbose -WhatIf

^{Note: The -WhatIf common parameter in the command above previews the operation. Remove -WhatIf once you're sure the operation will do what you want.}

As for parallelism:

Windows PowerShell (as opposed to PowerShell (Core) 7 ) has only child-process-based parallelism built in, via Start-Job, which is both resource-intensive and slow.

However, the install-on-demand third-party Invoke-Parallel function that comes as part of the PSParallelPipeline module offers thread-based parallelism with pipeline support, similar to ForEach-Object -Parallel in PowerShell (Core) 7 .

You'll have to test yourself whether its use results in any performance improvement in your case, but here's how you would invoke it:

First, install the module:

# Install the module that has the Invoke-Parallel function.
# (Only needed once).
Install-Module PSParallelPipeline -Scope CurrentUser

Then use Invoke-Parallel to process up to 5 input files in parallel (you can modify this number with -ThrottleLimit):

$destfolder = '.\destfolder\'
Get-ChildItem -Recurse -File .\srcfolder\ |
  Invoke-Parallel {
    $_ | Copy-Item -Destination (
      Join-Path $using:destfolder ($_.directory.parent.name, $_.directory.name, $_.name -join '-') 
    ) -Verbose -WhatIf
  }