How to filter a big .log file on Windows?-CodePudding

I have a huge text-based log file, bigger than 1 GB. I'd like to filter it to create a new file with lines that contain only "error" messages.

I tried the following Powershell script:

(Get-Content -Path 'filename.log') -match 'error' | Set-Content -Path 'newFilename.log'

It works pretty good with a small files, but with this file it's just stuck...

CodePudding user response：

Perhaps even faster if you use switch:

$errorData = switch -Wildcard -File 'X:\PathTo\filename.log' {
    '*error*' { $_ }  # output the line if keyword error is found
}
$errorData | Set-Content -Path 'X:\PathTo\newFilename.log'

Instead of using -Wildcard, you could also use -Regex. In that case, output the line on this condition: 'error' { $_ }

CodePudding user response：

To complement Theo's helpful answer, here is another memory friendly and fast alternative using the StreamReader and StreamWriter .NET APIs to read and write line-by-line:

try {
    $reader = [System.IO.StreamReader] 'path\to\filename.log'
    $writer = [System.IO.StreamWriter] 'path\to\newFilename.log'

    while(-not $reader.EndOfStream) {
        $line = $reader.ReadLine()
        if($line -match 'Error') {
            $writer.WriteLine($line)
        }
    }
}
finally {
    $reader, $writer | ForEach-Object Dispose
}