Counting within counting , within counting-CodePudding

I am trying to turn a tedious task of looking through an error log folder and manually looking into 10's of files individually to understand what they are for further diagnosis. At the moment I have a working powershell script which does the steps below but still requires me to read each line to find the date and the error type within the file:

finds files in a directory ending in .txt
collects just the date and time from the filename (they are always in the same syntax e.g. Error Log 2021-11-25 21-22-14.txt)
counts the total amount of files
Loops through the files
reads the first two lines of the file and if the firstlines are like a string, add to a count specific to the string
Writes to the console total counted files and total count per string found

The count of EOF is: 8
The count of Handshake fails is: 7
The count of SSPI fails is: 2
The count of Transport Connection is: 36
The count of Authentication failed is: 4
The count of Object Reference is: 3
Total files in this folder: 60
Total files analysed: 60

writes the file name and the string found within the file to the console.

2021-12-01 22-06-57.txt - contains an 'Transport Connection' error
2021-12-01 20-15-19.txt - contains an 'Authentication failed' error
2021-11-26 14-02-05.txt - contains an 'Unexpected EOF' error.
2021-11-25 21-22-14 - contains an 'SSPI Fail' error.

I then need to transfer the number of occurrences per date to another excel sheet on another server (this one is air-gapped) As you can see below from the entire script it's quite inefficient and I would like to output the following style. I have tried adding my error names to a list and looping through the error names to the files, but really struggling with the right syntax. I've also tried grouping my files into the date names then per error message, but also really struggling with the syntax.

It does what I want for the most part, but what I hoping for is something along these lines.

On 2021-12-01 I count 2 occurrence(s) in total:
    'Transport Connection' - 1
    'Authentication failed' - 1
    'Unexpected EOF' - 2
On 2021-11-25 I count 1 occurrence(s) in total:
    'SSPI Fail' - 1
On 2021-11-26 I count 1 occurrence(s) in total:
    'Unexpected EOF' - 1

Any help would be greatly appreciated! Thank you so much! Please be kind and don't jump down my throat. Too many people have done this and it doesn't bode well for me learning.

Entire Code

$path = 'D:\logs'
$files = Get-ChildItem -Path $path -Include *.txt
$filecount = (Get-ChildItem -path $path | Measure-Object).Count

$countEOF = 0
$countHandshakeFail = 0 
$countSSPI = 0
$countTrnsptConn = 0 
$ccountAuthFail = 0
$countObjRef = 0
$countCertInval = 0 
$countUnauthdAccess = 0
$countArithmetic = 0



foreach($file in $files)
    {
        $filename = Split-Path $file -leaf
        
        $firstLines = Get-Content $file | Select -First 2

        $date = $filename.Substring($filename.IndexOf('.txt')-19,19)      

        if($firstLines -like '*Certificate*')
            {
                $countObjRef  = 1
                Write-Output "$date - contains an 'Object reference' error."    
            }
        if($firstLines -like '*EOF*')
            {
                $countEOF  = 1
                Write-Output "$date - contains an 'Unexpected EOF' error."    
            }
        if($firstLines -like '*handshake*')
            {
                $countHandshakeFail  = 1
                Write-Output "$date - contains an 'Handshake fail due to unexpected packet format' error."
            }
        if($firstLines -like '*SSPI*')
            {
                $countSSPI  = 1
                Write-Output "$date - contains an 'SSPI Fail' errror."
            }
        if($firstLines -like '*transport connection*')
            {
                $countTrnsptConn  = 1
                Write-Output "$date - contains an 'Transport Connection' error"
            }
        if($firstLines -like '*Authentication failed*')
            {
                $ccountAuthFail  = 1
                Write-Output "$date - contains an 'Authentication failed' error"
            }
        if($firstLines -like '*Object reference*')
            {
                $countObjRef  = 1
                Write-Output "$date - contains an 'Object reference' error."    
            }

    }
Write-Output "The count of EOF is: $countEOF"
Write-Output "The count of Handshake fails is: $countHandshakeFail"
Write-Output "The count of SSPI fails is: $countSSPI"
Write-Output "The count of Transport Connection is: $countTrnsptConn"
Write-Output "The count of Authentication failed is: $ccountAuthFail"
Write-Output "The count of Object Reference is: $countObjRef"

Write-Output "Total files in this folder: $filecount"
write-output ("Total files analysed: "   ($countEOF   $countSSPI   $countHandshakeFail   $countTrnsptConn   $ccountAuthFail   $countObjRef))

CodePudding user response：

The way I'd approach that is this:

Define the error types first
Loop through all files and look for issues, iterating through the predefined error types earlier
Store the result to output for easier further processing
Write the output you want based on the stored result, which can now be grouped and sorted at will.

Based on that, here is what the result would look like:

$path = 'D:\logs\*'
$files = Get-ChildItem -Path $path -Include *.txt
$filecount = (Get-ChildItem -path $path | Measure-Object).Count

# List of results for easy processing afterward (eg: such as export to excel)
$Output = [System.Collections.Generic.List[PSObject]]::new()

# Ordered hashtable containing an internal name for each error type and the corresponding string that will be searched against using the like operator / written to console
$ErrorTypes = [Ordered]@{
    AuthenticationFailed = 'Authentication failed'
    Certificate          = 'Certificate'
    EOF                  = 'EOF'
    Handshake            = 'Handshake'
    ObjectReference      = 'Object Reference'
    SSPI                 = 'SSPI'
    TransportConnection  = 'transport connection'
}
# Just for better display, so that all the numbers get aligned in the output later on.
$ErrorTypesLongest = ($ErrorTypes.Values | Measure-Object -Maximum).Maximum.Length
foreach ($file in $files) {
    $filename = Split-Path $file -leaf
        
    $firstLines = Get-Content $file | Select -First 2

    $date = $filename.Substring($filename.IndexOf('.txt') - 19, 19)      
    
    # object containing reference information about Filename, date and errors
    $Item = [PSCustomObject]@{
        Date       = $date.Substring(0, 10)
        Filename   = $filename
        ErrorTypes = [System.Collections.Generic.List[String]]::new()
    }
    
    # Same as your multiple if but we iterate through errors types contained in the hashtable. If found,
    # we add the error to our "errortypes" for that file
    foreach ($ErrorKey in $ErrorTypes.Keys) {
        if ($firstLines -like "*$($ErrorTypes[$ErrorKey])*") {
            $Item.ErrorTypes.Add($ErrorKey)
            break # Since $firstLine, based on the last line of the initial sample, seems like it will contains only one error type.
        }
    }
    
    # That is the list we use to keep track of everything
    $Output.Add($Item)
}

# We do want everything sorted by date, then grouped by date so we can count the number of occurences and
# summarize what happened for that day.
$Grouped = $Output | Sort-Object -Property Date | Group-Object -Property Date 
Foreach ($G in $Grouped) {
    # Because we can
    $OccurenceStr = if ($G.Count -eq 1) { "occurence" } else { "occurences" }
    
    # I like cyan.
    Write-host "On $($G.Name), I count $($G.Count) $OccurenceStr in total:" -ForegroundColor Cyan

    # We group error types of the same date together to get a count and we sort the result alphabetically
    # then we iterate through them and write the output
    $G.Group.ErrorTypes | Group-Object | Sort-Object Name | % {
        Write-host " $($_.Name.PadRight($ErrorTypesLongest  1,' ')): $($_.Count)"     } 
}
Write-Host "Summary" -ForegroundColor Cyan
Write-Host " Total files in this folder: $filecount"
Write-Host " Total files with issues: $(($OutputErrorTypes | Measure-Object -Property Count -Sum).Sum)" 

$OutputErrorTypes = $Output.errortypes | Group-Object | Sort-Object Name
# If you want to avoid get the count for error types where the count is 0, use this.
$OutputErrorTypes | % {
    Write-Host " The count of $($_.Name) is: $($_.Count)"
}

# If you want the "0" count item, use that instead
# $ErrorTypes.Keys | % {
#     Write-Host "The count of $($ErrorTypes[$_]) is: $(($OutputErrorTypes | Where Name -eq $ErrorTypes[$_]).Count)"
# }

File names sample used & their 1 line Content

2021-12-01 22-06-57.txt - contains an 'Transport Connection' error
2021-12-01 22-07-57.txt - contains an 'Transport Connection' error
2021-12-01 20-15-19.txt - contains an 'Authentication failed' error
2021-11-26 14-02-05.txt - contains an 'Unexpected EOF' error.
2021-11-25 21-22-14 - contains an 'SSPI Fail' error.

Result

On 2021-11-25, I count 1 occurence in total:
 SSPI                 : 1
On 2021-11-26, I count 1 occurence in total:
 EOF                  : 1
On 2021-12-01, I count 3 occurences in total:
 AuthenticationFailed : 1
 TransportConnection  : 2
Summary
 Total files in this folder: 5
 Total files with issues: 5
 The count of AuthenticationFailed is: 1
 The count of EOF is: 1
 The count of SSPI is: 1
 The count of TransportConnection is: 2