Bash check if string appears more than X times in logs-CodePudding

I have a log file like:

135.148.41.73 - [11/Dec/2022:12:21:11  0100]  - "-" code=400 upstream="-" req=0.449 up=- ua="-"
135.148.41.73 - [11/Dec/2022:12:21:14  0100]  - "-" code=400 upstream="-" req=0.413 up=- ua="-"
135.148.41.73 - [11/Dec/2022:12:21:15  0100]  - "-" code=400 upstream="-" req=1.001 up=- ua="-"
135.148.41.73 - [11/Dec/2022:12:21:16  0100]  - "-" code=400 upstream="-" req=1.002 up=- ua="-"
45.87.212.41 - [11/Dec/2022:12:22:50  0100]  - "-" code=400 upstream="-" req=0.004 up=- ua="-"

I need to perform some action if IP appears more than X times on the logs.

Using awk '{print $1}' access.log | sort | uniq -c | sort -rn I can count them, but how do I perform action on it using bash?

Like:

For each IP that appears more than 5 times in access.log, do something.

Thanks.

CodePudding user response：

#!/usr/bin/env bash

while IFS= read -r ip; do
    some_command "$ip"
done < <(awk '  cnt[$1] == 5{print $1}' file)

Replace some_command with whatever it is you want to do with that IP.

CodePudding user response：

You are close. I just added filtering out IP's with count below 5, and running your_script on each output using xargs.

cat access.log | awk '{print $1'} | uniq -c | sort -rn | awk '{if ($1>=5) printf ("%s\0", $2);}' | xargs -0 -I {} your_script {}

Add whatever you want to do with the IP in your_script.

Printing with \0 and using xargs with -0 is related. It enables using \0 as separator between records instead of newline, which is a better practice.

CodePudding user response：

I've managed to do it this way:

for ip in $(sort access.log | awk '{print $1}' | uniq -cd | awk -v limit=10 '$1 > limit{print $2}'); do
    // Do something on "$ip"
done

CodePudding user response：

An alternative form to the answer of Ed Morton, you can also make use of xargs:

$ awk '(  cnt[$1] == 5){ print $1 }' file | xargs -n 1 some_command