How can I get the count of https response code from a log file?-CodePudding

I have a set of log files of load balancer and I want to get the list with total number of count https code wise. To do the same I'm using below set of commands but I'm not getting a proper output. Can someone please help me with that and let me know what is the issue?

    cat `ls -rt SampleFile.log||tail -1|head -1` \
    |awk '{print $4,$7,$8,$9,$10}' \
    |cut -c2- \
    |awk '{print $1" "$3}' FS='"' \
    |awk '{print $1,$2":"$3,$4}' FS=":" OFS=" " \
    |awk '{print $1,$2,$4}' \
    |sed 's/\/Jan\// 01 /;s/\/Feb\// 02 /;s/\/Mar\// 03 /;s/\/Apr\// 04 /;s/\/May\// 05 /;s/\/Jun\// 06 /;s/\/Jul\// 07 /;s/\/Aug\// 08 /;s/\/Sep\// 09 /;s/\/Oct\// 10 /;s/\/Nov\// 11 /;s/\/Dec\// 12 /' \
    |awk '{print $3,$2,$1,$4,$5}' \
    |sort \
    |uniq -c \
    |awk 'BEGIN { FS=" " }{print $4"."$3"."$2" "$5","$6","$1}' \
    |gawk 'BEGIN{FS=","} {LN[$1]; HD[$2]; MX[$1,$2]=$3} END { printf "%s", "\n \n Timestamp      "; for (i in HD) printf " |  %s ", i; print ""; for (j in LN) { printf "%s",j; for (i in HD) { if (MX[j,i] =="") { printf " | ]",0 } else { printf " | ]", MX[j,i] } } print "" } }' \
    |sort

SampleFile.log

10.99.2.216 - - [06/Sep/2021:19:00:00 0200] "GET /mceapp/customer/subscriptions/1053995219/accounts;filter=all;scope=node HTTP/1.1" 200 1136 dCvQi12Mt20000000 delxvi13.de.pri.o2.com:36093 10.99.2.216 - - [06/Sep/2021:19:00:00 0200] "PUT /api/emailverification/tokens/d11111d2-278a-4fed-ae57-9f50bf277025/emails HTTP/1.1" 200 78 gY4NZ1BMv20000000 delxvi18.de.pri.o2.com:36090 10.99.2.216 - - [06/Sep/2021:19:00:00 0200] "GET /mceapp/prepaid/inlife/brands/77/prepaidsubscriptions/mobile/4915733639572/possibleproducts HTTP/1.1" 200 10877 dCvQi13Mt20000000 delxvi08.de.pri.o2.com:36091

Expected Output(Sample): (It should give the output as total number of https response wise count.)

The same command is working without any issue, if I'm using with below set of log files. SampleLog_2.log:

10.99.2.216 - - [27/Jun/2021:16:00:00 0200] [105000] "GET /mceapp/contractextension/subscriptions/1151976376/contractextensioneligibility HTTP/1.1" 200 943 03eEr11g400000000 delxvi38.de.pri.o2.com:10582 10.99.2.216 - - [27/Jun/2021:16:00:00 0200] [280000] "GET /mceapp/invoice/xstack/accounts/6046093360/invoiceoverview HTTP/1.1" 200 18589 uCWE718c400000000 delxvi12.de.pri.o2.com:36094 10.99.2.216 - - [27/Jun/2021:16:00:00 0200] [408000] "GET /mceapp/invoice/xstack/accounts/6046093360/invoice/documents HTTP/1.1" 200 5962 GrDqn1Jf400000000 delxvi11.de.pri.o2.com:36094

What is the difference with both the sample_files and what command I need to change to get the expected output.

CodePudding user response：

It's very easy to cut another set of block from your log file. Use below command and it will do the needful what your have requested.

cat `ls -rt access-duration.log||tail -1|head -1`|awk '{print 
$4,$6,$7,$8,$9}'|cut -c2-|awk '{print $1" "$3}' FS='"'|awk '{print 
$1,$2":"$3,$4}' FS=":" OFS=" "|awk '{print $1,$2,$4}'|sed 's/\/Jan\// 
01 /;s/\/Feb\// 02 /;s/\/Mar\// 03 /;s/\/Apr\// 04 /;s/\/May\// 05 
/;s/\/Jun\// 06 /;s/\/Jul\// 07 /;s/\/Aug\// 08 /;s/\/Sep\// 09 
/;s/\/Oct\// 10 /;s/\/Nov\// 11 /;s/\/Dec\// 12 /'|awk '{print 
$3,$2,$1,$4,$5}'|sort|uniq -c|awk 'BEGIN { FS=" " }{print $4"."$3"."$2" 
"$5","$6","$1}'|gawk 'BEGIN{FS=","} {LN[$1]; HD[$2]; MX[$1,$2]=$3} END 
{ printf "%s","\n \n  Timestamp      "; for (i in HD) printf "|     
%s", i; print ""; for (j in LN) {printf "%s",j; for (i in HD) { if 
(MX[j,i] =="") { printf " | ]",0 } else { printf " | ]", MX[j,i] } 
} print "" } }' |sort

I made the changes only in below line and replaced with -1 after $4 '{print $4,$6,$7,$8,$9} and it worked.

CodePudding user response：

What is the difference with both the sample_files

The difference is the additional bracketed column in SampleLog_2.log after the timestamp.

and what command I need to change to get the expected output.

In order to be less dependent on unneeded columns, you can for example replace the pipeline

cat `ls -rt SampleFile.log||tail -1|head -1` \
|awk '{print $4,$7,$8,$9,$10}' \
|cut -c2- \
|awk '{print $1" "$3}' FS='"' \
|awk '{print $1,$2":"$3,$4}' FS=":" OFS=" " \
|awk '{print $1,$2,$4}'

with

sed 's?.*\(../.../....\):\(..:..\):.. .*" \(...\).*?\1 \2 \3?' SampleFile.log

There's probably room for further simplifications in the following stages.