How to get data HTTP response code 5xx from multiple files with bash script in different directory-CodePudding

I am learning about bash scripting, but I might need some example for this: am has 2 or more files, file1 in directory /home/usr/file1 with this data:

83.149.9.216 - - [17/May/2015:10:05:07  0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 501 2892 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:34  0000] "GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1" 502 430406 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:57  0000] "GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Bold.ttf HTTP/1.1" 500 38720 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

so also, file2 in directory /home/usr/usr2/file2 with this data:

46.105.14.53 - - [17/May/2015:11:05:33  0000] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 500 14872 "-" "UniversalFeedParser/4.2-pre-314-svn  http://feedparser.org/"
5.102.173.71 - - [17/May/2015:11:05:13  0000] "GET /robots.txt HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)"
5.102.173.71 - - [17/May/2015:11:05:06  0000] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "-" "Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)"
208.91.156.11 - - [17/May/2015:11:05:05  0000] "GET /files/logstash/logstash-1.3.2-monolithic.jar HTTP/1.1" 404 324 "-" "Chef Client/10.18.2 (ruby-1.9.3-p327; ohai-6.16.0; x86_64-linux;  http://opscode.com)"

well, how to combine/read all data in different directory with bash script. I want output like this to get response 5xx from that file:

83.149.9.216 [17/May/2015:10:05:07  0000 HTTP/1.1" 501
83.149.9.216 [17/May/2015:10:05:34  0000] HTTP/1.1" 502
46.105.14.53 [17/May/2015:11:05:33  0000] HTTP/1.1" 500
5.102.173.71 [17/May/2015:11:05:13  0000] HTTP/1.1" 500
etc...

CodePudding user response：

How about

sort /home/usr/file1  /home/usr/usr2/file2

CodePudding user response：

I am not sure what your question is, but just in case, you do not know how to write the regular expression to parse your web server logs:

#! /bin/bash

exec <<EOF
83.149.9.216 - - [17/May/2015:10:05:07  0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 501 2892 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:34  0000] "GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1" 502 430406 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:57  0000] "GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Bold.ttf HTTP/1.1" 500 38720 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
46.105.14.53 - - [17/May/2015:11:05:33  0000] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 500 14872 "-" "UniversalFeedParser/4.2-pre-314-svn  http://feedparser.org/"
5.102.173.71 - - [17/May/2015:11:05:13  0000] "GET /robots.txt HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)"
5.102.173.71 - - [17/May/2015:11:05:06  0000] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "-" "Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)"
208.91.156.11 - - [17/May/2015:11:05:05  0000] "GET /files/logstash/logstash-1.3.2-monolithic.jar HTTP/1.1" 404 324 "-" "Chef Client/10.18.2 (ruby-1.9.3-p327; ohai-6.16.0; x86_64-linux;  http://opscode.com)"
EOF

while read -r line; do
  if [[ $line =~ ^(((^|\.)[0-9]{1,3}){4})\ -\ -(\ \[[^\]] \])\ \"GET\ [^\"]*\"(\ [0-9]{3}).*$ ]]; then
    for i in 1 4 5; do
      printf %s "${BASH_REMATCH[$i]}"
    done
    printf '\n'
  fi
done