How to find unused API from the log file using bash script?-CodePudding

The goal is to find out the APIs that have not been used in the past seven days. There is a file that stores all API lists, and a file folder that contains daily log data.

api.txt(Store all api lists):

/api/user/status
/api/property/status
/api/video/banner/list
/api/user/info
/api/user/giftList
/api/user/ticket
...

log data(store seven days):

each of log file will look like this

2023-01-16 01:20:28.646  00:00: ID: 1865555 IP:1.2.3.4 POST: /api/user/status Response:1
2023-01-16 01:20:28.646  00:00: ID: 1444444 IP:1.2.3.4 POST: /api/user/info Response:1
2023-01-16 01:20:28.646  00:00: ID: 1865555 IP:1.2.3.4 POST: /api/property/status Response:1
2023-01-16 01:20:28.646  00:00: ID: 1333333 IP:1.2.3.4 GET: /api/video/comments Response:2
2023-01-16 01:20:28.646  00:00: ID: 1865555 IP:1.2.3.4 GET: /api/user/rate Response:1
...

Methods:

Start from the first line of the api data (/api/user/status), take the files in the log folder and compare them one by one (seven files). if find that the API is used in the middle, change to the next API(/api/property/status) to check, if the API seven files have not been used, just echo the API.

What I try:

Problem1: if [[ ! $line == *$i* ]]; doesn't work, it works act as if [[ $line == *$i* ]]; but I want to get the not equal result

Problem2: This is only compare two file, but how can I check the rest of six files inside the loop?

file1="api.txt"

file2="2023-01-10_00-00-00.log" 

if [ ! -f "$file1" ] || [ ! -f "$file2" ]; then
    echo "Error: One of the files does not exist."
    exit 1
fi

declare -a lines1

while IFS= read -r line; do
    lines1 =("$line")
done < "$file1"

while IFS= read -r line; do
    for i in "${lines1[@]}"; do 
        if [[ ! $line == *$i* ]];
        then
            echo "$i"
            break
        fi
    done
done < "$file2"

CodePudding user response：

Grep(awk,sed) can work with multiple files. I'd suggest to loop through APIs and grep them in logs for 7 days. Something like this:

api_names=( $(cat api.txt) )
log_files=( $(find path_to_logs -type f -daystart -mtime  7) )

for api in "${api_names[@]}"; { grep -q " $api " "${log_files[@]}" || echo $api not used; }

CodePudding user response：

I would first create from the logs a list of all APIs which are actually used, i.e. something like

grep -hoE ' (GET|POST): /api/[^ ] ' $logdir/*.log | cut -d ' ' -f 2 | sort -u >used_api.txt

Then I would search in api.txt for all lines which are absent from the freshly generated used_api.txt

grep -vxF -f used_api.txt api.txt >unused_api.txt