Counting results from my curl command in bash-CodePudding

I am running a curl command that connects with my ServiceNow PDI. I am trying to pull all the records from the Incident table and print out how many there are. Additionally my code takes input from a file and loops thru the list of keywords and then searches for which record's description contains the keyword

while IFS= read -r string
do
    curl --user admin:password "https://dev000000.service-now.com/incident.do?CSV&sysparm_query=descriptionLIKE"$string"&sysparm_view=ess" --proxy http://sub.proxy.att.com:8080 -k  
done < keywords.txt

my keywords.txt looks something like:

email
server
hard drive

right now my code successfully loops thru keywords, handles the white space exception, and prints the records, but i would like to print the total and the keyword like this:

3, email
7, server
2, hard drive

CodePudding user response：

You can use the grep command with the -c option to count the result and then pass it to the awk command like the below to print your output:

while IFS= read -r string
do
    curl "https://foor.bar?string" | grep -c "$string" | awk -v string=$string '{print $1, string}'
done < keywords.txt

CodePudding user response：

You are doing a curl for every keyword. Flip the paradigm.

mapfile -t keys < keywords.txt
declare -A lookup=()
while read -r line; do
  for k in "${keys[@]}"; do
    case "${line,,}" in *"${k,,}"*) ((lookup[$k]  ));; esac
  done
done < <(curl --user admin:password "https://dev000000.service-now.com/incident.do?CSV&sysparm_query=descriptionLIKE"$string"&sysparm_view=ess" --proxy http://sub.proxy.att.com:8080 -k)
for k in "${keys[@]}"; do echo "${lookup[$k], $k"; done

This doesn't url-encode though, you should probably do that in your file, as mentioned elsewhere. Likewise, I downcased everything for a case match which may not be your best bet here. YMMV.

You can make it a lot faster (probably) by putting the whole logic into awk.

awk 'NR==FNR{lookup[$0]=0; next}
     { for (k in lookup) { if ($0 ~ k) { lookup[k]   } } }
     END{ for (k in lookup) { print lookup[k]", "k } }' keywords.txt <(curl ... )

You can also do case-insensitive searches by adding a BEGIN{IGNORECASE=1}.