Reading JSON file using shell scriptt-CodePudding

I am new to shell scripting and I am trying to read an API to fetch the values...

But I have a couple of doubts to make it executable effectively.

It is a simple program to display the universities and the domains in the cities

The link to the API is : http://universities.hipolabs.com/search?name=middle

and my code is

# A simple shell program to find the all the universities and domains present in the country depending on the user-provided.

echo "Description : List of universities based on the location provided \n" 

user_input_location_of_the_university=$1

if [ -z "$user_input_location_of_the_university" ] 
then 
   echo "location required. please provide a valid country name"
   
   exit 1 
   
fi 

user_input_location_of_the_university_replacing_spaces=$( echo "$user_input_location_of_the_university" | sed 's/ / /g' )

#user_input_location_of_the_university=$(echo "$user_input_location_of_the_university | sed 's/ / /g')


response_contianing_university_data=$(curl -sS -X GET "http://universities.hipolabs.com/search?name=middle&country=$user_input_location_of_the_university_replacing_spaces")

if [ -z "$response_contianing_university_data" ] 
then 
   echo "invalid request, provide with a valid location"
   
else
    
  # echo "total response is: "

  # echo $response_contianing_university_data | jq .

   echo "Name of the university based the location $user_input_location_of_the_university is: \r"

   echo $response_contianing_university_data | jq -r '.[] .name'

   echo "\n"

   echo "domain of the university based the location $user_input_location_of_the_university is: "

   echo $response_contianing_university_data | jq -r '(.[] .domains[0:] | @sh)'
   
    
fi

If I execute the program by providing with an invalid name like sh file_anme.sh "tur" instead of sh file_anme.sh "turkey" it should be printed like:

invalid request, provide with a valid location

but it is printing...

Name of the university based the location tur is: 


domain of the university based the location tur is:

what do I have to put in the if condition to test the URL provided is working or not?

and

another doubt is, how to remove the quotes while displaying the output at the terminal? I have used the -r flag with the help of jq when fetching details of the domain(second part)...

I am providing it in the below:

#while executing
sh myfiles.sh "United States"  

#The output
Decription : List of universities based on the location provided 

Name of the university based the location United States is: 
Middlesex County College
Middlesex Community College
Middlebury College
Middle Tennessee State University
Middle Georgia State College


domain of the university based the location United States is: 
'middlesexcc.edu'
'middlesex.mass.edu'
'middlebury.edu'
'mtsu.edu'
'mga.edu'

But I need to display it as the following without the quotes:

domain of the university based the location United States is: 
middlesexcc.edu
middlesex.mass.edu
middlebury.edu
mtsu.edu
mga.edu

What should I do to remove the quotes other than "jq -r"?

for the reference, I am attaching the JSON data from the link mentioned above...

[{"country": "United Kingdom", "domains": ["middlesbro.ac.uk", "mbro.ac.uk"], "name": "Middlesbrough College", "state-province": null, "web_pages": ["https://www.mbro.ac.uk/"], "alpha_two_code": "GB"}, {"country": "United States", "domains": ["middlesexcc.edu"], "name": "Middlesex County College", "state-province": null, "web_pages": ["http://www.middlesexcc.edu"], "alpha_two_code": "US"}, {"country": "United States", "domains": ["middlesex.mass.edu"], "name": "Middlesex Community College", "state-province": null, "web_pages": ["http://www.middlesex.mass.edu"], "alpha_two_code": "US"}, {"country": "United Kingdom", "domains": ["mdx.ac.uk"], "name": "Middlesex University", "state-province": null, "web_pages": ["http://www.mdx.ac.uk/"], "alpha_two_code": "GB"}, {"country": "Kuwait", "domains": ["aum.edu.kw"], "name": "American University of Middle East", "state-province": null, "web_pages": ["http://www.aum.edu.kw/"], "alpha_two_code": "KW"}, {"country": "United States", "domains": ["middlebury.edu"], "name": "Middlebury College", "state-province": null, "web_pages": ["http://www.middlebury.edu/"], "alpha_two_code": "US"}, {"country": "United States", "domains": ["mtsu.edu"], "name": "Middle Tennessee State University", "state-province": null, "web_pages": ["http://www.mtsu.edu/"], "alpha_two_code": "US"}, {"country": "United States", "domains": ["mga.edu"], "name": "Middle Georgia State College", "state-province": null, "web_pages": ["http://www.mga.edu/"], "alpha_two_code": "US"}, {"country": "Jordan", "domains": ["meu.edu.jo"], "name": "Middle East University", "state-province": null, "web_pages": ["http://www.meu.edu.jo/"], "alpha_two_code": "JO"}, {"country": "Turkey", "domains": ["metu.edu.tr"], "name": "Middle East Technical University", "state-province": null, "web_pages": ["http://www.metu.edu.tr/"], "alpha_two_code": "TR"}]

CodePudding user response：

First question:

If the place name doesn't match, the JSON response is empty, which is something you can check for. The API apparently does not provide a specific error code or message for this state, other than the response being empty.

Check for an empty JSON message:

if [ "$response_contianing_university_data" = '[]' ]; then
    echo "invalid request, provide with a valid location"
else
# etc

You could also do if [ -z "$(echo "$response_contianing_university_data" | jq '.[]')" ] or similar if you want. The processed JSON will be literally empty.

Second question:

The single quotes around the domains are being added because you are using the @sh operator in jq. Simply remove it.

List the domains without quotes:

echo "$response_contianing_university_data" | jq -r '.[].domains[]'

Also:

Your variable names are too long. $location is much clearer for example.
You've misspelled "contianing" (should be "containing").
You must quote your variables, with double quotes ("), when echoing them in to jq.

CodePudding user response：

Interesting API (JSON-file). If you'd use xidel, then you wouldn't need curl, sed, or a Bash-script.

$ location='United States'; xidel -s "http://universities.hipolabs.com/search?name=middle&country=$location" -e '
  if (exists($json())) then (
    "Names of the universities based on the location '"$location"' is:",
    $json()/name,
    "",
    "Domains of the universities based on the location '"$location"' is:",
    $json()/(domains)()
  )
  else
    "Invalid request. Provide with a valid location."
'

$ location='United States' xidel -s --variable location -e '
  let $json:=json-doc("http://universities.hipolabs.com/search?name=middle&country="||$location) return
  if (exists($json())) then (
    x"Name of the university based the location {$location} is:",
    $json()/name,
    "",
    x"Domain of the university based the location {$location} is:",
    $json()/(domains)()
  )
  else
    "Invalid request. Provide with a valid location."
'

which returns:

Name of the university based the location United States is:
Middlesex County College
Middlesex Community College
Middlebury College
Middle Tennessee State University
Middle Georgia State College

Domain of the university based the location United States is:
middlesexcc.edu
middlesex.mass.edu
middlebury.edu
mtsu.edu
mga.edu

Alternatively you could just as well query the JSON directly, as that's what this API is also doing in the background.

$ xidel -s "https://raw.githubusercontent.com/Hipo/university-domains-list/master/world_universities_and_domains.json" -e '
  parse-json($raw)()[country="United States" and starts-with(name,"Middle")]/concat(name," - ",join((domains)(),", "))
'
Middlebury College - middlebury.edu
Middle Georgia State College - mga.edu
Middle Tennessee State University - mtsu.edu
Middlesex Community College - mxcc.commnet.edu
Middlesex Community College - middlesex.mass.edu
Middlesex County College - middlesexcc.edu

(For the github-url -e '$headers' returns Content-Type: text/plain; (instead of Content-Type: application/json), so you have to parse the raw content as JSON. The global default variable $json is only set when the input is Content-Type: application/json)