Unable to produce json response issuing get requests with parameters-CodePudding

I'm trying to fetch json response from a webpage issuing get http requests with appropriate parameters but I get 403 Forbidden as response. When I do the same in python, I get required response accordingly. However, things go wrong when I go for the following approach:

function capterraScraper() {
  var Url = 'https://www.capterra.com/directoryPage/rest/v1/category';
  var params = {
      'htmlName': '360-degree-feedback-software',
      'rbr': 'false',
      'countryCode': 'BD'
  };
  var options = {
    'method' : 'GET',
    'params' : params,
    'headers' : {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'
    }
  };
  var response = UrlFetchApp.fetch(Url, options);
  console.log(JSON.parse(response));
}

How can I make the script produce json response?

CodePudding user response：

From your script, I thought that params might be used as the query parameter. When I tested it, I confirmed that the JSON data could be obtained. But when I tested the URL using Google Apps Script, I also confirmed the error of 403 Forbidden. Fortunately, when I accessed the URL using Javascript, the JSON data can be obtained. So in this case, as a workaround, I would like to propose a workaround for using the JSON data at the Google Apps Script side.

The flow of workaround is as follows.

Open a dialog.
Retrieving the JSON data using Javascript, and returning the data to Google Apps Script side.

By this, you can use the JSON data at Google Apps Script.

Sample script:

Please copy and paste the following script to the script editor of Google Spreadsheet, and run main() at the script editor. By this, a dialog is opened at Google Spreadsheet and the JSON data is retrieved, and return the data to the Google Apps Script side.

function main(obj) {
  if (!obj) {
    var html = HtmlService.createHtmlOutput(`<script>fetch('https://www.capterra.com/directoryPage/rest/v1/category?'   (new URLSearchParams({htmlName: '360-degree-feedback-software',rbr: 'false',countryCode: 'BD'}))).then(response => response.json()).then(res => google.script.run.withSuccessHandler(google.script.host.close).main(res)).catch(err => console.log(err));</script>`);
    SpreadsheetApp.getUi().showModalDialog(html, "sample");
    return;
  }
  console.log(obj)
  DriveApp.createFile("sample.txt", JSON.stringify(obj));
}

When this sample script is run, the retrieved JSON data is saved in a file on the root folder. Of course, you can use the data as obj.

The Javascript in this script is as follows.

  <script>
  fetch('https://www.capterra.com/directoryPage/rest/v1/category?'   (new URLSearchParams({htmlName: '360-degree-feedback-software',rbr: 'false',countryCode: 'BD'})))
  .then(response => response.json())
  .then(res => google.script.run.withSuccessHandler(google.script.host.close).main(res))
  .catch(err => console.log(err));
  </script>

Note:

This workaround uses a dialog on Google Spreadsheet. So, unfortunately, in this case, even when this script is run from the outside and the time-driven trigger, the script cannot be used. Please be careful about this.

Reference:

Dialogs and Sidebars in Google Workspace Documents