Google Apps Script cUrl POST Request-CodePudding

I am trying to pull data from an API from https://app.scrapingant.com/ using Google Apps Script.

The documentation is here: https://docs.scrapingant.com/request-response-format#passing-parameters-via-post-method

And the cURL request I am provided with is as follows:

curl --request POST \
--url 'https://api.scrapingant.com/v1/general' \
--header 'x-api-key: XXX-XXX' \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{ "url": "https://creditcardgenius.ca/credit-cards/hsbc-world-elite-mastercard"}'

I have this function setup in Apps Script:


function getCcgRate() {
  var root = 'https://api.scrapingant.com/v1/general';
  var params = {
    'headers': {
      'x-api-key': 'XXX-XXX',
      'accept': 'application/json',
      'content-type': 'application/json'
    },
    "url" : "https://creditcardgenius.ca/credit-cards/hsbc-world-elite-mastercard",
    "muteHttpExceptions": true,
    "followRedirects": true,
    "validateHttpsCertificates": true
  };
  var response = UrlFetchApp.fetch(root, params);
  var data = response.getContentText();
  var json = JSON.parse(data);
  Logger.log(json);
}

I am getting the following error: {detail=url param is required. Please check out the documentation: https://docs.scrapingant.com/}

I've never used GAS before so its a bit of a struggle figuring out what it wants right now. Any help would be appreciated.

CodePudding user response：

I believe your goal is as follows.

You want to convert the following curl command to Google Apps Script.

  curl --request POST \
  --url 'https://api.scrapingant.com/v1/general' \
  --header 'x-api-key: XXX-XXX' \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '{ "url": "https://creditcardgenius.ca/credit-cards/hsbc-world-elite-mastercard"}'

In this case, how about the following modification?

Modified script:

function getCcgRate() {
  var root = 'https://api.scrapingant.com/v1/general';
  var params = {
    'headers': {
      'x-api-key': 'XXX-XXX',
      'accept': 'application/json',
    },
    "muteHttpExceptions": true,
    'payload': JSON.stringify({ "url": "https://creditcardgenius.ca/credit-cards/hsbc-world-elite-mastercard" }),
    'contentType': 'application/json',
    'method': 'post',
  };
  var response = UrlFetchApp.fetch(root, params);
  var data = response.getContentText();
  var json = JSON.parse(data);
  Logger.log(json);
}

Note:

This modified script supposes that your curl command works fine. Please be careful about this.

Reference:

fetch(url, params)

CodePudding user response：

Scrapingant does not support parameter passing on body for the GET requests, it is only viable for POST requests.

What you have to do is to encode your parameters into your request url (notice that url is the one that you send requests to, not the url you want to scrape).

This page explains how you can encode the url you want to send a GET request with encodeURIComponent() function.