Is there anyway to increase performance of my google sheet app script connected with Firebase?-CodePudding

I'm trying to connect my Google Sheet to Firebase-RTDB as i want to transfer my data to proper database (due to gigantic amouth of data) and fetch them through google sheet based on first column value.

The code works, but the performance is not very satisfying at all. It takes about 20 seconds, even with the If statement catching whether the cell is empty before doing a loop. If i don't do that it will take a whooping 60 seconds just for 80 rows. Which is not good at all.

function getAllData_ID33() {
  //Read sheet
  var ss = SpreadsheetApp.openById('**MY_GOOGLESHEET_ID**');
  let sheet = ss.getSheetByName('Test')
  let startRow = 2

  //Get Range
  let numRow = sheet.getDataRange().getLastRow();
  let lastCols = sheet.getDataRange().getLastColumn();
  let dataRange = sheet.getRange(startRow, 1, numRow, lastCols);
  let celldata = dataRange.getValues();

  //connect_link
  //ID33_Headoffice
  var ID33_HEADOFFICE_DATABASE = "**MY_FIREBASE_LINK**";


      //if first and fourth column is not empty, pass.
      //Otherwise, start the fetching loop.
  for (j = 0; j < celldata.length-1; j  ) {
    //row = row[column]
    row = celldata[j];

    let phyid = ID33_HEADOFFICE_DATABASE   "//"   row[0]
    let base = FirebaseApp.getDatabaseByUrl(phyid);
    let dataSet_33headoffice = [base.getData()]; 
    let id33_rows = [], data_33_headofflice;

    if (row[0] != "") {
      if (row[4] == "") {
        for (i = 0; i < dataSet_33headoffice.length; i  ) {
      //need Booking ID, Stockout-ID, Branch ID,  Branch To Name, Comment
          data_33_headofflice = dataSet_33headoffice[i];
            //Logger.log([
            //data_33_headofflice['Booking ID'], 
            //data_33_headofflice['Stock Out (ID)'],
            //data_33_headofflice['Branch (ID)'],
            //data_33_headofflice['Branch To (Name)'],
            //data_33_headofflice['Comment']
            //]);
            
            id33_rows.push([
            data_33_headofflice['Booking ID'], 
            data_33_headofflice['Stock Out (ID)'],
            data_33_headofflice['Branch (ID)'],
            data_33_headofflice['Branch To (Name)'],
            data_33_headofflice['Comment']
            ]);      
          let id33dataRange = sheet.getRange(j   2,2,id33_rows.length,5);
          id33dataRange.setValues(id33_rows); 
        }
      }
    }     
  }
}

Should be noted that i'm not very familliar with Javascript/Google App Script at all. I don't even know if Firebase is even fit for a task like this. So if you do know what's causing it to be slow and how to improve, please explain. Thanks for all the support!

CodePudding user response：

You need to move these two lines out of the loop:

      let id33dataRange = sheet.getRange(j   2, 2, id33_rows.length,5);
      id33dataRange.setValues(id33_rows);

Otherwise, you're updating the spreadsheet after every loop iteration.

CodePudding user response：

I believe your goal is as follows.

You want to reduce the process cost of your script.

From The code works, but the performance is not very satisfying at all., I understood that your script worked. From this situation, how about the following modification?

Modification points:

When setValues is used in a loop, the process cost becomes high. Ref
From your showing script, I guessed that dataSet_33headoffice.length might be always 1. Because when dataSet_33headoffice.length is 2, I thought that the row is deviated. But, in your question, you say The code works. So, I guessed that the value might be 1.
In order to put the values to the discrete rows by reducing the process cost, I would like to propose using Sheets API. By this, by one API call, the values can be put.

When these points are reflected in your script, how about the following modification?

Modified script:

Before you use this script, please enable Sheets API at Advanced Google services.

function getAllData_ID33() {
  var spreadsheetId = "###"; // Please set your Spreadsheet ID.
  var sheetName = 'Test'; // Please set your sheet name.
  var ID33_HEADOFFICE_DATABASE = "**MY_FIREBASE_LINK**"; // Please set this value.

  var ss = SpreadsheetApp.openById(spreadsheetId);
  let sheet = ss.getSheetByName(sheetName)
  let startRow = 2
  let numRow = sheet.getDataRange().getLastRow();
  let lastCols = sheet.getDataRange().getLastColumn();
  let dataRange = sheet.getRange(startRow, 1, numRow, lastCols);
  let celldata = dataRange.getValues();
  var data = [];
  for (j = 0; j < celldata.length - 1; j  ) {
    row = celldata[j];
    let phyid = ID33_HEADOFFICE_DATABASE   "//"   row[0]
    let base = FirebaseApp.getDatabaseByUrl(phyid);
    let dataSet_33headoffice = [base.getData()];
    let id33_rows = [], data_33_headofflice;
    if (row[0] != "") {
      if (row[4] == "") {
        for (i = 0; i < dataSet_33headoffice.length; i  ) {
          data_33_headofflice = dataSet_33headoffice[i];
          id33_rows.push([
            data_33_headofflice['Booking ID'],
            data_33_headofflice['Stock Out (ID)'],
            data_33_headofflice['Branch (ID)'],
            data_33_headofflice['Branch To (Name)'],
            data_33_headofflice['Comment']
          ]);
        }
        if (id33_rows.length > 0) {
          data.push({ range: `'${sheetName}'!B${j   2}`, values: id33_rows });
        }
      }
    }
  }
  if (data.length == 0) return;
  Sheets.Spreadsheets.Values.batchUpdate({ data, valueInputOption: "USER_ENTERED" }, spreadsheetId);
}

When this script is run, the request body is created in the loop. And, the request body is used outside of the loop using Sheets API. By this, the process cost can be reduced.

Note:

This sample script supposes that your script worked fine. Please be careful about this.

Reference:

Method: spreadsheets.values.batchUpdate