Hello there, im trying to upload files (.csv, .xlsx of 5000 row and 6 col) to firebase with a cloud functions that i call from the frontend, the cloud function start and after a few seconds of run its make an error "code":"4" "details":"Deadline exceeded"

(Blaze plan)

1. the upload is very slow
2. getting an Deadline exceeded error with code 4
3. its crashing the server after i get the error

explication of the code below:
i do a for of loop that iterate over an array of array that looks like this [['Dufour', 'Annie', '[email protected]', 33683333005, 'f'], ['john','Doe', '[email protected]', 33223424, 'm']] then i format each row in order to structure it as a firestore doc with many fields and try to add it to firestore then its crash i cant figure out when its crashing its really random and when im trying a smaller file (1000 row) its also crashing after some seconds of execution

  export const createNewLeads = functions.region("europe-west3").https.onCall(async (data, context) => {
  if (!context.auth) {
    throw new functions.https.HttpsError('failed-precondition', 'The function must be called'   'while authenticated.');
  }
  const { category, subcategory, allLeads, downloadUrl, file_id, fileDetails } = data;
  
  for (const lead of allLeads) {
    
    const leads_infos = format_lead_infos(lead , fileDetails);
    const leads_meta = format_lead_metadata(downloadUrl, file_id, fileDetails);
    const db_lead_ref = admin.firestore().collection('leads');
    const new_lead_doc_id =  db_lead_ref.doc().id;

     db_lead_ref
    .doc(new_lead_doc_id).set({
      lead_id: new_lead_doc_id,
      category: category,
      subcategory: subcategory,
      createdOn: Date(),
      uploaded_by: context.auth.uid,
      purchased_by: null,
      metadata: leads_meta,
      infos: leads_infos

    }).then(() => {
      functions.logger.info('createLeads SUCCESS', { docId: new_lead_doc_id });
      return {
        user_id: new_lead_doc_id,
        created: true,
      };
    }).catch((error: any) => {
      functions.logger.info('createLeads ERROR', { error });
      return {
        error: true,
        err_mess: error,
      };
    });
  }  
});

CodePudding user response：

As far as I can see you are not handling the asynchronous nature of writing to Firestore correctly, as you're not awaiting the call to set(). You're also returning a result on the first successful document write in the loop, which seems suspicious.

This should be closer to what you need:

try {
  for (const lead of allLeads) {    
    const leads_infos = format_lead_infos(lead , fileDetails);
    const leads_meta = format_lead_metadata(downloadUrl, file_id, fileDetails);
    const db_lead_ref = admin.firestore().collection('leads');
    const new_lead_doc_id =  db_lead_ref.doc().id;

    await db_lead_ref.doc(new_lead_doc_id).set({
      lead_id: new_lead_doc_id,
      category: category,
      subcategory: subcategory,
      createdOn: Date(),
      uploaded_by: context.auth.uid,
      purchased_by: null,
      metadata: leads_meta,
      infos: leads_infos
    });
  }  
}
catch (error: any) => {
  functions.logger.info('createLeads ERROR', { error });
  return {
    error: true,
    err_mess: error,
  };
}
functions.logger.info('createLeads SUCCESS', { docId: new_lead_doc_id 
return {
  user_id: new_lead_doc_id,
  created: true,
};

In this code we await each document write operation and then return an error if any of them fails, and success only once all of them succeed.

CodePudding user response：

Hello Frank thanks for helping us i copy the code you give to us bu after a few seconds of running i got this error error of the code from the firebase console log

export const createNewLeads = functions.region("europe-west3").https.onCall(async (data, context) => {
  if (!context.auth) {
    throw new functions.https.HttpsError('failed-precondition', 'The function must be called'   'while authentificated.');
  }
  const { category, subcategory, allLeads, downloadUrl, file_id, fileDetails } = data;
  const db_lead_ref = admin.firestore().collection('leads');
  const new_lead_doc_id =  db_lead_ref.doc().id;
  
  try {
    for (const lead of allLeads) {    
      const leads_infos = format_lead_infos(lead , fileDetails);
      const leads_meta = format_lead_metadata(downloadUrl, file_id, fileDetails);
  
      await db_lead_ref.doc(new_lead_doc_id).set({
        lead_id: new_lead_doc_id,
        category: category,
        subcategory: subcategory,
        createdOn: Date(),
        uploaded_by: context.auth.uid,
        purchased_by: null,
        metadata: leads_meta,
        infos: leads_infos
      });
    }  
  } catch (error:any) {
    functions.logger.info('createLeads ERROR', { error });
  return {
    error: true,
    err_mess: error,
  }
  
  }
  functions.logger.info('createLeads SUCCESS', { docId: new_lead_doc_id })
return {
  user_id: new_lead_doc_id,
  created: true,
}
});