Google Cloud Storage pushing the files into bucket very slowly-CodePudding

I'm using the below code to recursively get all the paths in the folder and push them one by one into the google cloud storage bucket. The problem is, it's extremely slow. I have around 30-40K files that need to be pushed every day and each one is taking like 0.25 to 0.5 second to push. Is there any way I could push them all together? In bulk? Or another way that makes it way faster?

const {Storage} = require('@google-cloud/storage');
const fs = require('fs');
const path = require('path');

function getAllFilesInDirectoryRecursively(dir){
    const files = fs.readdirSync(dir, {withFileTypes: true});
    for (const file of files) {
        if (file.isDirectory()) {
            yield* getAllFilesInDirectoryRecursively(path.join(dir, file.name));
        } else {
            yield path.join(dir, file.name);
        }
    }
}

const storage = new Storage();

(async function(){
    for (let filePath of getAllFilesInDirectoryRecursively('./main/')) {
        await storage.bucket('bucket.domain.com').upload('./'   filePath, {
            destination: filePath.replace('main', ''),
        });
    }
})()

CodePudding user response：

You can use gsutil to upload contents of a directory using:

gsutil cp -r ./main gs://bucket-name

To run this command periodically, you can use a cron job or run the command from NodeJS script after the files have been generated.

For a solution without gsutil, it might be better to use Promise.all() instead of running all upload promises individually.