Node.js AWS Lambda Archiver lib - Error in zip file creation-CodePudding

everyone,

I have written a lambda in Node.js that takes a folder and zips its contents. To do this I used the Archiver library: Archiver

The following is my code to create the File:

const zipFilePath = '/tmp/'   'filename'   '.zip'
const output = fs.createWriteStream(zipFilePath);

const archive = archiver('zip', {
    zlib: { level: 9 }
});

output.on('close', function () {
    console.log(archive.pointer()   ' total bytes');
    console.log('archiver has been finalized and the output file descriptor has closed.');
});

archive.on('error', function(err){
    throw err;
});

await archive.pipe(output);
await archive.directory(destFolder, false);
await archive.finalize();

To write the files I am using the /tmp folder of the lambdas, which is the only folder with write permissions.

The flow is as follows:

I get the path to the folder
Zip the content and save it in the folder destFolder

The file is then saved on an S3 bucket:

const file = await fs.readFileSync(filePath)
const params = {
    Bucket: bucketName,
    Key: fileName,
    Body: file
};
const res = await s3.upload(params).promise()
return res.Location

The zip file is generated, the problem is that when I download it, it is corrupted. I tried analysing it with an online zip file analyser (this) and the result of the analysis is the following:

Type = zip
ERRORS:
Unexpected end of archive
WARNINGS:
There are data after the end of archive
Physical Size = 118916
Tail Size = 19
Characteristics = Local

and the analyser shows that the files are all present in the .zip (I can see their names).

The strangest thing is that if instead of .zip the file (again with the same library) I create it in .tar

const archive = archiver('tar', {
    zlib: { level: 9 }
});

The file is generated correctly and I can extract it as an archive. Basically, it is as if something is wrong with the .zip format alone.

Has anyone ever experienced a similar episode? Can you help me find a solution? I need to create files in .zip format. Thank you.

CodePudding user response：

The problem is that you are not being able to zip your files properly, this can occurs by many issues, including:

You are not waiting the file to be processed, you need to use .close() event to do this.
You are not sending the correct path to the files or dirs to zipped, normally the files on the root dir of your project stay on /var/task/ on the Lambda dir system, so to send the correct file use __dirname '/file.name'
You are not appending correctly the files, check the .file() and .append() methods if you are sending the files correctly

If you have the following Lambda structure:

~/my-function
├── index.js
└── node_modules
├── package.json
├── package-lock.json
├── test.txt //file to be sent zipped on s3

The following example will works for you:

const archiver = require('archiver')
const fs = require('fs')
const AWS = require('aws-sdk');
const s3 = new AWS.S3({apiVersion: '2006-03-01'});


const sendToS3 = async (filePath) => {
    const bucketName = "bucket_name";
    const fileName = "zipped.zip"
    const file = await fs.readFileSync(filePath)
    const params = {
        Bucket: bucketName,
        Key: fileName,
        Body: file
    };
    const res = await s3.upload(params).promise()
    return res.Location
}

exports.handler = async event => {
  return new Promise((resolve, reject) => {
    const zippedPathName = '/tmp/example.zip';
    const output = fs.createWriteStream(zippedPathName);
    const fileToBeZipped = __dirname   '/test.txt';
    const zip = archiver('zip')

    zip
      .append(fs.createReadStream(fileToBeZipped), { name: 'test.txt' })
      .pipe(output)

    zip.finalize()

    output.on('close', (result => {
        sendToS3(zippedPathName).then(result => {
            resolve(result)
        })
    }))
  })
}