I have an Express server that receives FormData with an attached FLAC audio file. The code works as expected for several files of varying size (10 - 70MB), but some of them get stuck in the 'file' event and I cannot figure out why this happens. It is even more strange when a file that previously did not fire the file.on('close', => {}) event, as can be seen in the documentation for Busboy, suddenly does so, with the file being successfully uploaded.
To me, this seems completely random, as I have tried this with a dozen files of varying size and content type (audio/flac & audio/x-flac), and the results have been inconsistent. Some files will, however, not work at all, even if I attempt to parse them many times over. Whereas, certain files can be parsed and uploaded, given enough attempts?
Is there some error that I fail to deal with in the 'file' event? I did try to listen to the file.on('error', => {}) event, but there were no errors to be found. Other answers suggest that the file stream must be consumed for the 'close' event to proceed, but I think that file.pipe(fs.createWriteStream(fileObject.filePath)); does that, correct?
Let me know if I forgot to include some important information in my question. This has been bothering me for about a week now, so I am happy to provide anything of relevance to help my chances of overcoming this hurdle.
app.post('/upload', (request, response) => {
response.set('Access-Control-Allow-Origin', '*');
const bb = busboy({ headers: request.headers });
const fields = {};
const fileObject = {};
bb.on('file', (_name, file, info) => {
const { filename, mimeType } = info;
fileObject['mimeType'] = mimeType;
fileObject['filePath'] = path.join(os.tmpdir(), filename);
file.pipe(fs.createWriteStream(fileObject.filePath));
file.on('close', () => {
console.log('Finished parsing of file');
});
});
bb.on('field', (name, value) => {
fields[name] = value;
});
bb.on('close', () => {
bucket.upload(
fileObject.filePath,
{
uploadType: 'resumable',
metadata: {
metadata: {
contentType: fileObject.mimeType,
firebaseStorageDownloadToken: fields.id
}
}
},
(error, uploadedFile) => {
if (error) {
console.log(error);
} else {
db.collection('tracks')
.doc(fields.id)
.set({
identifier: fields.id,
artist: fields.artist,
title: fields.title,
imageUrl: fields.imageUrl,
fileUrl: `https://firebasestorage.googleapis.com/v0/b/${bucket.name}/o/${uploadedFile.name}?alt=media&token=${fields.id}`
});
response.send(`File uploaded: ${fields.id}`);
}
}
);
});
request.pipe(bb);
});
UPDATE: 1
I decided to measure the number of bytes that were transferred upon each upload with file.on('data', (data) => {}), just to see if the issue was always the same, and it turns out that this too is completely random.
let bytes = 0;
file.on('data', (data) => {
bytes = data.length;
console.log(`Loaded ${(bytes / 1000000).toFixed(2)}MB`);
});
First Test Case: Fenomenon - Sleepy Meadows Of Buxton
Source: https://fenomenon.bandcamp.com/track/sleepy-meadows-of-buxton
- Size: 30.3MB
- Codec: FLAC
- MIME: audio/flac
Results from three attempts:
- Loaded 18.74MB, then became stuck
- Loaded 5.05MB, then became stuck
- Loaded 21.23MB, then became stuck
Second Test Case: Almunia - New Moon
Source: https://almunia.bandcamp.com/track/new-moon
- Size: 38.7MB
- Codec: FLAC
- MIME: audio/flac
Results from three attempts:
- Loaded 12.78MB, then became stuck
- Loaded 38.65, was successfully uploaded!
- Loaded 38.65, was successfully uploaded!
As you can see, the behavior is unpredictable to say the least. Also, those two successful uploads did playback seamlessly from Firebase Storage, so it really worked as intended. What I cannot understand is why it would not always work, or at least most of the time, excluding any network-related failures.
UPDATE: 2
I am hopelessly stuck trying to make sense of the issue, so I have now created a scenario that closely resembles my actual project, and uploaded the code to GitHub. It is pretty minimal, but I did add some additional libraries to make the front-end pleasant to work with.
There is not much to it, other than an Express server for the back-end and a simple Vue application for the front-end. Within the files folder, there are two FLAC files; One of them is only 4.42MB to prove that the code does sometimes work. The other file is much larger at 38.1MB to reliably illustrate the problem. Feel free to try any other files.
Note that the front-end must be modified to allow files other than FLAC files. I made the choice to only accept FLAC files, as this is what I am working with in my actual project.
CodePudding user response：
You'll need to write the file directly when BusBoy emits the file event.
It seems there is a race condition if you rely on BusBoy that prevents the file load from being completed. If you load it in the file event handler then it works fine.
app.post('/upload', (request, response) => {
response.set('Access-Control-Allow-Origin', '*');
const bb = busboy({
headers: request.headers
});
const fileObject = {};
let bytes = 0;
bb.on('file', (name, file, info) => {
const {
filename,
mimeType
} = info;
fileObject['mimeType'] = mimeType;
fileObject['filePath'] = path.join(os.tmpdir(), filename);
const saveTo = path.join(os.tmpdir(), filename);
const writeStream = fs.createWriteStream(saveTo);
file.on('data', (data) => {
writeStream.write(data);
console.log(`Received: ${((bytes = data.length) / 1000000).toFixed(2)}MB`);
});
file.on('end', () => {
console.log('closing writeStream');
writeStream.close()
});
});
bb.on('close', () => {
console.log(`Actual size is ${(fs.statSync(fileObject.filePath).size / 1000000).toFixed(2)}MB`);
console.log('This is where the file would be uploaded to some cloud storage server...');
response.send('File was uploaded');
});
bb.on('error', (error) => {
console.log(error);
});
request.pipe(bb);
});