Node.js Express Temporary File Serving-CodePudding

I'm trying to do a reverse image search using googlethis on an image the user uploads. It supports reverse image searching, but only with a Google-reachable image URL. Currently, I upload the image to file.io, which deletes it after it gets downloaded.

This is the current application flow:

User POSTs file -> Server uploads file to file.io -> Google downloads the file -> Server does things with the reverse image search

However, I want to skip the middleman and have Google download files directly from the server:

User POSTs file -> Server serves file at unique URL -> Google downloads the file -> Server deletes the file -> Server does things with the reverse image search

I've looked at Serving Temporary Files with NodeJs but it just shows how to serve a file at a static endpoint. If I added a route to /unique-url, the route would stay there forever (a very slow memory leak! Probably! I'm not really sure!)

The only way I can think of is to save each file with a UUID and add a parameter: /download?id=1234567890, which would probably work, but if possible, I want to do things in memory.

So:

How do I do this using normal files?
How do I do this in-memory?

Currently working (pseudo) code:

app.post('/', (req, res) => {
  const imagePath = saveImageTemporarily(req)
  const tempUrl = uploadToFileIo(imagePath)
  const reverseImageResults = reverseGoogleSearch(tempUrl)
  deleteFile(imagePath)
  doThingsWithResults(reverseImageResults).then((result) => { res.send(result) })
}

CodePudding user response：

The other answer is a good one if you are able to use Redis -- it offers lots of helpful features like setting a time-to-live on entries so they're disposed of automatically. But if you can't use Redis...

The basic idea here is that you want to expose a (temporary) URL like example.com/image/123456 from which Google can download an image. You want to store the image in memory until after Google accesses it. So it sounds like there are two (related) parts to this question:

Store the file in memory temporarily

Rather than saving it to a file, why not create a Buffer holding the image data. Once you're done with it, release your reference to the buffer and the Node garbage collector will dispose of it.

let image = Buffer.from(myImageData);
// do something with the image
image = null; // the garbage collector will dispose of it now

Serve the file when Google asks for it

This is a straightforward route which determines which image to serve based on a route parameter. The query parameter you mention will work, and there's nothing wrong with that. Or you could do it as a route parameter:

app.get('/image/:id', (req, res) => {
  const id = req.params.id;
  res.status(200).send(/* send the image data here */);
});

Putting it all together

It might look something like this:

// store image buffers here
const imageStore = {};

app.post('/image', (req, res) => {
  // get your image data here; there are a number of ways to do this,
  // so I leave it up to you
  const imageData = req.body;
  // and generate the ID however you want
  const imageId = generateUuid();
  // save the image in your store
  imageStore[imageId] = imageData;
  // return the image ID to the client
  res.status(200).send(imageId);
});

app.get('/image/:id', (req, res) => {
  const imageId = req.params.id;
  // I don't know off the top of my head how to correctly send an image
  // like this, so I'll leave it to you to figure out. You'll also need to
  // set the appropriate headers so Google recognizes that it's an image
  res.status(200).send(imageStore[imageid]);
  // done sending? delete it!
  delete imageStore[imageId];
});

CodePudding user response：

I would use REDIS for the in-memory DB, and on the server, I would transform the image to base64 to store it in Redis. In Redis, you can also set TTL on the images.

Check my code below

import {
  nanoid
} from 'nanoid'

function base64_encode(file) {
  // read binary data
  var bitmap = fs.readFileSync(file);
  // convert binary data to base64 encoded string
  return new Buffer(bitmap).toString('base64');
}

app.post('/', async(req, res) => {
      const client = redisClient;
      const imagePath = saveImageTemporarily(req)
      //const tempUrl = uploadToFileIo(imagePath)
      var base64str = base64_encode(imagePath);
      const id = nanoid()
      await client.set(id, JSON.stringify({
        id,
        image: base64str
      }));
      const reverseImageResults = reverseGoogleSearch(JSON.parse(await client.get(id)).image)
      await client.del(id);
      doThingsWithResults(reverseImageResults).then((result) => {
        res.send(result)
      })
    }