Home > Mobile >  Retrieve title from multiple web pages with node.js and send as a response
Retrieve title from multiple web pages with node.js and send as a response

Time:06-21

/I/want/title/?address=google.com&address=www.dawn.com/events/

I have multiple query param in a URL and I want to fetch the title from each and show them as a list inside an HTML page. My question is how can I store the parsed response outside this map so I can somehow make a list and send back a response.

const parseTitle = (body) => {
  let match = body.match(/<title>(.*?)<\/title>/);
  if (!match || typeof match[1] !== "string") "Title Not FOund";
  return match[1];
};

app.get("/I/want/title/", (req, res) => {
  const query = url.parse(req.url, true).query;
  if (!query.address) {
    res.status(400).send("Missing url query parameter");
  }
  const list = typeof query.address === "string" ? [query.address] : query.address;
  list.map((address) => {
    axios
      .get(address)
      .then((res) => res.data)
      .then((body) => {
        console.log(`${address} -- ${parseTitle(body)}`);
      })
      .catch((error) => {
        console.error(`${address} --  NO RESPONSE`);
      });
  });

  res.send();
});

CodePudding user response:

I expanded upon @Leon Braun answer :

const express = require('express');
const url = require('url');
const axios = require('axios');
const app = express();

const parseTitle = (body) => {
  let match = body.match(/<title>(.*?)<\/title>/);
  if (!match || typeof match[1] !== "string") "Title Not FOund";
  return match[1];
};

app.get("/I/want/title/", (req, res) => {
  const results = [];
  const query = url.parse(req.url, true).query;
  if (!query.address) {
    res.status(400).send("Missing url query parameter");
  }
  const list = typeof query.address === "string" ? [query.address] : query.address;
  list.map((address) => {
    results.push(axios.get(address).then((res) => {
      return `${address} -- ${parseTitle(res.data)}`
    }));
  });

  let response = '';
  Promise.all(results).then((resolved) => {
    resolved.forEach((s) => {
      response = response   s;
    })
    res.send(response);
  })
});

app.listen(3000);

You use then to return the data you want and wait for all promises with Promise.all, and you only send the response when you have your data

CodePudding user response:

Just define a array above your list.mapstatement and push the results into it.

const results = [];
// Inside your list.map()
results.push('A title');

Edit: @LK77 mentioned that you have to await the axios request. I would suggest that you read about async/await syntax first. Always remember, every function that returns a promise will be executed async. That means, if you need the data, you have to wait for it.

  • Related