I am using the code given below to run multiple https GET request for Wikipedia API.

app.get("/data_results", (req, res) => {

    const articlesData = names.map(nameObj => {
        let name = nameObj.name;
        let articleExtract = "";
        let contentURL =
            `https://en.wikipedia.org/w/api.php? 
 action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;

        // Getting article content
        https.get(contentURL, response => {
            response.on("data", async data => {
                const wikiArticle = JSON.parse(data);
                // Mapping the keys of the JSON data of query to its values.
                articleExtract = await Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
                nameObj.article = articleExtract.substring(0,350);
            })
        });

        return nameObj;
    });

    res.send(articlesData);});

This is my names array

[
  { name: 'Indus Valley Civilisation' },
  { name: 'Ramayana' },
  { name: 'Mahabharata' },
  { name: 'Gautama Buddha' }
]

My aim :-

1. Run the HTTPS GET request for every value in the names array sequentially.
2. Add the article extract with its name and save in it in an objects array.

You could also suggest me better ways of doing this.

CodePudding user response：

I would suggest you use fetch and promise. all. Map over your array of names and create promises as input. and then return the array of resolved promises.

something like this

const namePromises = [
      { name: 'Indus Valley Civilisation' },
      { name: 'Ramayana' },
      { name: 'Mahabharata' },
      { name: 'Gautama Buddha' }
    ].map((value) => fetch(baseurl   value.name).then(res=>res.json()));

    Promise.all(namePromises).then((response) => {
      console.log(response);
    });

CodePudding user response：

Try this solution. it creates an object like

{
Mahabharata: response   
}

The code

const getValues = async () => {
  const name_array = [
    {name: 'Indus Valley Civilisation'},
    {name: 'Ramayana'},
    {name: 'Mahabharata'},
    {name: 'Gautama Buddha'},
  ];
  const namePromises = name_array.map(value =>
    fetch('baseurl'   value.name).then(res => res.json()),
  );
  const response = await Promise.all(namePromises);
  return response.reduce((obj, responseResults, index) => {
    obj[name_array[index].name] = responseResults;
  }, {});
};

CodePudding user response：

Problems:

So, you have several things going on here.

1. http.get() is non-blocking and asynchronous. That means that the rest of your code continues to run while your multiple http.get() operations are running. That means yo uend up calling res.send(articlesData);}); before any of them are complete.

2. .map() is not promise-aware or asynchronous-aware so it will not wait its loop for any asynchronous operation. As such, you're actually running ALL your http requests in parallel (they are all in-flight at the same time). The .map() loop starts them all and then they all finish some time later.

3. The data event for http.get() is not guaranteed to contain all your data or even a whole piece of data. It may just be a partial chunk of data. So, while your code may seem to get a complete response now, different network or server conditions could change all that and reading the result of your http requests may stop functioning correctly.

4. You don't show where names comes from in your request handler. If it's a statically declared array of objects declared in a higher-scoped variable, then this code has a problem that you're trying to modify the objects in that array in a request handler and multiple requests to this request handler can be conflicting with one another.

5. You don't show any error handling for errors in your http requests.

6. You're using await in a place it doesn't belong. await only does something useful when you are awaiting a promise.

Solutions:

Managing asynchronous operations in Javascript, particularly when you have more than one to coordinate) is a whole lot easier when using promises instead of the plain callback that http.get() uses. And, it's a whole lot easier to use promises when you use an http request interface that already supports promises. My goto library for built-in promise support for http requests in nodejs is got(). There are many good choices shown here.

You can use got() and promises to control the asynchronous flow and error handling like this:

const got = require('got');

app.get("/data_results", (req, res) => {

    Promise.all(names.map(nameObj => {
        let name = nameObj.name;
        // create separate result object
        let result = { name };
        let contentURL =
            `https://en.wikipedia.org/w/api.php?
 action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;

        return got(contentURL).json().then(wikiArticle => {
            // Mapping the keys of the JSON data of query to its values.
            let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
            result.article = articleExtract.substring(0, 350);
            return result;
        });
    })).then(articlesData => {
        res.send(articlesData);
    }).catch(err => {
        console.log(err);
        res.sendStatus(500);
    });
});

This code attempts to fix all six of the above mentioned problems while still running all your http requests in parallel.

If your names array was large, running all these requests in parallel may consume too many resources, either locally or on the target server. Or, it may run into rate limiting on the target server. If that was an issue, you can run sequence the http requests to run them one at a time like this:

const got = require('got');

app.get("/data_results", async (req, res) => {

    try {
        let results = [];
        for (let nameObj of names) {
            let name = nameObj.name;
            let result = { name };
            let contentURL =
                `https://en.wikipedia.org/w/api.php?
     action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;

            const wikiArticle = await got(contentURL).json();
            let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
            result.article = articleExtract.substring(0, 350);
            results.push(result);
        }
        res.send(results);
    } catch (e) {
        console.log(e);
        res.sendStatus(500);
    }

});