Requesting URL's one by one-CodePudding

I'm trying to get some data from a lot of URLs using 'request', but I can't manage to do it one url at a time.

I've tried to understand async/promises in order to make it work, but with no success. Trial and error didn't work.

I've seen other methods using different modules, but this requires rewriting most of my code and I believe there is an easier way to adapt it to my current code.

Here is a minimized version of the code :

const request = require('request');
const fs = require('fs');
const prod = fs.readFileSync('prod.txt', "utf8");
const prodid = prod.split("|");
var i;
var summary=[];


for (i=0;i<prodid.length;i  ){
request('https://www.api.example.com/id=' prodid[i], { json: true }, (err, res, body) => {
  if (err) { return console.log(err); }
if (body == 'NULL') {
   console.log("Page "   i   " out of "   prodid.length   " is NULL!");
} else {
summary.push(body.items[0].Name);
summary.push(body.items[0].ISOnr);
summary.push(body.items[0].GTIN);
console.log("Page "   i   " out of "   prodid.length   " is done!");
fs.appendFileSync('data.txt',JSON.stringify(summary));
}

});
}

There is no async/promise involved in the example above, just the requests inside a loop.

From what I've seen, when I get the results, there is no particular order (probably is the order of which finishes first).

In the console, I always see page 500 out of 500, not 1/500, 2/500, etc.

What I'm trying to achieve, is by making each request in the order of URLs (preferably with a 1000ms delay between them)

CodePudding user response：

You can promisify your request:

for (i = 0; i < prodid.length; i  ) {
    const result = await new Promise((resolve, reject) =>
        request(
            'https://www.api.example.com/id='   prodid[i],
            { json: true },
            (err, res, body) => {
                if (err) {
                    reject(err);
                }
                if (body == 'NULL') {
                    console.log('Page '   i   ' out of '   prodid.length   ' is NULL!');
                } else {
                    resolve(body);
                }
            }
        )
    );
    if (result) {
        summary.push(result.items[0].Name);
        summary.push(result.items[0].ISOnr);
        summary.push(result.items[0].GTIN);
        console.log('Page '   i   ' out of '   prodid.length   ' is done!');
        fs.appendFileSync('data.txt', JSON.stringify(summary));
    }
}

CodePudding user response：

Note: The request module has been deprecated, you should no longer use it (see https://github.com/request/request/issues/3142).

With that being said ... you can wrap the request in a promise and await this promise inside your loop. Something like (still needs error-handling of course):

for (const currentProdId of prodid) {
    // requests will now be sent one after the other
    await sendRequest(currentProdId);
}

function sendRequest(prodId) {
    return new Promise((resolve, reject) => {
        request('https://www.api.example.com/id='   prodId, {json: true}, (err, res, body) => {
            if (err) {
                console.log(err);
                return reject(err);
            }
            // ... rest of your code, make sure to resolve the promise once you're done
            resolve();
        });
    });
}