How to chain JavaScript/axios Promises, where the current promise determines if future promises are-CodePudding

I'm working on a node.js project where I need to use axios to hit an API. That API returns along with the data, a different URL (a query param changes in the URL) that I need to hit for every subsequent call, to paginate the data basically. That query param value is not predictable(it's not numbered "1 2 3 4").

I can't get all of the URLs all at once, I have to get just the next one with each request.

I need all of the data from all of the api calls.

How I'm thinking this needs to be done:

create an array
make a request to the api
push the response to the array
make another request to the API but with the query params from the prior call.
push the response to the array, repeat step 4 and 5 until API no longer gives a next URL.
Act on that received data after all the promises are complete. (example can simply be console logging all of that data)

I suppose since I'm needing to chain all of these requests I'm not really gaining the benefits of promises/async. So maybe axios and promises are the wrong tool for the job?

What I've tried:

I've tried inside the first axios().then() a while loop, executing axios requests until there are no more "next" links. That obviously failed because the while loop doesn't wait until the request is returned.
I've tried declaring a function which contains my axios request plus a .then(). Inside that .then() if a next URL was present in the results, I have the function call itself.
Looked into but didn't try promise.all()/axios.all() as that seemed to only work if you knew all of the URLs you were hitting upfront, so it didn't seem to apply to this scenario.
tried what was suggested in this answer, but seems the .then doesn't occur after all promises have been returned.

Example of that recursive function method:

const apiHeaders={
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${appToken}`,
    'Accept': 'application/json'
};
let initialURL = 'https://example.com/my-api';

let combinedResults =[];


function requestResultPage(pageURL){
    let options = {
        url:pageURL,
        method: 'GET',
        headers: apiHeaders,
        params:{
            limit:1
        }
    };
    axios(options)
    .then(function (response) {
        // handle success
        console.log("data response from API",response.data);

        
        combinedResults.push(response.data.results);

        if(typeof response.data.paging.next !== 'undefined'){
            console.log("multiple pages detected");
        
            let paginatedURL = response.data.paging.next.link;
            console.log("trying:",paginatedURL);
            requestResultPage(paginatedURL);
            

            
        }


    }).catch(function(error){
        console.log(error);
    })
};

requestResultPage(initialURL);
console.log(combinedResults);

I understand that the console log at the end won't work because it happens before the promises finish... So I've gotta figure that out. It seems though my loop of promises fails after the first promise.

Thinking in promises is still confusing to me at times, and I appreciate any wisdom folks are willing to share.

I suppose since I'm needing to chain all of these requests I'm not really gaining the benefits of promises/async. So maybe axios and promises are the wrong tool for the job? If that's the case feel free to call it out.

CodePudding user response：

Let's try this.

const apiHeaders={
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${appToken}`,
    'Accept': 'application/json'
};
let initialURL = 'https://example.com/my-api';

let combinedResults =[];


function requestResultPage(pageURL){
    let options = {
        url:pageURL,
        method: 'GET',
        headers: apiHeaders,
        params:{
            limit:1
        }
    };
    return axios(options)
    .then(function (response) {
        // handle success
        console.log("data response from API",response.data);

        
        combinedResults.push(response.data.results);

        if(typeof response.data.paging.next !== 'undefined'){
            console.log("multiple pages detected");
        
            let paginatedURL = response.data.paging.next.link;
            console.log("trying:",paginatedURL);
            return requestResultPage(paginatedURL);   
        }


    }).catch(function(error){
        console.log(error);
    })
};

requestResultPage(initialURL)
.then(() => {
    console.log(combinedResults);
});

CodePudding user response：

Ultimately at the end of the day I determined that axios was simply the wrong tool for the job.

If anyone else runs into this situation, maybe just don't use axios for this.

I switched to using node-fetch and it became very straightforward.

import fetch from 'node-fetch';


const appToken = "xxxxxxxxxxxxxxxx";
const apiHeaders={
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${appToken}`,
    'Accept': 'application/json'
};

let initialURL = 'https://api.example.com/endpoint';

initialURL = initialURL.concat("?limit=1");
console.log(initialURL);

let apiURL = initialURL;

let combinedResults =[];
let morePages = true;

while(morePages){
    console.log("fetching from",apiURL);
    const response = await fetch(apiURL, {
        method: 'get',
        headers: apiHeaders
    });
    let data = await response.json();
    console.log(data);

    combinedResults = combinedResults.concat(data.results);
    if(typeof(data.paging) !== 'undefined' && typeof(data.paging.next) !== 'undefined'){
        console.log("Another page found.")
        apiURL = data.paging.next.link;
        
    } else{
        console.log("No further pages, stopping.");
        morePages = false;
    }
}

console.log(combinedResults);

Appreciate the effort all invested in trying to help.

CodePudding user response：

The problem with your original code is that you don't wait on the axios call or the recursive calls to requestResultPage, so it exits requestResultPage before the sequence is complete. I prefer to avoid recursive calls if possible (sometimes they are a good choice), but for the sake of answering your question about why this approach failed, I'll continue with a recursive approach. Your while loop solution is a better approach.

Note that your original requestResultPage doesn't return anything at all. Normally a function that is dealing with async operations should return a Promise so that any callers can know when it's done or not. This can be done in your original code without too much trouble like this:

function requestResultPage(pageURL){
    let options = {
        url:pageURL,
        method: 'GET',
        headers: apiHeaders,
        params:{ limit: 1 }
    };
    return axios(options)   // <---- return the Promise here
    .then(function (response) {
        console.log("data response from API",response.data);
        
        combinedResults.push(response.data.results);

        if(typeof response.data.paging.next !== 'undefined'){
            console.log("multiple pages detected");
        
            let paginatedURL = response.data.paging.next.link;
            console.log("trying:",paginatedURL);
            return requestResultPage(paginatedURL);  // <---- and again here
        }
    })
};

requestResultPage(initialURL).then(() => {
    // now that all promises have resolved we have the full set of results
    console.log(combinedResults);
}).catch(function(error){   // <--- move the catch out here
    console.log(error);
});

This can also be done using async/await which makes it a bit cleaner. Declaring a function async means anything it returns is done as a resolved Promise implicitly, so you don't need to return the promises, just use await as you call anything that gives a Promise back (axios and requestResultPage itself).:

async function requestResultPage(pageURL){  // <--- declare it async
    let options = {
        url:pageURL,
        method: 'GET',
        headers: apiHeaders,
        params:{ limit: 1 }
    };
    const response = await axios(options)   // <---- await the response
    console.log("data response from API",response.data);
        
    combinedResults.push(response.data.results);

    if(typeof response.data.paging.next !== 'undefined'){
        console.log("multiple pages detected");
        
        let paginatedURL = response.data.paging.next.link;
        console.log("trying:",paginatedURL);
        await requestResultPage(paginatedURL);  // <---- and again here
    }
    // There's no return statement needed. It will return a resolved `Promise`
    // with the value of `undefined` automatically
};

requestResultPage(initialURL).then(() => {
    // now that all promises have resolved we have the full set of results
    console.log(combinedResults);
}).catch(function(error){
    console.log(error);
});

In your original implementation the problem was that you kicked off multiple axios calls but never waited for them, so the processing got to the final console.log before the calls completed. Promises are great but you do need to take care to make sure none slip through the cracks. Each one needs to either be returned so that it can have a .then() call after, or use await so we know it's resolved before continuing.

I like your solution using morePages and looping until there are no more. This avoids recursion which is better in my opinion. Recursion runs the risk of blowing the stack since the entire chain of variables are kept in memory until all calls complete which is unnecessary.

Note that by using await you need to make the function you're doing this in be async, which means it now returns a Promise, so even though you do the console.log(combinedResults) at the end, if you want to return the value to the caller, they'll also need to await your function to know it's done (or use .then() alternatively).