I have a situation where I have to fetch data from multiple apis for a variable amount of times in order to get all the data into a nested format. Basically I have to fetch a list of my available groups from one api, and then individually fetch the data for each of those groups, and repeat for any of the children groups. The solution I came up with uses recursion to fetch data and add nested properties to the original array from the first request:

fetch("/mygroups")
  .then((res) => res.json())
  .then((data) => {
    return data.map((group) => ({ value: group.id, label: group.name, children: [] }));
  })
  .then((groups) => {
    groups.forEach((group) => {
      fetchGroupData(group);
    });
    return groups;
  })
  .then((groups) => {
    // I need this groups variable to be the final groups variable with all it's
    // nested children
    functionToCallAfterAllRequestsAreMade(groups);
  });

async function fetchGroupData(group) {
  const res = await fetch(`/groups/${group.value}`);
  const data = await res.json();
  // A group can have locations and/or more groups as its children.
  // if it has groups, it will call fetchGroupData for those
  // child groups
  const locations = data.locations.map((location) => ({
    value: location.id,
    label: location.location_name,
  }));
  const groups = data.groups.map((group) => ({
    value: group.id,
    label: group.name,
    children: [],
  }));
  group.children = [...groups, ...locations];
  if (groups.length > 0) {
    group.children.forEach((child) => {
      if (child.hasOwnProperty("children")) {
        fetchGroupData(child);
      }
    });
  }
}

The problem is that this code appears to call fetchGroupData for the initial groups array and adds their children, but then returns only one layer deep, without waiting for the next layer of calls to keep going. The fetchGroupData function keeps getting called the correct amount of times, but I only have access to the array from the first round of calls for some reason. How can I wait until all of the calls are finished? I've tried adding a bunch of awaits everywhere and even using promise.all for the forEach with no luck. Also, if there's an entirely different way to go about this formatting problem that would be easier, that would be much appreciated too. Thanks.

CodePudding user response：

The problem with your current code is that you have multiple calls to fetchGroupData() that you aren't either chaining into an existing promise or awaiting. Thus, they end up as independent promise chains that have no connection at all to the .then() where you want to examine the final data.

So, to fix it, you have to chain all new promises into an existing promise chain so they are all connected. There are multiple ways to do it. Since you appear to be passing around one common data structure that everyone is working on groups, I chose the method that uses await on all other asynchronous calls to force everything to be sequenced. It might be possible to parallelize the two loops and use Promise.all(), but the way you are modifying the single groups data structure in each sub-call means you can't preserve any order so it seems safer not to run things in parallel.

Note, I also got rid of the .forEach() loops since they are not promise-aware and I changed the to a regular for loop which is promise-aware and will pause its iteration with await.

fetch("/mygroups")
    .then((res) => res.json())
    .then(async (data) => {
        const groups = data.map((group) => ({ value: group.id, label: group.name, children: [] }));
        for (let group of groups) {
            await fetchGroupData(group);
        }
        return groups;
    })
    .then((groups) => {
        // I need this groups variable to be the final groups variable with all it's
        // nested children
        functionToCallAfterAllRequestsAreMade(groups);
    });

async function fetchGroupData(group) {
    const res = await fetch(`/groups/${group.value}`);
    const data = await res.json();
    // A group can have locations and/or more groups as its children.
    // if it has groups, it will call fetchGroupData for those
    // child groups
    const locations = data.locations.map((location) => ({
        value: location.id,
        label: location.location_name,
    }));
    const groups = data.groups.map((group) => ({
        value: group.id,
        label: group.name,
        children: [],
    }));
    group.children = [...groups, ...locations];
    if (groups.length > 0) {
        for (let child of group.children) {
            if (child.hasOwnProperty("children")) {
                await fetchGroupData(child);
            }
        }
    }
}

Changes:

1. Combine code from two .then() handlers since one contained purely synchronous code. Make one .then() handler async so we can use await in it which means it will return a promise which will be hooked into this promise chain by virtue of returning it from a .then() handler.
2. Change groups.forEach(...) to for (let group of groups) { ... } so we can use await and pause the loop.
3. Change fetchGroupData(group); to await fetchGroupData(group);
4. In fetchGroupData() change group.children.forEach((child) => { ...}) to for (let child of group.children) { ... }); again so we can use await to pause the loop.
5. In fetchGroupData() change fetchGroupData(group); to await fetchGroupData(group);

This makes sure that the promise that fetchGroupData() returns does not resolve until all recursive calls to it have resolved. And, it makes sure that those calling it are also watching the promise it returns and don't advance until that promise resolves. This hooks everything together into one giant promise chain (and await chain) so things are properly sequenced and you can call functionToCallAfterAllRequestsAreMade(groups); when everything is done.

CodePudding user response：

I would suggest breaking your function down into separate and manageable parts -

function getJson(url) {
  return fetch(url).then(r => r.json())
}

We write fetchGroups (plural) to fetch all groups -

async function fetchGroups(url) {
  const groups = await getJson(url)
  return Promise.all(groups.map(fetchGroup))
}

Where fetchGroup (singular) fetches a single group -

async function fetchGroup({ id, name, locations = [], groups = [] }) {
  return {
    value: id,
    label: name,
    children: [
      ...locations.map(l => ({ value: l.id, label: l.location.name })),
      ...await fetchGroups(`/groups/${id}`)
    ]
  }
}

This arrangement of functions is called mutual recursion and is a perfect fit for creating, traversing, and manipulating recursive structures, such as the tree in your question.

Now we simply call fetchGroups on your initial path -

fetchGroups("/mygroups")
  .then(console.log, console.error)