Home > Back-end >  Fetch data from huge amount fetched URLs
Fetch data from huge amount fetched URLs

Time:07-01

Not sure how to go about this. I fetch from 1 URL to get an array of objects like this:

[{name: 'apple', url: 'appleURL'}, {name: 'orange', 'orangeURL', ...}]

then map through each URL to fetch data, the returned data for each item has another URL... like this:

[{name: 'apple', colors: ['green', 'red'], type: 'http://URL-that-needs-fetching'},{...etc}]

heres what i have, and it works but it's extremely slow...

  useEffect(() => {
    const fetchFruit = async () => {
      setLoading(true);
      const fruitResponse = await fetch(fruitPath);
      const fruitJson: Fruit =
        await fruitResponse.json();

      const fruitDetails = await Promise.all(
        fruitJson.results.map(async (f: FruitURLs) => {
          const fDetails = await fetch(f.url);
          const json: FruitDetails = await fDetails.json();

          return json;
        })
      );

      const getFruitDescriptions = await Promise.all(
        fruitDetails.map(async (item: FruitDetails, index: number) => {
          const fruitType = await fetch(item.type.url);
          const json: FruitInfo = await fruitType.json();
          return json;
        })
      );

      let full: FruitDetails[] = fruitDetails;

      for (let index = 0; index < fruitDetails.length; index  ) {
        full[index].fruitInfo = getFruitDescriptions[index];
      }
      setFruitDetails(full);
      setLoading(false);
    };

    fetchFruit();
  }, []);

I'm new to using promises and not 100% confident with them, also new to TS, and react-native so apologies if there's mistakes here....

CodePudding user response:

The problem is that by using await inside Promise.all(), you are resolving the fetches instead of having them happen in parallel and returning it wrapped by a promise.

You can take a reference:

const fetchFruit = async () => {
  const fruitResponse = await fetch(fruitPath);
  const fruitJson: Fruit = await fruitResponse.json();

  const fruitDetails = await Promise.all(
    fruitJson.results.map(async (f: FruitURLs) => fetch(f.url))
  );

  const getFruitDescriptions = await Promise.all(
    fruitDetails.map(async (item: FruitDetails, index: number) =>
      fetch(item.type.url)
    )
  );

  const [fruitDetailsAsJson, getFruitDescriptionsAsJson] = Promise.all([
    fruitDetails.map(async (f) => f.json()),
    getFruitDescriptions.map(async (f) => f.json()),
  ]);

  const full = fruitDetailsAsJson.map((fruit, index) => ({
    ...fruit,
    fruitInfo: getFruitDescriptionsAsJson[index],
  }));

  setFruitDetails(full);
  setLoading(false);
};

Use this resource to learn about the best practice.

CodePudding user response:

If there are a lot of entries in the fruitresponse, and the back end cannot aggregate all the data, then it is inevitable to be slow. At this time, we should consider loading a little and displaying a little

const FruitList = ()=> {
  const { data, loading } = useFetch(fruitPath);
  if (loading) return <div>Loading...</div>
  return data.results.map(x=> <Fruit key={x.id} data={x} />)
}
const Fruit = ({data}: { data: FruitDetails })=> {
  const { data, loading } = useFetch(data.url);
  if (loading) return <div>Loading...</div>;
  return (
     <div>
        {data.name}
        <FruitDescription url={data.type.url} />
     </div>

  )
}
const FruitDescription = ({url}: { url: string })=> {
  const { data, loading } = useFetch(url);
  if (loading) return <div>Loading...</div>;
  return <div>{data.description}</div>
}
  • Related