NodeJS and MongoDB Query Slow-CodePudding

so running into an issue where I am using NodeJS with Express for API calls. I am fetching all documents in a collection using

    export async function main(req, res) {
    try {
        const tokens = await tokenModel.find({}).lean();
        res.json(tokens);
    } catch {(err) => {
        res.status(500).json({ message: err.message })
        console.log('err', err.message)
        }
    }
  console.log('Get Data')
}

Now this request works great and returns me the data I need. The problem is I have over 10K documents, and on a PC takes about 10 seconds to return that data, and on a mobile phone it takes over 45 seconds. I know network on phone matters, but is there any way I can increase this? Nothing I have tried works. I keep finding that lean is the option to use, and I am already using it with no success or improvements.

CodePudding user response：

Well, it's slow because you are returning all 10k results.

Do you actually need all 10k results? If not, you should consider filtering only results that you actually need.

If not, I suggest implementing pagination, where you would return results in batches (50 per page for example).

In addition, if you are using only some of the fields from the documents, you should tell MongoDB to return only these fields, and not all of them. That would also increase the performance since less data will be transferred through the network.

CodePudding user response：

It sounds like you are experiencing performance issues when fetching a large number of documents from your database. There are a few ways you can improve the performance of this operation:

Use pagination to limit the number of documents returned in each request. Instead of fetching all documents in a single request, you can use pagination to fetch the documents in smaller batches, which can improve the performance of the operation.

Use indexes to improve the performance of your queries. Indexes can help the database engine to more efficiently locate the documents that match your query, which can improve the performance of the operation.

Use projections to limit the amount of data returned in each document. By specifying which fields you want to include or exclude in the response, you can reduce the amount of data that needs to be transmitted, which can improve the performance of the operation.

Use caching to store the results of frequently-used queries. If you have certain queries that are used frequently, you can store the results of those queries in a cache, and retrieve the results from the cache instead of querying the database every time. This can significantly improve the performance of the operation.

In summary, there are several ways you can improve the performance of your database queries when fetching a large number of documents. By using pagination, indexes, projections, and caching, you can reduce the amount of data that needs to be transmitted and processed, which can improve the performance of your queries.