I was wondering what would be a good way to retrieve large amounts (20000 rows) of data with Laravel 8.

I'm working with a Laravel 8 REST API. One of the endpoints (when unfiltered) gets all the rows in a table, along with some model relations. Then, we have an Angular app that uses all of that data and transforms it into an Excel file for the client to use.

The problem is, the database keeps growing and growing, and if our client wants to download the whole table, the app runs out of memory.

For example

if($request->has('country_id')) {
        $users->where('country_id', $request->country_id);
    }

return $users->with($relations)->get(['id', 'first_name', 'last_name', 'email' ...]);

Assuming the request sometimes needs the whole table unfiltered, what would be a better way to approach sending a lot of data over a REST API?

Thanks!

CodePudding user response：

1. you should use pagination.
2. other than you can use ->toBase(), it will reduce memory dramatically.
3. if you want to work with many rows and don't need all columns probably, you can use ->select to select proper columns.

CodePudding user response：

I think the best way to handle this would be utilizing Laravel's Lazy Collections, or maybe even Cursors. You can then paginate or do whatever you want in your Angular app. Make sure you are only pulling out the attributes you want in the call. e.g using a cursor:

$users = $request->has('country_id')) ?
  User::cursor()
    ->filter( function($user) use ( $request->country_id ){
        return $user->country_id === $request->country_id
    })
    ->pluck('id', 'first_name','last_name', 'email' ) : 
  null;

If you are needing to load relations, then you'll need to use a Lazy Collection.