Query returning too much data for Laravel collection?-CodePudding

To avoid going to the db thousands of times for each user request to fetch their projects, I have refrained from using Eloquent relationships between the models that constitute a Project object. Originally, I had one vast SQL query to gather all needed data in one trip to the db, but am now breaking it down into two queries. But even this is apparently too much for the returned collection, which drops data silently (no error) after a certain point.

EDIT (added image): Screenshot of how "dropped data" appears to manifest in output from dd():

3 dots where a toggle-triangle should be

A Project object consists of, e.g:

PROJECT
 ->assigned to a Group or Team (Team belongs to Group)
 -> has many Tasks
  -> Tasks can have many Notes
  -> Tasks assigned to many one or more Members of Group or Team, or to whole Team (within assigned Group)
   -> Members of Tasks are either Users or Teams 
 -> Project can have many Notes (... etc.)

Suffice it to say that the SQL needed to gather all the required data to compile a user's projects returns much data. Excerpt from original single query:

$rawProjects = Project::where('projects.owner', $userId)
    ->where('projects.completed', false)
    ->leftJoin('tasks AS Ta1', function ($join) {
        $join->on('Ta1.project_id', '=', 'projects.id');
    })
    ->leftJoin('memberships AS Me1', function ($join) {
        $join->on('Me1.membershipable_id', '=', 'Ta1.id');
    })
    ->leftJoin('groups AS Gr1', function ($join) {
        $join->on('Gr1.id', '=', 'projects.group_id');
    })
    ->leftJoin('groups AS Gr2', function ($join) {
        $join->on('Gr2.id', '=', 'Ta1.taskable_id');
    })
    ->leftJoin('teams AS Te1', function ($join) {
        $join->on('Te1.id', '=', 'projects.team_id');
    })
    ->leftJoin('teams AS Te2', function ($join) {
        $join->on('Te2.id', '=', 'Ta1.taskable_id');
    }) // [cut]

I've just begun to divide this process, to fetch Project and Task data separately (in two queries), then combine the fetched data downstream. Sadly, the collection returned by the task-getting query alone is too much. The collection drops data silently; I'm not getting any error. What surprises me is that, according to debugbar, the data delivered to the VueJs page totals a paltry 2MB.

(I'm now going to recode the Eloquent queries to raw SQL and bypass Laravel collections.)

Question: Is this expected behaviour from Laravel collections? (It seems like quite a severe limitation to me.)

UPDATE Results from the raw SQL are the same, but it seems part of the problem is dd() itself, which seems to 'max out' at a certain point, confusing the issue somewhat. For example, print_r() returns all expected data. So the 3-dots-instead-of-toggle-triangle phenomenon is a red herring (see my comment in reply to apokryfos).

To expand on why I wrote "drops data" above: When I perform the query on a single project, the project is returned complete with all its data. When I fetch all the user's projects, data is missing from what is the 'bottom' – users assigned to the last project's task – of the array of Projects. This behaviour has nothing to do with dd() vs. print_r().

CodePudding user response：

For larger data sets use eager loading with eloquent relations and laravel Api resources, it's the best practice I think.. Here is reference docs Conditional Loading Relation

CodePudding user response：

My bad!

It was a bug in my code that was triggered by having the same user assigned to different tasks in 2 different projects, who then 'disappeared' from the query data when those 2 projects were captured by the query, only to 'reappear' when other combinations of projects were queried. Combined with the unexpected behaviour from dd(), I drew the wrong conclusion.

The fix was to reset/clear the array used to capture the users assigned to each task on each iteration of the relevant foreach() loop.