What is the best practice for split result rows before process it?-CodePudding

I am trying to create an email blaster for more than 100k emails at once using the Laravel framework.

My command:

public function handle()
{
    $blastId = $this->argument('blast');

    $blast = Blast::find($blastId);
    activity()->performedOn($blast)->log('Blast is sending');
    foreach ($blast->unsentLogs as $log) {
        try {
            Mail::queue((new BlastEmail($log))->onQueue('emails'));
        } catch (Exception $e) {
            activity('send_failed')->performedOn($log->contact)->causedBy($log)->log('Email send failed: '.$e->getMessage());
        }
    }
    $blast->status = 'sent';
    $blast->save();
    activity()->performedOn($blast)->log('Blast is sent');

    return 0;
}

my blast mail construct:

public function __construct(BlastLog $log)
{
    $contact = $log->contact;
    $blast = $log->blast;
    $this->content = $blast->content;
    $this->content = str_replace('**UNSUB**', 'https://urltounsubscribe.com', $this->content);
    $this->subject = $blast->subject;
    $this->to($contact->email, "Name");
    $this->from($blast->from_email, $blast->from_name);
    $log->sent_at = now();
    $log->save();
}

My code can handle under 1000 emails in 30 seconds, but why does it take a long time to send about 100k emails? It should only need 3000 seconds to handle 100k emails, but it's more than 3 hours and still not completed. How to improve my code?

I think the problem is $blast->unsentLogs, it has more than 100k collection rows, make it takes a lot amount of RAM.

Maybe I need to split $blast->unsentLogs to some parts/chunks. But what is the best practice for doing that? It should be using a query builder right? But I don't have idea to do that.

CodePudding user response：

Yes, your issue is clearly in $blast->unsentLogs. PHP cannot handle 100k collections, that is hell a lot...

You should chunk the results instead of getting them all at once...

Your code should be like:

public function handle()
{
    $blastId = $this->argument('blast');

    $blast = Blast::find($blastId);
    activity()->performedOn($blast)->log('Blast is sending');
    $blast->unsentLogs()->chunk(100, function (Collection $logs) {
        foreach ($logs as $log) {
            try {
                Mail::queue((new BlastEmail($log))->onQueue('emails'));
            } catch (Exception $e) {
                activity('send_failed')->performedOn($log->contact)->causedBy($log)->log('Email send failed: '.$e->getMessage());
            }
        }
    });
    $blast->status = 'sent';
    $blast->save();
    activity()->performedOn($blast)->log('Blast is sent');

    return 0;
}