Using file_put_contents inside of a foreach loop only downloads the last item-CodePudding

I am trying to download a large list of mp4 files by looping through them and using file_put_contents() to save to a directory. The problem is that only the last item in the video list is getting downloaded.

Here is my code:

<?php
$i = 0;
foreach ($response['videos'] as $row){
    $i  ;
    if($row['status'] != 'failed') {
        $videoId= '';
        $videoName = '';
        $videoId = $row['key'];
        $videoName = $row['title'];
        $filename = '';
        $filename = str_replace(' ','-',$videoName); // remove spaces from filename created by name
        

        // Initialize a file URL to the variable
        $url = "";
        $url = "http://content.jwplatform.com/videos/{$videoId}.mp4";
          
        // Use file_get_contents() function to get the file
        // from url and use file_put_contents() function to
        // save the file
        if (file_put_contents("Videos/".$filename.$i.".mp4", file_get_contents($url)))
        {
            echo "File downloaded successfully.";
            //sleep(5);
        }
        else
        {
            echo "File downloading failed.";
        }
        
    }
}
?>

I tried to use a CURL function to do this instead of file_put_contents() and it successfully placed all of the files to my Videos directory, but they were all empty files. I believe they were empty because these mp4 URLs are secure videos, so when you open them in the browser they actually bring you to a different secure URL to view and download the video. The CURL function could not get the file data successfully, but it seems like file_get_contents() does get it successfully (only the last item though).

In my code above, I believe what is happening is the variables in the loop are getting overridden over and over until it reaches the last item and then it executes the file_put_contents() function. If that is the case, how can I ensure that it executes the function on each loop so all of the files are downloaded?

Edits: Here is some of the output of var_export($response['videos'])

array ( 0 => array ( 'key' => 'eewww123', 'title' => 'Video Name Example 1', 'description' => NULL, 'date' => 1604004019, 'updated' => 1640011490, 'expires_date' => NULL, 'tags' => NULL, 'link' => NULL, 'author' => NULL, 'size' => '240721720', 'duration' => '229.79', 'md5' => 'f0023423423423423423', 'views' => 0, 'status' => 'ready', 'error' => NULL, 'mediatype' => 'video', 'sourcetype' => 'file', 'sourceurl' => NULL, 'sourceformat' => NULL, 'upload_session_id' => NULL, 'custom' => array ( ), ), 1 => array ( 'key' => 'rr33445', 'title' => 'Another Video Name Example 1', 'description' => '', 'date' => 1594316349, 'updated' => 1640011493, 'expires_date' => NULL, 'tags' => NULL, 'link' => '', 'author' => NULL, 'size' => '525702235', 'duration' => '840.90', 'md5' => '0044455sfsdgsdfs3245', 'views' => 0, 'status' => 'ready', 'error' => NULL, 'mediatype' => 'video', 'sourcetype' => 'file', 'sourceurl' => NULL, 'sourceformat' => NULL, 'upload_session_id' => NULL, 'custom' => array ( ), ), )

None of the rows have a failed status, and there are about 30 rows in total but I have some other video lists to download with 900 rows.

I enabled error reporting and I see

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 132120608 bytes)

on the line where my file_put_contents() function is.

Here is the CURL function I used that worked to download all of the filenames successfully but all of the files were empty:

    function multiple_download(array $urls, $save_path = 'Videos') {
    $multi_handle = curl_multi_init();
    $file_pointers = [];
    $curl_handles = [];

    // Add curl multi handles, one per file we don't already have
    foreach ($urls as $key => $url) {
        $file = $save_path . '/' . basename($url);
        if(!is_file($file)) {
            $curl_handles[$key] = curl_init($url);
            $file_pointers[$key] = fopen($file, "w");
            curl_setopt($curl_handles[$key], CURLOPT_FILE, $file_pointers[$key]);
            curl_setopt($curl_handles[$key], CURLOPT_HEADER, 0);
            curl_setopt($curl_handles[$key], CURLOPT_CONNECTTIMEOUT, 60);
            curl_multi_add_handle($multi_handle,$curl_handles[$key]);
        }
    }

    // Download the files
    do {
        curl_multi_exec($multi_handle,$running);
    } while ($running > 0);

    // Free up objects
    foreach ($urls as $key => $url) {
        curl_multi_remove_handle($multi_handle, $curl_handles[$key]);
        curl_close($curl_handles[$key]);
        fclose ($file_pointers[$key]);
    }
    curl_multi_close($multi_handle);
}



multiple_download($videoURLS);

$videoURLs is an array that I built containing all the unique URLs using the first PHP function above (with the other part commented out).

CodePudding user response：

It turns out the issue was that file_get_contents was exhausting the memory size. From this post, I used the following function

function custom_put_contents($source_url='',$local_path=''){

    $time_limit = ini_get('max_execution_time');
    $memory_limit = ini_get('memory_limit');

    set_time_limit(0);
    ini_set('memory_limit', '-1');      

    $remote_contents=file_get_contents($source_url);
    $response=file_put_contents($local_path, $remote_contents);

    set_time_limit($time_limit);
    ini_set('memory_limit', $memory_limit); 

    return $response;
}

This effectively sets the memory to unlimited so the file can be retrieved and then it restores the memory back to original state after it is done. With this function I was able to download the files.

CodePudding user response：

You must use a flag to append to the file instead of overwrite.

See documentation https://www.php.net/manual/fr/function.file-put-contents.php Flag FILE_APPEND

Edit: if all file have the same name, it is possible it overwrite them. You must provide diferent name in your loop.

foreach ($response['videos'] as $key => $row) {
    ...
    if (file_put_contents("Videos/" . $filename .$key ".mp4", file_get_contents($url))) {
   ...

Using the $key of the loop in your file name make it uniq and will not be overwritten