Situation:

I have a PHP script that attempts to merge 26 different RSS feeds and insert up to 1000 items from those feeds into a MySQL DB.

Hosting: Bluehost

MySQL Version: 5.7.23-23

PHP version via phpversion(): 8.0.17

PHP version via My Admin info:

• PHP: 7.4.30
• Database client version: libmysql
• mysqlnd 7.4.30
• PHP extension: mysqliDocumentation curlDocumentation mbstringDocumentation

Issue:

The script only inserts 26 RSS feed items, 1 item from each RSS feed into the DB, i.e. it does not insert ALL RSS items from the 26 feeds (up to 1000). I've used this same script to build an RSS feed and JSON file that includes all items from the 26 RSS items.

Question:

Can anyone help identify why this script isn't inserting all RSS items from all the RSS feeds as expected? Thank you for the help.

PHP Script Code:

<?php
$json = new DOMDocument();

$data = array();

$resources = array(
    array( 'type' => 'Article', 'source' => 'Source 1', 'feedurl' => 'http://www.example1.com/feed/', 'id' => '1' ),
    array( 'type' => 'Article', 'source' => 'Source 2', 'feedurl' => 'https://example2.com/feed', 'id' => '2' ),
    array( 'type' => 'Article', 'source' => 'Source 3', 'feedurl' => 'https://example3.com/feed', 'id' => '3' )
);

foreach ( $resources as $resource ) {
  $json->load( $resource['feedurl'] );

  foreach ( $json->getElementsByTagName( 'item' ) as $node ) {
  $item = array(
    'source'  => $resource['source'],
    'type'  => $resource['type'],
    'title' => $node->getElementsByTagName( 'title' )->item( 0 )->nodeValue,
    'link'  => $node->getElementsByTagName( 'link' )->item( 0 )->nodeValue,
    'date'  => $node->getElementsByTagName( 'pubDate' )->item( 0 )->nodeValue,
    'id'  => $resource['id']
  );

  array_push( $data, $item );
  }
}

usort( $data, function( $a, $b ) {
  return strtotime( $b['date'] ) - strtotime( $a['date'] );
});

$servername = '???';
$username = '???';
$password = '???';
$dbname = '???';

// Create connection
$DBconnection = new mysqli($servername, $username, $password, $dbname);
/* check connection */
if (mysqli_connect_errno()) {
    printf("Connect failed: %s\n", mysqli_connect_error());
    exit();
}

//sql query
$sql = "INSERT INTO articles(source, type, title, url, date, id) VALUES(?, ?, ?, ?, ?, ?)";

//insert into mysql table
$stmt = $DBconnection->prepare($sql);

$limit = 1000;

for ( $x = 0; $x < $limit; $x   ) {
    $source = $data[ $x ]['source'];
    $type = $data[ $x ]['type'];
    $title = htmlspecialchars(str_replace( ' & ', ' & ', $data[ $x ]['title'] ));
    $link = htmlspecialchars($data[ $x ]['link']);
    $date = date( 'Y-m-d H:i:s', strtotime( $data[ $x ]['date'] ) );
    $id = $data[ $x ]['id'];

    $stmt->bind_param("sssssi", $source, $type, $title, $link, $date, $id);

    $stmt->execute();
}

$stmt->close();
$DBconnection->close();

?>

CodePudding user response：

I rewrote the code and tested locally, it appears to be working:

$resources = [
  ['type' => 'Article', 'source' => 'Hackaday', 'feedurl' => 'https://hackaday.com/blog/feed/', 'id' => '1'],
  ['type' => 'Article', 'source' => 'The Daily WTF', 'feedurl' => 'http://syndication.thedailywtf.com/TheDailyWtf', 'id' => '2']
];

$dom = new DOMDocument();

$articles = [];

foreach ($resources as $resource) {
  $dom->load($resource['feedurl']);

  foreach ($dom->getElementsByTagName('item') as $node) {
    $articles[] = [
      'id' => $resource['id'],
      'source' => $resource['source'],
      'type' => $resource['type'],
      'title' => $node->getElementsByTagName('title')->item(0)->nodeValue,
      'link' => $node->getElementsByTagName('link')->item(0)->nodeValue,
      'date' => $node->getElementsByTagName('pubDate')->item(0)->nodeValue,
    ];
  }
}

usort($articles, function ($a, $b) {
  return strtotime($b['date']) - strtotime($a['date']);
});

$server = '???';
$user = '???';
$pass = '???';
$db = 'so';

$mysql = new mysqli($server, $user, $pass, $db);
$mysql->query('CREATE TABLE IF NOT EXISTS articles (source TEXT, type TEXT, title TEXT, url TEXT, date DATETIME, id INT);');

if (mysqli_connect_errno()) {
  printf("Connect failed: %s\n", mysqli_connect_error());
  exit;
}

$sql = "INSERT INTO articles (source, type, title, url, date, id) VALUES (?, ?, ?, ?, ?, ?);";

$stmt = $mysql->prepare($sql);

foreach ($articles as $article) {
  $source = $article['source'];
  $type = $article['type'];
  $title = htmlspecialchars($article['title']);
  $link = htmlspecialchars($article['link']);
  $date = date('Y-m-d H:i:s', strtotime($article['date']));
  $id = $article['id'];

  $stmt->bind_param("sssssi", $source, $type, $title, $link, $date, $id);
  $stmt->execute();
}

$stmt->close();
$mysql->close();

That adds all the records:

MariaDB [so]> SELECT COUNT(*) FROM articles;
 ---------- 
| COUNT(*) |
 ---------- 
|       22 |
 ---------- 
1 row in set (0.000 sec)