What I'm looking to do is to get content inside a div on another website (which I do not control) and display the content on my own website. For example, we will use these two sites:

• mysite.com
• othersite.com

On othersite.com, there is some text content inside a div that I want to fetch with javascript and update periodically (suppose, check every hour). For example, <div >9384</div>.

Now, on mysite.com, I want to output this result into my own div using javascript. I suppose this would effectively be web scraping.

To do this, I would like to save a variable which updates occasionally like every hour.

Using jquery, I could define a variable and pull the content. If it was on the same site, which it is not, I could do this:

/* HTML with content to fetch */
<div >9384</div>

/* get the content with jquery */
var getDiv = document.getElementsByClassName('some-div');
var getContent = getDiv.innerHTML;

If it was on the same site, then it would get the content inside the div and store it to the variable getContent. Here, getContent will equal 9384.

Then I could use it anywhere I want, like replacing the content of a blank div on mysite.com with the content fetched from the somediv, like this:

/* HTML where the content will be output */
<div ></div>   

/* replace the content with jquery form a stored variable */
var getMyDiv = document.getElementsByClassName('output-div');
getMyDiv.innerHTML = getContent.innerHTML; 

The result would be:

<div >9384</div>   

But, the content for the some-div div is not on the same URL (and I don't control the other site).

How do I do the above, but instead of fetching some-div from the current page or site, to instead fetch some-div content from an external site (that I do not control)?

CodePudding user response：

if you want to use jquery

$.get('http://othersite.com', function(data) {
    var $othersite = $(data);

    var getDiv = $othersite.find('.some-div');
    var getContent = getDiv.html();

    var getMyDiv = $('.output-div');
    getMyDiv.html(getContent);
});

if you want to use javascript without jquery

var xhr = new XMLHttpRequest();
xhr.open('GET', 'http://othersite.com', true);
xhr.responseType = 'document';
xhr.send();
xhr.onload = function(e) {  
    var othersite = e.target.responseXML;

    var getDiv = othersite.getElementsByClassName('some-div');
    var getContent = getDiv.innerHTML;
 
    var getMyDiv = document.getElementsByClassName('output-div');
    getMyDiv.innerHTML = getContent;
}

excuse any typo, i wrote this from my phone.

CodePudding user response：

In other not to worry about CORS, you can try these steps:

1. Make an AJAX request to your server via jquery/javascript.
2. Make a request from your server to othersite.com via your serverside language (eg. PHP).
3. You can then manipulate the AJAX response any how you like.

Example - Step 1 and Step 3

$.get('mysite.com/some-php-file.php', function(data) {
    let $othersiteHtml = $('<div />').html(data);

    let getSomeText = $othersiteHtml.find('.some-div').text();
    $('.output-div').html(getSomeText);
});

Step 2 - You can put below code in a PHP file on your server.

<?php

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "othersite.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);

echo $output;

?>