Home > Back-end >  JavaScript fetch HTML from a given link
JavaScript fetch HTML from a given link

Time:10-01

I'm trying to fetch HTML data from specific link(https://www.airforce.mil.kr/user/indexMain.action?siteId=last2) through JavaScript, but couldn't find any solution. This seems to be related with the CORS issue, since my browser says CORS error whenever I execute my works. Below is the list what I've tried:

  1. Fetch API → CORS error
fetch(url)
    .then(res => res.text())
    .then(text => console.log(text))
  1. Attached mode: 'no-cors' in fetch() → end up receiving empty response
fetch(url, {
        mode: 'no-cors'
    })
    .then(res => res.text())
    .then(text => console.log(text))
  1. cors-anywhere as a prefix of the url and remove mode: 'no-cors' → this works for https://www.google.com but not for the link above, additionally requires regular registration by clicking button manually
  2. Looked up for a substitute → doesn't require 'button clicking', but still doesn't work for my link
  3. Set up a React project and import puppeteer module → even importing entails chains of error
  4. Tried BeautifulSoup of Python experimentally → only this works for me
import requests
from bs4 import BeautifulSoup
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

url = f'https://www.airforce.mil.kr/user/indexMain.action?siteId=last2'
response = requests.get(url, verify = False)
if response.status_code == 200:
    html = response.text
    soup = BeautifulSoup(html, 'html.parser')
    print(soup)
else:
    print(response.status_code)

This makes me confused and I hope to know the reason and solution for the problem if possible. Please excuse me for potentially awkward expressions, since I'm not used to writing question in this place...

CodePudding user response:

It's a browser security setting, the chrome plugin mod header allows you modify the allow cross origin setting but it will only work for you.

You could fetch the html from the URL using python and then pass it through an API so it can be used in JS.

  • Related