Golang HTTP Get Request Not Resolving for some URL-CodePudding

I was trying to build some sort of website status checker. I figure out that the golang HTTP get request is not resolved and hung forever for some URL like https://www.hetzner.com. But the same URL works if we do curl.

Golang

Here there is no error thrown. It just hangs on http.Get

func main() {
  resp, err := http.Get("https://www.hetzner.com")
  if err != nil {
        fmt.Println("Error while retrieving site", err)
  }
  defer resp.Body.Close()
  body, err := io.ReadAll(resp.Body)
    if err != nil {
      fmt.Println("Eroor while reading response body", err)
  }
  fmt.Println("RESPONSE", string(body))}

CURL

I get the response while running following command.

curl https://www.hetzner.com

What may be the reason? And how do I resolve this issue from golang HTTP?

CodePudding user response：

Your specific case can be fixed by specifying HTTP User-Agent Header:

import (
    "fmt"
    "io"
    "net/http"
)

func main() {
    client := &http.Client{}

    req, err := http.NewRequest("GET", "https://www.hetzner.com", nil)
    if err != nil {
        fmt.Println("Error while retrieving site", err)
    }

    req.Header.Set("User-Agent", "Golang_Spider_Bot/3.0")

    resp, err := client.Do(req)
    if err != nil {
        fmt.Println("Error while retrieving site", err)
    }

    defer resp.Body.Close()
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        fmt.Println("Eroor while reading response body", err)
    }
    fmt.Println("RESPONSE", string(body))
}

Note: many other hosts will reject requests from your server because of some security rules on their side. Some ideas:

Empty or bot-like User-Agent HTTP header
Location of your IP address. For example, online shops in the USA don't need to handle requests from Russia.
Autonomous System or CIDR of your provider. Some ASNs are completely blackholed because of the enormous malicious activities from their residents.

Note 2: Many modern websites have DDoS protection or CDN systems in front of them. If Cloudflare protects your target website, your HTTP request will be blocked despite the status code 200. To handle this, you need to build something able to render JavaScript-based websites and add some scripts to resolve a captcha.

Also, if you check a considerable amount of websites in a short time, you will be blocked by your DNS servers as they have some inbuild rate limits. In this case, you may want to take a look at massdns or similar solutions.