Verify URL exists cloudflare snag-CodePudding

I am using the function below to determine if a URL exists:

Public Function URLExists(ByVal url As String) As Boolean
    Dim webRequest As System.Net.WebRequest = System.Net.WebRequest.Create(url)
    webRequest.Method = "HEAD"
    Try
        Dim response As System.Net.HttpWebResponse = CType(webRequest.GetResponse, System.Net.HttpWebResponse)
        If (response.StatusCode.ToString = "OK") Then
            Return True
        End If

        Return False
    Catch
        Return False
    End Try
End Function

For the most part this works as it should, but when a site is hosted through cloudflare, it does not return the page headers, which means the function returns false even if the destination URL does exist.

As an example you can use https://ezclix.club/m/1125 which redirects to a warriorplus.com website (which has cloudflare) before finally redirecting to https://ezclix.club/index.asp as the final destination which does exist, but the function returns false once it hits warriorplus which has cloudflare enabled.

CodePudding user response：

An option you have is to check if the request URI is not the same as the response URI. If it isn't, then recursively check the response URI.

For example:

' Imports System.Net
Public Function URLExists(url As String) As Boolean
    Dim request = WebRequest.Create(url)
    request.Method = "GET"
    Try
        Dim response = DirectCast(request.GetResponse, HttpWebResponse)
        If (request.RequestUri <> response.ResponseUri) Then
            Return URLExists(response.ResponseUri.ToString())
        End If
        Return response.StatusCode = HttpStatusCode.OK
    Catch
        Return False
    End Try
End Function