VB.NET HtmlAgilityPack scraping text-CodePudding

I want to use HtmlAgilityPack to scrape content from https://steamid.io/, specifically, I want to scrape the status of the user's "Online Status".

I can correctly scrape the information I need using XPath, but.. Each user's profile is different there for the XPath changes, only slightly.

If the user's profile is "private" the XPath is

/html/body/div/div[2]/div[2]/section/dl/dd[9]

If the users profile is not "private" the XPath is

/html/body/div/div[2]/div[2]/section/dl/dd[9]/span

How would I be able to make HtmlAgilityPack check which div it is that I want to display, as I can set my own profile to private and unprivate but obviously I need to change the XPath accordingly to be able to show the status.

If i use the XPath containing the /span and the user's profile is private I get System.NullReferenceException: 'Object reference not set to an instance of an object.'

If i un-private the profile and use the /span XPath is works fine and shows me what I want.

Here is my entire code

'Online Status - from steamid.io
        Dim curOnline = idIo.DocumentNode.SelectNodes("/html/body/div/div[2]/div[2]/section/dl/dd[9]/span")
        'Dim curOnline2 = idIo.DocumentNode.SelectNodes("/html/body/div/div[2]/div[2]/section/dl/dd[9]")

        For Each node In curOnline
            onlinelbl.Text = node.InnerText
            If node.InnerText.ToString().Contains("private") Then
                onlinelbl.ForeColor = Color.Red
                onlinelbl.Text = "Private"
            ElseIf onlinelbl.Text.Contains("away") Then
                onlinelbl.Text = onlinelbl.Text.Replace("away", "Away")
                onlinelbl.ForeColor = Color.Orange
            ElseIf onlinelbl.Text.Contains("online") Then
                onlinelbl.Text = onlinelbl.Text.Replace("online", "Online")
                onlinelbl.ForeColor = Color.Green
            ElseIf onlinelbl.Text.Contains("offline") Then
                onlinelbl.Text = onlinelbl.Text.Replace("offline", "Offline")
                onlinelbl.ForeColor = Color.Red
            End If
        Next
    ```

CodePudding user response：

You can simply look for the existence of the XPath with /span:

    Dim curOnline As HtmlNodeCollection
    curOnline = idIo.DocumentNode.SelectNodes("/html/body/div/div[2]/div[2]/section/dl/dd[9]/span")
    If curOnline Is Nothing Then
        curOnline = idIo.DocumentNode.SelectNodes("/html/body/div/div[2]/div[2]/section/dl/dd[9]")
    End If