I created python script to scrape facebook help page. I wanted to scrape cms_object_id, cmsID, name. so these values are in a script tag then firstly tried to find all <script> tags then tried to iterate over this and then there is __bbox inside the tags which contains the values wanted to scrape.

so this is my script:

import json
import requests
import bs4
from Essentials import Static


class CmsIDs:
    def GetIDs():
        # cont = requests.get(""https://www.facebook.com:443/help"", headers=Static.headers) # syntax error
        cont = requests.get("https://www.facebook.com:443/help", headers=Static.headers)
        soup = bs4.BeautifulSoup(cont.content, "html5lib")

        text = soup.find_all("script")

        start = ""
        txtstr = ""

  

        for i in range(len(text)):
    
            mystr = text[i]

            # mystr = text[i]
            print("this is: ", mystr.find('__bbox":'))
            if text[i].get_text().find('__bbox":') != -1:
                # print(i, text[i].get_text())
                txtstr  = text[i].get_text()
                start = text[i].get_text().find('__bbox":')   len('__bbox":')

        print('start:', start)

        count = 0
        for end, char in enumerate(txtstr[start:], start):
            if char == '{':
                count  = 1
            if char == '}':
                count -= 1
            if count == 0:
                break
        print('end:', end)

        # --- convert JSON string to Python structure (dict/list) ---

        data = json.loads(txtstr[start:end 1])
        # pp.pprint(data)
        print('--- search ---')
        CmsIDs.search(data)

        # --- use recursion to find all 'cms_object_id', 'cmsID', 'name' ---

    def search(data):
        if isinstance(data, dict):
            found = False
            if 'cms_object_id' in data:
                print('cms_object_id', data['cms_object_id'])
                found = True
            if 'cmsID' in data:
                print('cmsID', data['cmsID'])
                found = True
            if 'name' in data:
                print('name', data['name'])
                found = True
            if found:
                print('---')
            for val in data.values():
                CmsIDs.search(val)
        if isinstance(data, list):
            for val in data:
                CmsIDs.search(val)


if __name__ == '__main__':
    CmsIDs.GetIDs()

the page contains cms_object_id, cmsID, name. so wanted to scrape all these 3 values but I am getting an error:

for end, char in enumerate(txtstr[start:], start):
TypeError: slice indices must be integers or None or have an __index__ method

so how can I solve this error and reach ultimate goal?

CodePudding user response：

Note: Since I'm unfamiliar with and failed to install Essentials, and also because ""https://www.facebook.com:443/help"" raises a syntax error (there should only be one quote on each side of the string), I changed the requests line in my code.

        cont = requests.get('https://www.facebook.com:443/help', headers={'accept': 'text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'})

TypeError: slice indices must be integers or None or have an __index__ method

You've initiated start as a string [start = ""] and it needs to be an integer. Unless the if text[i].get_text().find('__bbox":') != -1.... block is entered, start remains a string.

if you just want to not get this error, you could just exit the program if start hasn't been updated [indicating that __bbox": wasn't found in any of the script tags].

        print('start:', start)
        if start == "":
            print('{__bbox":} not found')
            return

        count = 0

But that still leaves the problem of __bbox": not being found; I'm not sure why, but the issue is resolved for me if I don't use the html5lib parser - just changing to BeautifulSoup(cont.content) resolved this issue.

        # soup = bs4.BeautifulSoup(cont.content, "html5lib")
        soup = bs4.BeautifulSoup(cont.content) # don't define the parser 
        # soup = bs4.BeautifulSoup(cont.content, "html.parser") # you could also try other parsers

Other suggestions

Your code will probably work without these, but you might want to consider these suggested improvements for error-handling:

Filter the script Tags

If the text ResultSet only has script tags that contain __bbox":, you avoid looping unnecessarily through the 100 other scripts, and you won't have to check with if....find('__bbox":') anymore.

        text = soup.find_all(lambda t: t.name == 'script' and '"__bbox"' in t.get_text())

        for mystr in [s.get_text() for s in text]:
            print("this is: ", mystr.find('__bbox":'))
            txtstr  = mystr
            start = mystr.find('__bbox":')   len('__bbox":') 

Initiate end

You should initiate the end variable [like end = 0] before the for end, char ... loop because you're using it after the loop as well.

        print('end:', end)

        data = json.loads(txtstr[start:end 1])

If txtstr[start:] is empty somehow, these lines will raise error/s since end would not be defined yet.

Use a JavaScript Parser

This will make the previous suggestions unnecessary, but as it is json.loads will raise an error if txtstr[start:end 1] is empty somehow, or if it contains any unpaired [and likely escaped] }or{. So, it might be more reliable to use a parser rather than just trying to walk through the string.

I have this function that uses slimit to find values from strings containing JavaScript code. (It's far from perfect, but it seems to for this script, at least.) GetIDs() could be re-written as below.

# import json, requests, bs4, slimit
# from slimit.visitors import nodevisitor 
# def findObj_inJS... ## PASTE FROM https://pastebin.com/UVcLniSG

# class CmsIDs:
    def GetIDs():
        # cont=requests.get('https://www.facebook.com:443/help',headers=Static.headers)
        cont = requests.get('https://www.facebook.com:443/help', headers={
            'accept': ';'.join(
                [ 'text/html,application/xhtml xml,application/xml',
                  'q=0.9,image/avif,image/webp,image/apng,*/*',
                  'q=0.8,application/signed-exchange',
                  'v=b3', 'q=0.9'  ])})

        ## in case of request errors ##
        try: cont.raise_for_status()
        except Exception as e:
            print('failed to fetch page HTML -', type(e), e)
            return
        print('fetched', cont.url, 'with', cont.status_code, cont.reason)

        soup = bs4.BeautifulSoup(cont.content)
        scrCond = lambda t: t.name == 'script' and '"__bbox"' in t.get_text()
        jScripts = [s.get_text() for s in soup.find_all(scrCond)] 
        print(f'Found {len(jScripts)} script tags containing {{"__bbox"}}')

        data = [findObj_inJS(s,'"__bbox"') for s in jScripts]
        print('--- search ---')
        CmsIDs.search(data) 

    # def search(data)....

Return the Data

This isn't for error-handling, but if you return the data printed by CmsIDs.search you could save it for further use.

    def search(data):
        rList, dKeys = [], ['cms_object_id', 'cmsID', 'name']
        if isinstance(data, dict):
            dObj = {k: data[k] for k in dKeys if k in data}
            rList  = [dObj] if dObj else []

            for k, v in dObj.items(): print(k, v)
            if dObj: print('---')

            for val in data.values(): rList  = CmsIDs.search(val)
        if isinstance(data, list):
            for val in data: rList  = CmsIDs.search(val)
        return rList

The printed result will be the same as before, but if you change the last line of GetIDs to

        return CmsIDs.search(data)

and then define a variable cmsList = CmsIDs.GetIDs() then cmsList will be a list of dctionaries, which you could then [for example] save to csv with pandas and view as a table on a spreadsheet.

# import pandas
pandas.DataFrame(cmsList).to_csv('CmsIDs_GetIDs.csv', index=False)

or print the markdown for the table [of the results I got] below

print(pandas.DataFrame(cmsList, dtype=str).fillna('').to_markdown())