Don't understand why I can't do even the most simple data manipulation with this data i've scraped. I've tried all sorts of methjods to manipulate the data but all come up with the same sort of error. Is my data even in a data frame yet? I can't tell.

import pandas as pd
from urllib.request import Request, urlopen

req = Request('https://smallcaps.com.au/director-transactions/'
              , headers={'User-Agent': 'Mozilla/5.0'})
trades = urlopen(req).read()
df = pd.read_html(trades)
print(df) #<-- This line prints the df and works fine

df.drop([0, 1]) #--> THis one shows the error below
print(df) 

Error:

Traceback (most recent call last):
  File "C:\Users\User\PycharmProjects\Scraper\DirectorTrades.py", line 10, in <module>
    df.drop([0, 1])
AttributeError: 'list' object has no attribute 'drop'

CodePudding user response：

read_html returns a list of dataframes.

Try:

dfs = pd.read_html(trades)
dfs = [df.drop([0,1]) for df in dfs]

CodePudding user response：

Main issue is as mentioned that pandas.read_html() returns a list of dataframes and you have to specify by index wich you like to choose.

Is my data even in a data frame yet?

• df = pd.read_html(trades) No it is not, cause it provides a list of dataframes

• df = pd.read_html(trades)[0] Yes, this will give you the first dataframe from list of frames

Example

import pandas as pd
from urllib.request import Request, urlopen

req = Request('https://smallcaps.com.au/director-transactions/'
              , headers={'User-Agent': 'Mozilla/5.0'})
trades = urlopen(req).read()
df = pd.read_html(trades)[0]
df.drop([0, 1])
df

Output

...