Pandas not showing all the columns I asked for, Why?-CodePudding

I am new to pandas. The file I am working with is very large and I only want to read a few lines from 9 columns. However, when I run the code, it only shows me 3 of the 9 columns. I don't know why.

columns = [0, 1, 2, 3, 4, 5, 10, 12, 17]
header_names = ['Incident Number', 'Incident Type', 'Report Date', 'Nearest Populated Center',
                'Province', 'Company', 'Substance', 'Significant', 'What Happened category']
p_line = pd.read_csv(r"pipeline_data.csv",
                     skiprows=1,
                     nrows=5,
                     usecols=columns,
                     names=header_names,
                     index_col=0)

This is the ouput... Also, why is Incident Number not being printed?

                                                    Incident Type  ...    What Happened  category
Incident Number                                                     ...                          
INC2007-097                                   Release of Substance  ...    Corrosion and Cracking
INC2008-001                                   Release of Substance  ...    Corrosion and Cracking
INC2008-004                                   Release of Substance  ...         Equipment Failure
INC2008-008                                   Release of Substance  ...  Defect and Deterioration
INC2008-009      Release of Substance, Adverse Environmental Ef...  ...         Equipment Failure

[5 rows x 8 columns]

THEN I added

pd.set_option('display.max_columns', None)

but the output was then broken up....

                                                     Incident Type  \
Incident Number                                                      
INC2007-097                                   Release of Substance   
INC2008-001                                   Release of Substance   
INC2008-004                                   Release of Substance   
INC2008-008                                   Release of Substance   
INC2008-009      Release of Substance, Adverse Environmental Ef...   

                Report Date Nearest Populated Center          Province  \
Incident Number                                                          
INC2007-097      01/02/2008           Grande Prairie           Alberta   
INC2008-001      01/02/2008                   Cromer          Manitoba   
INC2008-004      01/23/2008                   Cromer          Manitoba   
INC2008-008      01/29/2008              Fort Nelson  British Columbia   
INC2008-009      01/29/2008                   Taylor  British Columbia   

                                                           Company  \
Incident Number                                                      
INC2007-097                                 Alliance Pipeline Ltd.   
INC2008-001                                Enbridge Pipelines Inc.   
INC2008-004                                Enbridge Pipelines Inc.   
INC2008-008      Westcoast Energy Inc., carrying on business as...   
INC2008-009      Westcoast Energy Inc., carrying on business as...   

                                              Substance Significant  \
Incident Number                                                       
INC2007-097                         Natural Gas - Sweet          No   
INC2008-001                           Crude Oil - Sweet          No   
INC2008-004                           Crude Oil - Sweet          No   
INC2008-008                         Natural Gas - Sweet          No   
INC2008-009      Potassium Hydroxide (caustic solution)         Yes   

                   What Happened category  
Incident Number                            
INC2007-097        Corrosion and Cracking  
INC2008-001        Corrosion and Cracking  
INC2008-004             Equipment Failure  
INC2008-008      Defect and Deterioration  
INC2008-009             Equipment Failure

I just want all the info in one long table, not some weird broken up table

CodePudding user response：

Maybe you can use this function and look at your table in a new tab

df.to_html('data.html')

CodePudding user response：

You can use following config options:

pd.options.display.max_columns = None

It sets the maximum number of columns displayed when a frame is pretty-printed and the truncated lines are replaced by an ellipsis.
pd.options.display.max_colwidth = None

It sets the maximum width of columns. Cells of this length or longer will be truncated with an ellipsis.
pd.options.display.width = 1000

You will have to experiment with this value. It sets number of characters displayed.

Other config options are explained here