I have the following dataframe, which the value should be increasing. Originally the dataframe has some unknown values.

Based on the assumsion that the value should be increasing, I would like to remove the value at index 10 and 11. This would be the desired dataframe:

Thank you very much

CodePudding user response：

Assuming NaN in the empty cells (if not, temporarily replace them with NaN), use boolean indexing:

# if not NaNs uncomment below
# and use s in place of df['value'] afterwards
# s = pd.to_numeric(df['value'], errors='coerce')

# is the cell empty?
m1 = df['value'].isna()

# are the values strictly increasing?
m2 = df['value'].ge(df['value'].cummax())

out = df[m1|m2]

Output:

    index  value
1       1    NaN
2       2    NaN
3       3    2.0
4       4    NaN
5       5    NaN
6       6    NaN
7       7    4.0
8       8    NaN
9       9    NaN
12     12    NaN
13     13    NaN
14     14    NaN
15     15    5.0

CodePudding user response：

def del_df(df):
    
    df_no_na = df.dropna().reset_index(drop = True)

    num_tmp = df_no_na['value'][0]   # First value which is not NaN.
    
    del_index_list = []   # indicies to delete

    for row_index in range(1, len(df_no_na)):

        if df_no_na['value'][row_index] > num_tmp :    #Increasing
            num_tmp = df_no_na['value'][row_index]   # to compare following two values.
        
        else :   # Not increasing(same or decreasing)
            del_index_list.append(df_no_na['index'][row_index])   # index to delete
    
    df_goal = df.drop([df.index[i] for i in del_index_list])

    return df_goal

output:

    index  value
0       0    1.0
1       1    NaN
2       2    NaN
3       3    2.0
4       4    NaN
5       5    NaN
6       6    NaN
7       7    4.0
8       8    NaN
9       9    NaN
12     12    NaN
13     13    NaN
14     14    NaN
15     15    5.0

I hope it meets your question.