Pandas: I want slice the data and shuffle them to genereate some synthetic data-CodePudding

Just want to help with data science to generate some synthetic data since we don't have enough labelled data. I want to cut the rows around the random position of the y column around 0s, don't cut 1 sequence.

After cutting, want to shuffle those slices and generate a new DataFrame.

It's better to have some parameters that adjust the maximum, and minimum sequence to cut, the number of cuts, and something like that.

The raw data

ts   v1    y
0    100   1
1    120   1
2    80    1
3    5     0
4    2     0
5    100   1
6    200   1
7    1234  1
8    12    0
9    40    0
10   200   1
11   300   1
12   0.5   0
...

Some possible cuts

ts   v1    y
0    100   1
1    120   1
2    80    1
3    5     0
--------------
4    2     0
--------------
5    100   1
6    200   1
7    1234  1
-------------
8    12    0
9    40    0
10   200   1
11   300   1
-------------
12   0.5   0
...

ts   v1    y
0    100   1
1    120   1
2    80    1
3    5     0
4    2     0
-------------
5    100   1
6    200   1
7    1234  1
8    12    0
9    40    0
10   200   1
11   300   1
------------
12   0.5   0
...

This is NOT CORRECT

ts   v1    y
0    100   1
1    120   1
------------
2    80    1
3    5     0
4    2     0
5    100   1
6    200   1
7    1234  1
8    12    0
9    40    0
10   200   1
11   300   1
12   0.5   0
...

CodePudding user response：

You can use:

#number of cuts
N = 3
#create random N index values of index if y=0
idx = np.random.choice(df.index[df['y'].eq(0)], N, replace=False)

#create groups with check membership and cumulative sum
arr = df.index.isin(idx).cumsum()
#randomize unique integers - groups  
u = np.unique(arr)
np.random.shuffle(u)
#change order of groups in DataFrame
df = df.set_index(arr).loc[u].reset_index(drop=True)

print (df)
    ts      v1  y
0    9    40.0  0
1   10   200.0  1
2   11   300.0  1
3   12     0.5  0
4    3     5.0  0
5    4     2.0  0
6    5   100.0  1
7    6   200.0  1
8    7  1234.0  1
9    8    12.0  0
10   0   100.0  1
11   1   120.0  1
12   2    80.0  1