Calculate correlations between two different tables with different structures-CodePudding

I have two tables, on containing a column of one value 'num1' per date, and another containing one value per name 'num2' per date. I'm interested in calculating the correlation between each name's 'num2' value and 'num1', but I'm unsure of whether or not I need to break the 'data2' table up into distinct tables for each name, or if there's a clean way to do this.

data1 = {'date':  ['2022-01-03', '2022-01-04', '2022-01-05'], 'num1': ['.024', '.035', '.04']}
data2 = {'date':  ['2022-01-03', '2022-01-03', '2022-01-03', '2022-01-04', '2022-01-04', '2022-01-04', '2022-01-05','2022-01-05','2022-01-05'], 'name': ['name1', 'name2', 'name3', 'name1', 'name2', 'name3', 'name1', 'name2', 'name3'], 'num2':['20','200','149','36','174','400','45','100','12']}
data1 = pd.DataFrame(data1).set_index('date')
data2 = pd.DataFrame(data2).set_index('date')
print(data1)
print(data2)

Is there a way to calculate correlations between num1 and num2 for each name without manipulating these tables heavily?

CodePudding user response：

You can use groupby.corr after ensuring having numerical values and joining the dataframes:

out = (data2
  .astype({'num2': int})
  .join(data1.astype({'num1': float}))
  .groupby('name').corr()
)

output:

                num2      num1
name                          
name1 num2  1.000000  0.998599
      num1  0.998599  1.000000
name2 num2  1.000000 -0.885346
      num1 -0.885346  1.000000
name3 num2  1.000000 -0.141869
      num1 -0.141869  1.000000