Approximating the conditional expectation E(X|Y)-CodePudding

I'm trying to find a way to approximate the conditional expectation E(X|Y) in python. All I have are two lists of numbers, X and Y and I don't have any other knowledge about them.

Is there a good method for doing so? I tried using all kind of smoother functions but the result was not good at all.

For example,
X = [ 1, 1, 1, 1, ....] and Y = [1, -1, 1, -1, ...] I would expect that E(X|Y) = 1 because X is constant regardless of Y.

CodePudding user response：

I'm going to make some assumptions here:

1 corresponds to True, -1 corresponds to False
X and Y are the same length and are independent
X and Y are stored in lists

Here's a review of Bayesian statistics (I don't know how to write equations here, sorry):

P(X) = probability of X being True = (# of True elements in X) / (# of elements in X)

P(Y) = probability of Y being True = (# of True elements in Y) / (# of elements in Y)

P(X and Y) = probability of both X and Y being True = P(X) * P(Y)

P(X given Y) = P(X | Y) = probability of X being True given that Y is True = P(X and Y) / P(Y)

And here's how to implement that into code:

X = np.array(X, dtype=int)
Y = np.array(Y, dtype=int)

p_X = len(np.where(X == 1)[0]) / len(X)
p_Y = len(np.where(Y == 1)[0]) / len(Y)

p_X_and_Y = p_X * p_Y
p_X_given_Y = p_X_and_Y / p_Y

This gives the correct answer of p_X_given_Y = 1.

CodePudding user response：

Your question is actually very broad, and thus the are many answers to your question as asked. Taking a broad view of the term machine learning, this is a question that people have been trying to answer in machine learning for a very long time. The answer really depends on the assumptions that you make about the distribution of X|Y and the relationship between X and Y.

I will suggest the sci-kit learn package because it gives a number of ways to calculate E(X|Y). Lets start with your example and use simple linear regression (this makes the assumption that the relationship between X and Y is linear)

import numpy as np
from sklearn.linear_model import LinearRegression
X = np.ones((100,1))
Y = np.ones((100,1))
Y[0:-1:2] = -1
LR = LinearRegression()
LR.fit(Y,X)
a = LR.coef_
b = LR.intercept_

Now in this case since we assumed a linear relationship E(X|Y) = aY b and in the example above the linear regression tells us that a = 0 and b = 1. This gives you the conditional expectation that you expect (all expectation values equal to 1 regardless of the value of Y).

As I mentioned though there are many many ways of trying to estimate E(X|Y), for instance k-nearest neighbors can be used (and is implemented in sklearn) or Gaussian Process Regression (GPR) both of which do not require the assumption of a linear relationship between variables (the work very differently of course, where GPR requires a kernel function which we know from Mercer's theorem allows us to pick varying levels of smoothness for the function that is fitted to the data, whereas k-nearest neighbors estimates E(X|Y) at every point by averaging over its k-nearest neighbors as measured by some distance function). And there are many many more examples which you could explore in this wonderful package. It is a very interesting field!