I want to approximate the function g(x) = exp(x) with a linear combination of functions, h(x) = sum_i(a_i * f_i(x)) using a neural network with TF2.

Now, the network input is just x and the outputs are the functions f_i.

The custom loss function is simple: the mean squared difference |g(x) - h(x)|^2.

My problem is that I don't understand how to define/use a_i?

First, all a_i are just scalars, In addition they don't depend on x.

Defining a_i as inputs doesn't make sense since I want them to be optimized.

Defining as outputs makes them depend on x which I don't want.

How can I add these variables as scalars to the network and make them optimized by the optimization process ?.

CodePudding user response：

During training, Tensorflow tracks all tf.Variable objects defined in the model. If you define your constant as a tf.Variable, it will be ajusted via backpropagation.

Let's say we have a dataset wi X, and y which is y = X * 2:

import tensorflow as tf

x = tf.random.uniform((10_000, 1))
y = x * 2

We will create a model which has a constant inside, which will need to replicate the relationship between X and y. We will of course initialize this value as something other than 2. The relationship between X and y is 2 so the training should make the constant converge towards 2. So let's define a model that is nothing but a constant.

class CustomModel(tf.keras.models.Model):
    def __init__(self):
        super(CustomModel, self).__init__()
        self.constant = tf.Variable(initial_value=0.1, dtype=tf.float32, trainable=True)

    def call(self, x, **kwargs):
        x = self.constant * x
        return x


model = CustomModel()

Now just compile and train the model:

model.compile(loss='mae')

history = model.fit(x, y, epochs=25, verbose=0)

Now look at the weights. The constant, which was initialized with value 0.1, is now 2. It was optimized, as it understood the relationship between X and y, which is 2.

model.weights

[<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=2.0>]