Creating BLEU loss method on tensorflow gives "No gradient provided"-CodePudding

The

As β approaches infinity, this will approach the the true maximum. For your purposes, β=10 or β=100 should accomplish your goals.

In Tensorflow, this could be accomplished as follows:

def differentiable_argmax_approx(x): return tf.reduce_sum(tf.cumsum(tf.ones_like(x)) * tf.exp(beta * x) / tf.reduce_sum(tf.exp(beta * x))) - 1

We can verify the functionality with a simple test case:

beta = 10 x = np.array([1, 2, 3, 10, 4, 5], dtype=np.float) y = differentiable_argmax_approx(x, beta) assert x.argmax() == y

One caveat to this approach: if the maximum value is not unique along the axis that we're applying the function to, the result will be the arithmetic mean of the indices. Providing another test case to illustrate:

beta = 10 x = np.array([1, 2, 10, 3, 10], dtype=np.float) y = differentiable_argmax_approx(x, beta) assert y == 3

The result is 3 here, because we have two occurrences of the maximum value (10): one at index 2, and the other at index 4. In contrast, the regular argmax function returns the first index of the maximum argument.