Tabular data: Implementing a custom tensor layer without resorting to iteration-CodePudding

I have an idea for a tensor operation that would not be difficult to implement via iteration, with batch size one. However I would like to parallelize it as much as possible.

I have two tensors with shape (n, 5) called X and Y. X is actually supposed to represent 5 one-dimensional tensors with shape (n, 1): (x_1, ..., x_n). Ditto for Y.

I would like to compute a tensor with shape (n, 25) where each column represents the output of the tensor operation f(x_i, y_j), where f is fixed for all 1 <= i, j <= 5. The operation f has output shape (n, 1), just like x_i and y_i.

I feel it is important to clarify that f is essentially a fully-connected layer from the concatenated [...x_i, ...y_i] tensor with shape (1, 10), to an output layer with shape (1,5).

Again, it is easy to see how to do this manually with iteration and slicing. However this is probably very slow. Performing this operation in batches, where the tensors X, Y now have shape (n, 5, batch_size) is also desirable, particularly for mini-batch gradient descent.

It is difficult to really articulate here why I desire to create this network; I feel it is suited for my domain of 'itemized tabular data' and cuts down significantly on the number of weights per operation, compared to a fully connected network.

Is this possible using tensorflow? Certainly not using just keras. Below is an example in numpy per AloneTogether's request

import numpy as np

features = 16
batch_size = 256

X_batch = np.random.random((features, 5, batch_size))
Y_batch = np.random.random((features, 5, batch_size))

# one tensor operation to reduce weights in this custom 'layer'
f = np.random.random((features, 2 * features))

for b in range(batch_size):
    X = X_batch[:, :, b]
    Y = Y_batch[:, :, b]
    for i in range(5):
        x_i = X[:, i:i 1]
        for j in range(5):
            y_j = Y[:, j:j 1]

            x_i_y_j = np.concatenate([x_i, y_j], axis=0)
            
            # f(x_i, y_j)
            # implemented by a fully-connected layer
            f_i_j = np.matmul(f, x_i_y_j)

CodePudding user response：

All operations you need (concatenation and matrix multiplication) can be batched. Difficult part here is, that you want to concatenate features of all items in X with features of all items in Y (all combinations). My recommended solution is to expand the dimensions of X to [batch, features, 5, 1], expand dimensions of Y to [batch, features, 1, 5] Than tf.repeat() both tensors so their shapes become [batch, features, 5, 5]. Now you can concatenate X and Y. You will have a tensor of shape [batch, 2*features, 5, 5]. Observe that this way all combinations are built. Next step is matrix multiplication. tf.matmul() can also do batch matrix multiplication, but I use here tf.einsum() because I want more control over which dimensions are considered as batch. Full code:

import tensorflow as tf
import numpy as np

batch_size=3
features=6
items=5

x = np.random.uniform(size=[batch_size,features,items])
y = np.random.uniform(size=[batch_size,features,items])

f = np.random.uniform(size=[2*features,features])

x_reps= tf.repeat(x[:,:,:,tf.newaxis], items, axis=3)
y_reps= tf.repeat(y[:,:,tf.newaxis,:], items, axis=2)

xy_conc = tf.concat([x_reps,y_reps], axis=1)

f_i_j = tf.einsum("bfij, fg->bgij", xy_conc,f)

f_i_j = tf.reshape(f_i_j , [batch_size,features,items*items])