Determining the Right Shape for a RNN with a Integer Sequence Target-CodePudding

I am trying to build an RNN based model in Tensorflow that takes a sequence of categorical values as an input, and sequence of categorical values as the output.

For example, if I have sequence of 30 values, the first 25 would be the training data, and the last 5 would be the target. Imagine the data is something like a person pressing keys on a computer keyboard and recording their key presses over time.

I've tried to feed the training data and targets into this model in different shapes, and I always get an error that indicates the data is in the wrong shape.

I've included a code sample that should run and demonstrate what I'm trying to do and the failure I'm seeing.

In the code sample, I've used windows for batches. So if there are 90 values in the sequence, the first 25 values would be the training data for the first batch, and the next 5 values would be the target. This next batch would be the next 30 values (25 training values, 5 target values).

import numpy as np
import tensorflow as tf
from tensorflow import keras 

num_categories = 20
data_sequence = np.random.choice(num_categories, 10000)

def create_target(batch):
  X = tf.cast(batch[:,:-5][:,:,None], tf.float32)
  Y = batch[:,-5:][:,:,None]
  return X,Y

def add_windows(data):
  data = tf.data.Dataset.from_tensor_slices(data)
  return data.window(20, shift=1, drop_remainder=True)

dataset = tf.data.Dataset.from_tensor_slices(data_sequence)
dataset = dataset.window(30, drop_remainder=True)
dataset = dataset.flat_map(lambda x: x.batch(30))
dataset = dataset.batch(5)
dataset = dataset.map(create_target)

model = keras.models.Sequential([
  keras.layers.SimpleRNN(20, return_sequences=True),
  keras.layers.SimpleRNN(20, return_sequences=True),
  keras.layers.TimeDistributed(keras.layers.Dense(num_categories, activation="softmax"))                           
])

optimizer = keras.optimizers.Adam()
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer)
model.fit(dataset, epochs=1)

The error I get when I run the above code is

Node: 'sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits'
logits and labels must have the same first dimension, got logits shape [125,20] and labels shape [25]

I've also tried the following model, but the errors are similar.

model = keras.models.Sequential([
  keras.layers.SimpleRNN(20, return_sequences=True),
  keras.layers.SimpleRNN(20),
  keras.layers.Dense(num_categories, activation="softmax"))                           
])

Does anybody have any recommendations about what I need to do to get this working?

Thanks.

CodePudding user response：

I figured out the issue. The size of the time dimension needs to be the same for the training data and the target.

If you look at my original example code, the training data has these shapes X.shape = (1, 25, 1) Y.shape = (1, 5, 1)

To fix it, the time dimension should be the same.

X.shape = (1, 15, 1) Y.shape = (1, 15, 1)

Here is the updated function that will let the model train. Note that all I did was update the array sizes so they are equally sized. The value of 15 is used because the original array length is 30.

def create_target(batch):
  X = tf.cast(batch[:,:-15][:,:,None], tf.float32)
  Y = batch[:,-15:][:,:,None]
  return X,Y