Can Keras Generator return lists instead of np.array? ValueError: Layer "sequential" expec-CodePudding

My goal is to use tf.keras.utils.Sequence to load images of variable dimensions from a directory, transform images, and build a generator to train fully convolutional CNN. The method works when I create a numpy array to store images

Method 1 that works

class Generator(Sequence):
    # the rest of Generator

    def __getitem__(self, idx):
        # create numpy arrays for X and y
        X = np.zeros((batch_size, height, width, n_channels))
        y = np.zeros((batch_size, height, width, n_channels))

        for i, image_fn in enumerate(image_fns):
            # transform images

            X[i] = self.to_float(image=X_transformed)["image"]
            y[i] = self.to_float(image=y_transformed)["image"]

        return X, y

Because I am working with images where height and width are variable, I tried using lists to store images

Method 2 that fails

class Generator(Sequence):
    # the rest of Generator

    def __getitem__(self, idx):
        # create list for X and y
        X, y = [], []

        for i, image_fn in enumerate(image_fns):
            # transform images

            X  = [ self.to_float(image=X_transformed)["image"] ]
            y  = [ self.to_float(image=y_transformed)["image"] ]

        return X, y

However, I get the error message below.

ValueError: Layer "sequential" expects 1 input(s), but it received 30 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, None, None) dtype=float32>...

I checked the type for X and y, and they are a list of <class 'numpy.ndarray'>.

CodePudding user response：

Sequential models expect all layers to have a single input and a single output. When you pass a plain Python list of numpy arrays to a model, it interprets each numpy array in the list as a separate input or output, hence the error.

You can fix it by converting your X and y into single RaggedTensor. However, using ragged tensors comes with some restrictions for your model, as not all TF operations are supported.

return tf.ragged.constant(X), tf.ragged.constant(y)

For your additional questions

maybe Sequential is always expecting a NumPy array

Not necessary. Sequential expects anything that is not a nested structure, i.e., anything that is an "atom". This includes numpy array, TF tensor, or even set (handling of set is not supported yet) but not e.g., list, tuple, or dictionary (see the link).

do I standardize the image dimensions for each batch, but between batches I can make the input image dimensions variable?

Whether you can use variable image size depends mostly on your sequential model's layers. If all layers are pure convolutional, then your model can handle variable image sizes but if e.g., there is a Dense layer somewhere, then it can't, as Dense requires fixed input shape.