How to load images with different image shape to tf.data pipe?-CodePudding

My goal is to have a preprocessing layers so it can handle any image size. This is because the data set that I use have 2 different image shape. The solution is simple, just resize it when I load the image. However, I believe this wont work when the model is deployed, I can't do manual resize like that. So I must use preprocessing layers.

The docs I used

What I've tried:

Put the preprocessing layers part of the model, it does not work.
I am thinking to use TensorSliceDataset.map(resize_and_rescale).

The problem is I need to convert the [tensor image 1, tensor image 2] to TensorSliceDataset. However, I can't convert it.

What I've tried:
1. tf.data.Dataset.from_tensor_slices((X_train, y_train))
  
  It throws error
```
InvalidArgumentError: {{function_node __wrapped__Pack_N_9773_device_/job:localhost/replica:0/task:0/device:GPU:0}} Shapes of all inputs must match: values[0].shape = [258,320,3] != values[23].shape = [322,480,3]
[[{{node Pack}}]] [Op:Pack] name: component_0
```

The load images function:

def load_images(df):
    paths = df['path'].values
    X = []
    
    for path in paths:
        raw = tf.io.read_file(path)
        img = tf.image.decode_png(raw, channels=3)
        X.append(img)
        
    y = df['kind'].cat.codes
    
    return X, y

CodePudding user response：

As far as I understand you wish to train on both image sizes simultaneously. The simplest way is probably to create two different datasets for each image size and concatenate them after the batching as follows:

dataset_1 = tf.data.Dataset.from_tensor_slices((X_train_1, y_train_1))
dataset_1 = dataset_1.batch(batch_size_1)

dataset_2 = tf.data.Dataset.from_tensor_slices((X_train_2, y_train_2))
dataset_2 = dataset_2.batch(batch_size_2)

dataset = dataset_1.concatenate(dataset_2)
dataset = dataset.shuffle(shuffle_buffer_size)

This case each batch consists of images of the same size. If you use .repeat() do not forget to put if after the concatination.

CodePudding user response：

You need to use ragged tensors to handle different image sizes:

dataset = tf.data.Dataset.from_tensor_slices((tf.ragged.constant(img_list), label_list))
dataset = dataset.apply(tf.data.experimental.dense_to_ragged_batch(batch_size=3))

Example