Keras Resizing layer not accepting different sized images as input-CodePudding

My understanding is that Keras' Resizing layer can take input images of different sizes, but I've been unable to get it to work. The following notebook shows the error:

import tensorflow as tf
import tensorflow_datasets as tfds
from tensorflow.keras import layers
dsTrain = tfds.load('CatsVsDogs', split='train[:10%]', as_supervised=True).batch(128).prefetch(tf.data.AUTOTUNE)

model = tf.keras.models.Sequential()
model.add(layers.InputLayer(input_shape=(None, None, 3)))
model.add(layers.Resizing(200, 200, crop_to_aspect_ratio=True)) # PROBLEM CAUSED HERE. See https://www.tensorflow.org/tutorials/images/data_augmentation
model.add(layers.Rescaling(1./255))
model.add(layers.Conv2D(8, (3, 3), activation='relu', kernel_initializer='random_normal', padding='same'))
model.add(layers.MaxPooling2D((3,3)))
model.add(layers.Dense(2))

model.compile(optimizer='rmsprop',loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),metrics=['accuracy'])
model.fit(dsTrain,epochs=10,batch_size=128) 

# Result:
# Cannot batch tensors with different shapes in component 0. First element had shape [262,350,3] and element 1 had shape [409,336,3].

Does anyone have a grasp of the intended setup for this layer? I can obviously resize the images from the dataset in advance of passing them to the model (this is what I've previously been doing) but I'm interested to understand the correct use of this layer.

EDIT 1

This is my attempt (linked in my reply as a notebook) to make a version that integrates the preprocessing into the model

import tensorflow as tf
import tensorflow_datasets as tfds
from tensorflow import keras
from tensorflow.keras import layers
dsTrain = tfds.load('CatsVsDogs', split='train[:10%]', as_supervised=True).batch(128).prefetch(tf.data.AUTOTUNE)
IMAGE_SIZE = 100

# https://keras.io/guides/preprocessing_layers/
# https://blog.tensorflow.org/2021/11/an-introduction-to-keras-preprocessing.html 

preprocessing_model = tf.keras.models.Sequential()
preprocessing_model.add(layers.InputLayer(input_shape=(None, None, 3)))
preprocessing_model.add(layers.Resizing(IMAGE_SIZE, IMAGE_SIZE, crop_to_aspect_ratio=True))
preprocessing_model.add(layers.Rescaling(1./255))

training_model = tf.keras.models.Sequential()
training_model.add(layers.InputLayer(input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3)))
training_model.add(layers.Conv2D(8, (3, 3), activation='relu', kernel_initializer='random_normal', padding='same'))
training_model.add(layers.MaxPooling2D((5,5)))
training_model.add(layers.Flatten())
training_model.add(layers.Dense(2))

inputs = keras.Input(shape=preprocessing_model.input_shape)
outputs = training_model(preprocessing_model(inputs))
model = tf.keras.Model(inputs, outputs)

model.summary()
model.compile(optimizer='rmsprop',loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),metrics=['accuracy'])
model.fit(dsTrain,epochs=3,batch_size=128)

This gives the error:

WARNING:tensorflow:Model was constructed with shape (None, None, None, 3) for input KerasTensor(type_spec=TensorSpec(shape=(None, None, None, 3), dtype=tf.float32, name='input_117'), name='input_117', description="created by layer 'input_117'"), but it was called on an input with incompatible shape (None, None, None, None, 3).
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-82-6e99f023839a> in <module>
     22 
     23 inputs = keras.Input(shape=preprocessing_model.input_shape)
---> 24 outputs = training_model(preprocessing_model(inputs))
     25 model = tf.keras.Model(inputs, outputs)
     26 

1 frames
/usr/local/lib/python3.7/dist-packages/keras/preprocessing/image.py in smart_resize(x, size, interpolation)
    110     if img.shape.rank < 3 or img.shape.rank > 4:
    111       raise ValueError(
--> 112           'Expected an image array with shape `(height, width, channels)`, '
    113           'or `(batch_size, height, width, channels)`, but '
    114           f'got input with incorrect rank, of shape {img.shape}.')

ValueError: Exception encountered when calling layer "resizing_72" (type Resizing).

Expected an image array with shape `(height, width, channels)`, or `(batch_size, height, width, channels)`, but got input with incorrect rank, of shape (None, None, None, None, 3).

Call arguments received:
  • inputs=tf.Tensor(shape=(None, None, None, None, 3), dtype=float32)

EDIT 2

I've now got the preprocess layers happening as part of the model, but they still won't accept images of different sizes. The below fails with the error noted, but works if I resize all the images to uniform dimensions. Notebook version

import tensorflow as tf
import tensorflow_datasets as tfds
from tensorflow import keras
from tensorflow.keras import layers
IMAGE_SIZE = 100
dsTrain = tfds.load('CatsVsDogs', split='train[:10%]', as_supervised=True).batch(128).prefetch(tf.data.AUTOTUNE)

preprocessing_model = tf.keras.models.Sequential()
preprocessing_model.add(layers.InputLayer(input_shape=(None, None, 3)))
preprocessing_model.add(layers.Resizing(IMAGE_SIZE, IMAGE_SIZE, crop_to_aspect_ratio=True))
preprocessing_model.add(layers.Rescaling(1./255))

training_model = tf.keras.models.Sequential()
training_model.add(layers.InputLayer(input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3)))
training_model.add(layers.Conv2D(8, (3, 3), activation='relu', kernel_initializer='random_normal', padding='same'))
training_model.add(layers.MaxPooling2D((5,5)))
training_model.add(layers.Flatten())
training_model.add(layers.Dense(2))

inputs = keras.Input(shape=preprocessing_model.input_shape[1:]) # [1:] Adds a dimension for a batching
outputs = training_model(preprocessing_model(inputs))
model = tf.keras.Model(inputs, outputs)

model.summary()
model.compile(optimizer='rmsprop',loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),metrics=['accuracy'])
model.fit(dsTrain,epochs=3,batch_size=128) 

# error from model.fit: Cannot batch tensors with different shapes in component 0. First element had shape [262,350,3] and element 1 had shape [409,336,3]

FINAL EDIT SOLUTION

Ok the message Souresh Mirzaei was very patiently trying to explain to me finally sunk in when I walked away from this, and I got my solution. For anyone looking at this the answer is that while training data cannot be of different sizes (I guess because it breaks batching) a trained model with a resizing layer at the start will accept different sized images as input

import tensorflow as tf
import tensorflow_datasets as tfds
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
import matplotlib.pyplot as plt
import random

IMAGE_SIZE = 100
dsTrain = tfds.load('CatsVsDogs', split='train[:10%]', as_supervised=True)
dsPredict = dsTrain.take(77)
def preprocess(image, label):
    image = tf.image.resize(image, (100,100))
    return image, label
dsTrain = dsTrain.map(preprocess).shuffle(1024).batch(128).prefetch(tf.data.AUTOTUNE)

preprocessing_model = tf.keras.models.Sequential()
preprocessing_model.add(layers.Resizing(IMAGE_SIZE, IMAGE_SIZE, crop_to_aspect_ratio=True, input_shape=(None, None, 3))) # redundant
preprocessing_model.add(layers.Rescaling(1./255))

training_model = tf.keras.models.Sequential()
training_model.add(layers.Conv2D(8, (3, 3), activation='relu', kernel_initializer='random_normal', padding='same'))
training_model.add(layers.MaxPooling2D((5,5)))
training_model.add(layers.Flatten())
training_model.add(layers.Dense(2))

inputs = keras.Input(shape=(None, None, None, 3)) # [1:] Remove the dimension added by batching
inputs = keras.Input(shape=preprocessing_model.input_shape[1:]) # [1:] Remove the dimension added by batching
outputs = training_model(preprocessing_model(inputs))
model = tf.keras.Model(inputs, outputs)

model.summary()
model.compile(optimizer='rmsprop',loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),metrics=['accuracy'])
model.fit(dsTrain,epochs=1,batch_size=128) 

# Test an image
for image, labelId in dsPredict.skip(random.randint(0, 50)).take(1):
    dsImage = tf.keras.preprocessing.image.img_to_array(image)
    dsImage = np.expand_dims(dsImage, axis = 0)
    prediction = model.predict(dsImage)
    plt.imshow(image)
    plt.title(f"{prediction[0][0]}, {prediction[0][1]}")
    plt.show()

notebook of the above

CodePudding user response：

preprocess dataset (Resize, Rescale, augment, etc.) before feeding into model istead of Resizing layer, because the model could not initialized on undefined features and then use it to train and predict on images with every size although you have implemented Resizing layer in your model.

so define function and then map on the dataset as below

dsTrain = tfds.load('CatsVsDogs', split='train[:10%]', as_supervised=True)

def preprocess(image, label):
    image = tf.image.resize(image, (200,200))
    return image, label

dsTrain = dsTrain.map(preprocess).shuffle(1024).batch(128).prefetch(tf.data.AUTOTUNE)

and even notice you didn't have implemented Flatten layer or GlobalAvgPool layer right after the Convolution layers, So implement Flatten layer before Dense layer unless it would raise an error.

Note, if you wanna create sequential of preprocessing operations, you should define it before same as below:

dsTrain = tfds.load('CatsVsDogs', split='train[:10%]', as_supervised=True)

resize_rescale = tf.keras.Sequential()
resize_rescale.add(layers.Resizing(200, 200, crop_to_aspect_ratio=True))
resize_rescale.add(layers.Rescaling(1./255))

dsTrain = dsTrain.map(lambda x, y : (resize_rescale(x), y)).batch(128).prefetch(tf.data.AUTOTUNE)

model = tf.keras.models.Sequential()
model.add(layers.Conv2D(8, (3, 3), activation='relu', kernel_initializer='random_normal', padding='same'))
model.add(layers.MaxPooling2D((3,3)))
model.add(layers.Flatten())
model.add(layers.Dense(2))

CodePudding user response：

The right edited code is here:

def prcs(imag, label):
    imag = tf.image.resize(imag, (100,100))
    imag = tf.reshape(imag, (100,100,3))
    return imag, label

dsTrain = dsTrain.map(prcs).batch(128).prefetch(tf.data.AUTOTUNE)

preprocessing_model = tf.keras.models.Sequential()
preprocessing_model.add(layers.Resizing(IMAGE_SIZE, IMAGE_SIZE))
preprocessing_model.add(layers.Rescaling(1./255))

training_model = tf.keras.models.Sequential()
training_model.add(layers.Conv2D(8, (3, 3), activation='relu', kernel_initializer='random_normal', padding='same'))
training_model.add(layers.MaxPooling2D((5,5)))
training_model.add(layers.Flatten())
training_model.add(layers.Dense(2))

inputs = keras.Input(shape=(100, 100, 3))
outputs = training_model(preprocessing_model(inputs))
model = tf.keras.Model(inputs, outputs)

Note, you have implemented so many input layers, and by this way i just used it one time only where you have defined inputs variable.

if You notice as sequence of layers, you implement two inputs layers for preprocessing layer, one of them is in the sequential

preprocessing_model.add(layers.InputLayer(input_shape=(None, None, 3)))
preprocessing_model.add(layers.Resizing(IMAGE_SIZE, IMAGE_SIZE, crop_to_aspect_ratio=True))
preprocessing_model.add(layers.Rescaling(1./255))

and the other one where you defined inputs variable

inputs = keras.Input(shape=preprocessing_model.input_shape)

and the so the sequence of layers would be something like this which is absolutely wrong:

preprocessing_model.add(keras.Input(shape=preprocessing_model.input_shape))
preprocessing_model.add(layers.InputLayer(input_shape=(None, None, 3)))
preprocessing_model.add(layers.Resizing(IMAGE_SIZE, IMAGE_SIZE, crop_to_aspect_ratio=True))
preprocessing_model.add(layers.Rescaling(1./255))

and even notice as the data goes through preprocessing layers just after the input layer, so there is not required to define input layer one another time in the training model sequential.