tensorflow.saved_model.load() takes extremely long time to execute-CodePudding

The

And now, let’s see the loading times. I included the number of parameters as well, so that you do not need to refer to the upper table. In the column “Weights Only”, the measured time is for both recreating the models and loading the saved weights.

Or if you like charts, you can see it graphically below. I left out TensorFlow lite as the order of magnitude is so different.

Can we optimize the models?

Not directly for model size only. There is quite a few resources about optimization in the official documentation. This page is mainly focused on TensorFlow Lite, including model selection, training and post-training optimization. In addition to that, you can directly check the TensorFlow Model Optimization Python API which works for standard models as well.

There are two simple methods you can use:

quantization: converting weighs to types with reduce prediction

pruning: removing insignificant weights.

As you can imagine, those methods influence many parts of the machine learning loop. This includes the model size and therefore loading time as well, but the main focus is on prediction time with the side effect of possible accuracy decrease.

CodePudding user response：

Got it! On the first model call, the graph is built, so the first call to the model is always slow. I tried your code on google colab using a GPU:

model = load_object_detection_model("ssd_mobilenet_v2_320x320_coco17_tpu-8")

%%time a= model(np.random.randint(0, 255, size=(1, 320, 320, 3)).astype("uint8"))

CPU times: user 4.32 s, sys: 425 ms, total: 4.75 s
Wall time: 4.71 s

%%time a= model(np.random.randint(0, 255, size=(1, 320, 320, 3)).astype("uint8"))

CPU times: user 124 ms, sys: 18.4 ms, total: 143 ms
Wall time: 85.4 ms

In the document, they say 22 ms for this model, but maybe they got a faster GPU.