How do I change the batch size in VGG16? I'm trying to address an issue of exceeding memory constraints by 10% by doing this.
Error:
2021-12-03 16:17:07.263665: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 4888553472 exceeds 10% of free system memory.
Here is my code:
def labelObjectFromImage(image_path, directory_filename):
img = cv2.imread(image_path directory_filename)
height = img.shape[0]
width = img.shape[1]
channels = img.shape[2]
img = load_img(image_path directory_filename, target_size=(height, width))
model = VGG16(weights="imagenet", include_top = False, input_shape = (height, width, channels))
img = img_to_array(img)
img = img.reshape((1, img.shape[0], img.shape[1], img.shape[2]))
img = preprocess_input(img)
yhat = model.predict(img)
label = decode_predictions(yhat)
label = label[0][0]
print(label)
I tried changing model.predict to:
yhat = model.predict(img, batch_size=1)
but it doesn't appear to have any impact in trying to resolve the issue
I tried using:
from tensorflow.keras import backend as K
K.clear_session()
but that did not help
I ran
pip3 uninstall tensorflow-gpu
and then installed normal tensorflow via
pip3 install tensorflow
but that did not help
Just fyi, I'm getting the same error with all of these attempts so far.
As suggested I tried:
img_resized = tf.image.resize(img, (height, width))
But I now get the following error:
Traceback (most recent call last):
File "organizeSpreadsheet.py", line 105, in <module>
main()
File "organizeSpreadsheet.py", line 86, in main
objects_from_image = labelObjectFromImage(path_to_images, directory_filename)
File "organizeSpreadsheet.py", line 53, in labelObjectFromImage
img = img_resized.reshape((1, height, width, channels))
File "/home/jr/.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 437, in __getattr__
raise AttributeError("""
AttributeError:
'EagerTensor' object has no attribute 'reshape'.
If you are looking for numpy-related methods, please run the following:
from tensorflow.python.ops.numpy_ops import np_config
np_config.enable_numpy_behavior()
I know I didn't do it exactly as instructed, but it was throwing an error so I followed this suggestion
I corrected the error by making the following change:
def labelObjectFromImage(image_path, directory_filename):
scale = 60
img = cv2.imread(image_path directory_filename)
height = int(img.shape[0] * scale / 100)
width = int(img.shape[1] * scale / 100)
channels = img.shape[2]
#img_resized = cv2.resize(img, dim, interpolation=cv2.INTER_AREA)
#img_resized = tf.image.resize(img, (height, width))
tf.image.resize(img, (height, width))
#img = load_img(image_path directory_filename, target_size=(height, width))
model = VGG16(weights="imagenet", include_top = False, input_shape = (height, width, channels))
#img = img_to_array(img)
#img = img.reshape((1, img.shape[0], img.shape[1], img.shape[2]))
img = img.reshape((1, height, width, channels))
img = preprocess_input(img)
yhat = model.predict(img, batch_size=1)
label = decode_predictions(yhat)
label = label[0][0]
print(label)
But now I get the error:
ValueError: cannot reshape array of size 63483840 into shape (1,3384,2251,3)
I think this can be resolved by trying multiple scales, is that correct?
So I have been addressing the issues one-by-one and the first issue was my lack of installation of cudnn. I followed these instructions to do that.
Additionally, I corrected my code as the latest suggested said. So now my code looks like this:
def labelObjectFromImage(image_path, directory_filename):
scale = 100
while(1):
try:
img = cv2.imread(image_path directory_filename)
height = int(img.shape[0] * scale / 100)
width = int(img.shape[1] * scale / 100)
channels = img.shape[2]
img = tf.image.resize(img, (height, width))
model = VGG16(weights="imagenet", include_top = False, input_shape = (height, width, channels))
img = img.reshape((1, height, width, channels))
img = preprocess_input(img)
yhat = model.predict(img, batch_size=1)
label = decode_predictions(yhat)
label = label[0][0]
print(label)
return label
except Exception as e:
print("Error:", e, "scale", scale)
scale -= 1
For anyone checking in the future, please note that this code does not handle the case in which the scale reaches below 0. That should explicitly be handled in the code. I will post the final results when I get this working properly.
So when I run the code as suggested I get the following error:
Error:
'EagerTensor' object has no attribute 'reshape'.
If you are looking for numpy-related methods, please run the following:
from tensorflow.python.ops.numpy_ops import np_config
np_config.enable_numpy_behavior() scale 100
I changed
img = img.reshape((1, height, width, channels))
to
img = tf.reshape(img, (1, height, width, channels))
and got the following error:
2021-12-03 18:03:09.909978: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-03 18:03:09.947651: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-03 18:03:09.947958: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
Num GPUs Available: 1
/usr/local/lib/python3.8/dist-packages/openpyxl/worksheet/_reader.py:312: UserWarning: Unknown extension is not supported and will be removed
warn(msg)
2021-12-03 18:03:39.195772: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-12-03 18:03:39.197491: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-03 18:03:39.197793: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-03 18:03:39.198003: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-03 18:03:39.569744: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-03 18:03:39.569968: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-03 18:03:39.570148: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-03 18:03:39.570267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4560 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5
2021-12-03 18:03:50.862568: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 5.04GiB (rounded to 5417287680)requested by op vgg16/block1_conv1/Conv2D
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2021-12-03 18:03:50.862619: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] BFCAllocator dump for GPU_0_bfc
2021-12-03 18:03:50.862648: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (256): Total Chunks: 23, Chunks in use: 23. 5.8KiB allocated for chunks. 5.8KiB in use in bin. 612B client-requested in use in bin.
2021-12-03 18:03:50.862654: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (512): Total Chunks: 2, Chunks in use: 2. 1.0KiB allocated for chunks. 1.0KiB in use in bin. 1.0KiB client-requested in use in bin.
2021-12-03 18:03:50.862659: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1024): Total Chunks: 5, Chunks in use: 4. 5.2KiB allocated for chunks. 4.2KiB in use in bin. 4.0KiB client-requested in use in bin.
2021-12-03 18:03:50.862666: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2048): Total Chunks: 8, Chunks in use: 6. 18.5KiB allocated for chunks. 13.0KiB in use in bin. 12.0KiB client-requested in use in bin.
2021-12-03 18:03:50.862671: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4096): Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2021-12-03 18:03:50.862675: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862679: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862682: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862686: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862689: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862694: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (262144): Total Chunks: 2, Chunks in use: 2. 705.0KiB allocated for chunks. 705.0KiB in use in bin. 432.0KiB client-requested in use in bin.
2021-12-03 18:03:50.862715: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (524288): Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2021-12-03 18:03:50.862720: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1048576): Total Chunks: 1, Chunks in use: 1. 1.97MiB allocated for chunks. 1.97MiB in use in bin. 1.12MiB client-requested in use in bin.
2021-12-03 18:03:50.862746: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2097152): Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2021-12-03 18:03:50.862751: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4194304): Total Chunks: 2, Chunks in use: 1. 9.00MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2021-12-03 18:03:50.862757: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8388608): Total Chunks: 6, Chunks in use: 5. 61.79MiB allocated for chunks. 48.29MiB in use in bin. 45.00MiB client-requested in use in bin.
2021-12-03 18:03:50.862761: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862765: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862768: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862786: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (134217728): Total Chunks: 3, Chunks in use: 2. 726.51MiB allocated for chunks. 484.34MiB in use in bin. 484.34MiB client-requested in use in bin.
2021-12-03 18:03:50.862790: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 3.67GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 18:03:50.862794: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Bin for 5.04GiB was 256.00MiB, Chunk State:
2021-12-03 18:03:50.862814: I tensorflow/core/common_runtime/bfc_allocator.cc:1039] Size: 3.67GiB | Requested Size: 576.0KiB | in_use: 0 | bin_num: 20, prev: Size: 242.17MiB | Requested Size: 242.17MiB | in_use: 1 | bin_num: -1
2021-12-03 18:03:50.862817: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 4782227456
2021-12-03 18:03:50.862822: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000000 of size 256 next 3
2021-12-03 18:03:50.862825: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000100 of size 256 next 4
2021-12-03 18:03:50.862827: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000200 of size 256 next 5
2021-12-03 18:03:50.862830: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000300 of size 256 next 6
2021-12-03 18:03:50.862833: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000400 of size 256 next 9
2021-12-03 18:03:50.862856: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000500 of size 256 next 10
2021-12-03 18:03:50.862859: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000600 of size 256 next 11
2021-12-03 18:03:50.862863: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000700 of size 256 next 14
2021-12-03 18:03:50.862885: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000800 of size 256 next 15
2021-12-03 18:03:50.862889: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000900 of size 512 next 18
2021-12-03 18:03:50.862892: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000b00 of size 256 next 19
2021-12-03 18:03:50.862895: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000c00 of size 256 next 20
2021-12-03 18:03:50.862899: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000d00 of size 256 next 51
2021-12-03 18:03:50.862902: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000e00 of size 256 next 21
2021-12-03 18:03:50.862933: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6000f00 of size 256 next 24
2021-12-03 18:03:50.862937: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6001000 of size 256 next 25
2021-12-03 18:03:50.862941: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6001100 of size 1024 next 28
2021-12-03 18:03:50.862945: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6001500 of size 256 next 29
2021-12-03 18:03:50.862948: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6001600 of size 256 next 30
2021-12-03 18:03:50.862967: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6001700 of size 1024 next 31
2021-12-03 18:03:50.862971: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 7f4da6001b00 of size 1024 next 34
2021-12-03 18:03:50.862974: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6001f00 of size 256 next 36
2021-12-03 18:03:50.862998: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6002000 of size 256 next 37
2021-12-03 18:03:50.863002: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6002100 of size 2048 next 40
2021-12-03 18:03:50.863027: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6002900 of size 256 next 41
2021-12-03 18:03:50.863032: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6002a00 of size 256 next 42
2021-12-03 18:03:50.863036: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 7f4da6002b00 of size 3584 next 7
2021-12-03 18:03:50.863055: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6003900 of size 256 next 55
2021-12-03 18:03:50.863059: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6003a00 of size 512 next 16
2021-12-03 18:03:50.863063: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6003c00 of size 1024 next 27
2021-12-03 18:03:50.863094: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6004000 of size 2048 next 33
2021-12-03 18:03:50.863098: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6004800 of size 3072 next 8
2021-12-03 18:03:50.863103: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6005400 of size 2048 next 43
2021-12-03 18:03:50.863107: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6005c00 of size 2048 next 48
2021-12-03 18:03:50.863111: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6006400 of size 2048 next 50
2021-12-03 18:03:50.863115: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 7f4da6006c00 of size 2048 next 52
2021-12-03 18:03:50.863119: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6007400 of size 256 next 53
2021-12-03 18:03:50.863124: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6007500 of size 6912 next 54
2021-12-03 18:03:50.863128: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6009000 of size 279552 next 13
2021-12-03 18:03:50.863132: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da604d400 of size 442368 next 17
2021-12-03 18:03:50.863137: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da60b9400 of size 2064384 next 22
2021-12-03 18:03:50.863143: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da62b1400 of size 589824 next 12
2021-12-03 18:03:50.863162: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6341400 of size 11206656 next 35
2021-12-03 18:03:50.863166: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da6df1400 of size 2359296 next 26
2021-12-03 18:03:50.863193: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da7031400 of size 2359296 next 39
2021-12-03 18:03:50.863197: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 7f4da7271400 of size 4718592 next 38
2021-12-03 18:03:50.863201: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da76f1400 of size 4718592 next 32
2021-12-03 18:03:50.863226: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 7f4da7b71400 of size 14155776 next 45
2021-12-03 18:03:50.863231: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da88f1400 of size 9437184 next 44
2021-12-03 18:03:50.863236: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da91f1400 of size 11115520 next 1
2021-12-03 18:03:50.863240: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da9c8b000 of size 1280 next 2
2021-12-03 18:03:50.863244: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4da9c8b500 of size 9437184 next 47
2021-12-03 18:03:50.863249: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4daa58b500 of size 9437184 next 49
2021-12-03 18:03:50.863253: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 7f4daae8b500 of size 253935360 next 56
2021-12-03 18:03:50.863258: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4dba0b7400 of size 253935360 next 46
2021-12-03 18:03:50.863262: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f4dc92e3300 of size 253935360 next 23
2021-12-03 18:03:50.863267: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 7f4dd850f200 of size 3938061824 next 18446744073709551615
2021-12-03 18:03:50.863271: I tensorflow/core/common_runtime/bfc_allocator.cc:1071] Summary of in-use Chunks by size:
2021-12-03 18:03:50.863277: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 23 Chunks of size 256 totalling 5.8KiB
2021-12-03 18:03:50.863282: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 512 totalling 1.0KiB
2021-12-03 18:03:50.863302: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 1024 totalling 3.0KiB
2021-12-03 18:03:50.863307: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1280 totalling 1.2KiB
2021-12-03 18:03:50.863326: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 5 Chunks of size 2048 totalling 10.0KiB
2021-12-03 18:03:50.863330: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 3072 totalling 3.0KiB
2021-12-03 18:03:50.863334: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 6912 totalling 6.8KiB
2021-12-03 18:03:50.863339: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 279552 totalling 273.0KiB
2021-12-03 18:03:50.863343: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 442368 totalling 432.0KiB
2021-12-03 18:03:50.863347: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 589824 totalling 576.0KiB
2021-12-03 18:03:50.863351: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2064384 totalling 1.97MiB
2021-12-03 18:03:50.863355: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 2359296 totalling 4.50MiB
2021-12-03 18:03:50.863359: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 4718592 totalling 4.50MiB
2021-12-03 18:03:50.863364: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 9437184 totalling 27.00MiB
2021-12-03 18:03:50.863368: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 11115520 totalling 10.60MiB
2021-12-03 18:03:50.863372: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 11206656 totalling 10.69MiB
2021-12-03 18:03:50.863377: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 253935360 totalling 484.34MiB
2021-12-03 18:03:50.863381: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 544.88MiB
2021-12-03 18:03:50.863385: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 4782227456 memory_limit_: 4782227456 available bytes: 0 curr_region_allocation_bytes_: 9564454912
2021-12-03 18:03:50.863391: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats:
Limit: 4782227456
InUse: 571349248
MaxInUse: 825284608
NumAllocs: 121
MaxAllocSize: 253935360
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2021-12-03 18:03:50.863400: W tensorflow/core/common_runtime/bfc_allocator.cc:474] **_____***********__________________________________________________________________________________
2021-12-03 18:03:50.863427: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops.cc:684 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[1,64,5640,3752] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "organizeSpreadsheet.py", line 107, in <module>
main()
File "organizeSpreadsheet.py", line 88, in main
objects_from_image = labelObjectFromImage(path_to_images, directory_filename)
File "organizeSpreadsheet.py", line 55, in labelObjectFromImage
yhat = model.predict(img, batch_size=1)
File "/home/jr/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/jr/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:
Detected at node 'vgg16/block1_conv1/Conv2D' defined at (most recent call last):
File "organizeSpreadsheet.py", line 107, in <module>
main()
File "organizeSpreadsheet.py", line 88, in main
objects_from_image = labelObjectFromImage(path_to_images, directory_filename)
File "organizeSpreadsheet.py", line 55, in labelObjectFromImage
yhat = model.predict(img, batch_size=1)
File "/home/jr/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1911, in predict
tmp_batch_outputs = self.predict_function(iterator)
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1730, in predict_function
return step_function(self, iterator)
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1719, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1712, in run_step
outputs = model.predict_step(data)
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1680, in predict_step
return self(x, training=False)
File "/home/jr/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1096, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File "/home/jr/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/functional.py", line 451, in call
return self._run_internal_graph(
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/functional.py", line 589, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File "/home/jr/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/home/jr/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1096, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File "/home/jr/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/home/jr/.local/lib/python3.8/site-packages/keras/layers/convolutional.py", line 248, in call
outputs = self.convolution_op(inputs, self.kernel)
File "/home/jr/.local/lib/python3.8/site-packages/keras/layers/convolutional.py", line 233, in convolution_op
return tf.nn.convolution(
Node: 'vgg16/block1_conv1/Conv2D'
OOM when allocating tensor with shape[1,64,5640,3752] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node vgg16/block1_conv1/Conv2D}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[Op:__inference_predict_function_528]
CodePudding user response：
you are already using batch_size = 1.
- check if you are using gpu by checking the logs when you are importing tensorflow.
- try to resize the image before predicting with
tf.image.resize(image, [small_height,small_width,N_channels])