Currently, I have a huge dataset, which did not fit in memory. For training, I used the tf.Dataset to train my neural network. Unfortunately, now I want to verify my results but the dataset printing do not fit into my 32 GB of memory. Is there a way to partially or sequential print a graph or multiple graphs to a file using matplotlib. So after each predict_on_batch I would draw a part of the graph (load image, draw, save image). In the end, I want a complete graph to verify my results. Any ideas or solutions to the "data do not fit in memory, but I need a graph to visualize it"-problem would be appreciated. It is not labeled data so there is no way besides checking it visually.
CodePudding user response:
A way I would approach this problem is to save the information in a list instead of a partial plot.
I am assuming you want your pipeline to look like this:
- create empty plot
- predict on batch
- plot new results
- repeat steps 2 and 3
What I would suggest looks something like this:
- initialize empty lists
- predict on batch
- append the results to lists
- repeat steps 2 and 3
- plot the final arrays