Home > Blockchain >  Matplotlib sequentially printing /append to image
Matplotlib sequentially printing /append to image

Time:03-28

Currently, I have a huge dataset, which did not fit in memory. For training, I used the tf.Dataset to train my neural network. Unfortunately, now I want to verify my results but the dataset printing do not fit into my 32 GB of memory. Is there a way to partially or sequential print a graph or multiple graphs to a file using matplotlib. So after each predict_on_batch I would draw a part of the graph (load image, draw, save image). In the end, I want a complete graph to verify my results. Any ideas or solutions to the "data do not fit in memory, but I need a graph to visualize it"-problem would be appreciated. It is not labeled data so there is no way besides checking it visually.

CodePudding user response:

A way I would approach this problem is to save the information in a list instead of a partial plot.

I am assuming you want your pipeline to look like this:

  1. create empty plot
  2. predict on batch
  3. plot new results
  4. repeat steps 2 and 3

What I would suggest looks something like this:

  1. initialize empty lists
  2. predict on batch
  3. append the results to lists
  4. repeat steps 2 and 3
  5. plot the final arrays
  • Related