I am making a video player using pygame, it takes numpy arrays of frames from a video file and streams them to a pygame window using a buffer.
I'm getting a weird issue where CPU usage is increasing (program is slowing down) over time, then CPU usage is sharply decreasing (program is speeding up). Memory usage stays pretty much constant, so I don't think it's a memory leak. This affects the program as I'm trying to stream a video, and long processing times turn it into a slideshow.
I would expect fluctuations, but not that steep. I cannot seem to figure what is going on here, I would really appreciate some help in debugging this!
Source code to replicate the issue at the end of the question. You can use any
.mp4 file, but the exact one I used is 
import cv2
import time
import pygame
import gc
from datetime import datetime
#Debug code
runtime_start = str(datetime.now().strftime("%H-%M-%S"))
def log(prefix, message):
string = "[" str(datetime.now().strftime("%H:%M:%S")) "] " "(" str(prefix) ") " str(message)
print(string)
with open(file="log" runtime_start ".txt", mode="a", encoding="utf-8") as file:
file.write(string "\n")
### ### ###
#Globals
buffer_size = 3
frame_buffer = []
### ### ###
#Functions
def loadVideoFileToMemory(file):
"""
Load file to memory.
"""
global capture
capture = cv2.VideoCapture(file)
def getFrameFromVideo(file, frame_number):
"""
Captures a frame from the loaded file, returns a numpy image array.
"""
#capture = cv2.VideoCapture(file)
capture.set(cv2.CAP_PROP_POS_FRAMES, frame_number)
ret, frame = capture.read()
#image = ImageTk.PhotoImage(image=Image.fromarray(frame))
image = cv2.resize(frame, (90*6, 160*6))
del frame
gc.collect()
return image
def setBuffer(start_frame):
"""
Fills the buffer at a set starting frame before playing the video.
"""
frame_buffer.clear()
for i in range(buffer_size):
frame_buffer.append(getFrameFromVideo("test_video1.mp4", start_frame i))
def updateBuffer(frame):
"""
Updates frame_buffer, deals with garabge collection.
"""
del frame_buffer[0]
gc.collect()
if len(frame_buffer) < buffer_size: #Limits to buffer size
frame_array = getFrameFromVideo("test_video1.mp4", frame buffer_size)
frame_buffer.append(frame_array)
del frame_array
gc.collect()
def displayImageToPygame(image, window):
"""
Converts a numpy image array to a pygame surface and blits to screen.
"""
window.blit(pygame.surfarray.make_surface(image), (0,0))
def createPygameWindow():
"""
Creates a pygame window, sets the frame_buffer and starts the main thread.
"""
window = pygame.display.set_mode((160*6, 90*6))
pygame.display.flip()
#Starting frame
current_frame = 0
setBuffer(current_frame) #Set from this position
displayImageToPygame(frame_buffer[0], window) #Show starting frame
pygameThread(window, current_frame)
def pygameThread(window, current_frame):
"""
Main thread. Plays the video.
"""
running = True
while running:
for event in pygame.event.get():
if event.type == pygame.QUIT:
running = False
timer_start = time.time() #DEBUG: PROCESSING TIME
current_frame = 1
displayImageToPygame(frame_buffer[1], window)
updateBuffer(current_frame)
pygame.display.flip()
#DEBUG: POST TO LOG.TXT
log("Variable: current_frame", current_frame)
log("Variable: len(frame_buffer)", len(frame_buffer))
timer_end = time.time() #DEBUG: PROCESSING TIME
log("Process time", str(round(timer_end - timer_start, 2)) " seconds")
###
### ### ###
video_file = "test_video1.mp4" #Replace this with a path to an .mp4 file. You can download the exact file I'm using here: https://drive.google.com/file/d/1_tfTVHmaoTEYxkLrjVS8NiAvdes3Gd77/view?usp=share_link
loadVideoFileToMemory(video_file)
createPygameWindow()
CodePudding user response：
Ok, so i could reproduce what you describe (which was very easy with the sources you provided).
Removing the GC also did nothing for me, as you already observed.
Now i did the following test: instead of getting each frame in order, i got a random frame from the video and recorded those times.
Here's the (trivial) code snippet to do that:
from random import randint, seed
[..]
def pygameThread(window, current_frame):
"""
Main thread. Plays the video.
"""
seed()
[..]
current_frame = 1
rand_frame = randint(1,500) # taking one of the first 500, doesn't really matter
displayImageToPygame(frame_buffer[1], window)
#updateBuffer(current_frame)
updateBuffer(rand_frame)
and more interestingly here's the recorded timing:
[09:02:37] Current_frame 202 len(frame_buffer)3 Process time 0.06 seconds
[09:02:37] Current_frame 423 len(frame_buffer)3 Process time 0.09 seconds
[09:02:37] Current_frame 258 len(frame_buffer)3 Process time 0.03 seconds
[09:02:37] Current_frame 464 len(frame_buffer)3 Process time 0.1 seconds
[09:02:37] Current_frame 176 len(frame_buffer)3 Process time 0.04 seconds
[09:02:37] Current_frame 463 len(frame_buffer)3 Process time 0.1 seconds
[09:02:37] Current_frame 425 len(frame_buffer)3 Process time 0.09 seconds
[09:02:37] Current_frame 154 len(frame_buffer)3 Process time 0.11 seconds
[09:02:37] Current_frame 54 len(frame_buffer)3 Process time 0.05 seconds
[09:02:37] Current_frame 110 len(frame_buffer)3 Process time 0.08 seconds
[09:02:38] Current_frame 133 len(frame_buffer)3 Process time 0.1 seconds
[09:02:38] Current_frame 41 len(frame_buffer)3 Process time 0.04 seconds
[09:02:38] Current_frame 471 len(frame_buffer)3 Process time 0.11 seconds
[09:02:38] Current_frame 458 len(frame_buffer)3 Process time 0.1 seconds
[09:02:38] Current_frame 44 len(frame_buffer)3 Process time 0.05 seconds
[09:02:38] Current_frame 406 len(frame_buffer)3 Process time 0.07 seconds
[09:02:38] Current_frame 66 len(frame_buffer)3 Process time 0.06 seconds
[09:02:38] Current_frame 439 len(frame_buffer)3 Process time 0.09 seconds
[09:02:38] Current_frame 32 len(frame_buffer)3 Process time 0.04 seconds
[09:02:38] Current_frame 89 len(frame_buffer)3 Process time 0.07 seconds
[09:02:38] Current_frame 221 len(frame_buffer)3 Process time 0.06 seconds
So as we can see, just accessing the frames takes a variable amount of time, independant of processing time. My educated guess is that accessing key frames in the stream is fast, while reconstructing the following frames takes longer. Maybe try running some decompression first, before running your algorithm ?
You should also run this test for a longer time than i have and plot it again, to make sure that there really is no increase over time.
CodePudding user response：
Seeking is random access in a media file.
Setting CAP_PROP_POS_FRAMES is seeking.
Do not seek if you don't have to. For most video codecs, it is more costly than sequential access. It can also be imprecise because seeking may decide, because it's cheaper to do that, to merely jump to the nearest "keyframe", instead of the exact time index you want.
Your access pattern appears to be one jump followed by sequential decoding from there. For this entire operation, you should seek once, then decode sequentially.
Calling .read() on the VideoCapture object is sequential decoding. It automatically advances in the video.