Why does OpenCV read video faster than FFMPEG?-CodePudding

I noticed that OpenCV reads video frames almost 2x faster than FFMPEG.

Why is that? I thought all OpenCV does is call FFMPEG under the hood, possibly adding its own overhead.

Here's the code I use to obtain these results

import cv2
import time
import numpy as np

cap = cv2.VideoCapture("BigBuckBunny.mp4", apiPreference=cv2.CAP_FFMPEG)
frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

start = time.perf_counter()
while True:
    ret, frame = cap.read()
    if ret is False:
        break
    assert frame.shape == (720, 1280, 3)
    assert frame.dtype == np.uint8
end = time.perf_counter()

print(f"{frames/(end-start):.1f} frames per second")
# Output: 692.3 frames per second

cap.release()

For FFMPEG (using the python-ffmpeg library:

import ffmpeg
import numpy as np
import time

vid_info = ffmpeg.probe("BigBuckBunny.mp4")['streams'][1]
frames = int(vid_info['nb_frames'])

process1 = (
    ffmpeg
    .input("BigBuckBunny.mp4")
    .output('pipe:', format='rawvideo', pix_fmt='bgr24')
)
print(process1.compile())
# Output: ['ffmpeg', '-i', 'BigBuckBunny.mp4', '-f', 'rawvideo', '-pix_fmt', 'bgr24', 'pipe:']


process1 = process1.run_async(pipe_stdout=True)

start = time.perf_counter()
while True:
    in_bytes = process1.stdout.read(1280 * 720 * 3)
    if not in_bytes:
        break
    frame = np.frombuffer(in_bytes, np.uint8).reshape([720, 1280, 3])
end = time.perf_counter()
print(f"{frames/(end-start):.1f} frames per second")
# Output: 373.6 frames per second
process1.wait()

Here's information about the video (~10 minutes length)

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'BigBuckBunny.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isomavc1mp42
    creation_time   : 2010-01-10T08:29:06.000000Z
  Duration: 00:09:56.47, start: 0.000000, bitrate: 2119 kb/s
  Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      creation_time   : 2010-01-10T08:29:06.000000Z
      handler_name    : (C) 2007 Google Inc. v08.13.2007.
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1991 kb/s, 24 fps, 24 tbr, 24k tbn, 48 tbc (default)
    Metadata:
      creation_time   : 2010-01-10T08:29:06.000000Z
      handler_name    : (C) 2007 Google Inc. v08.13.2007.
      vendor_id       : [0][0][0][0]

And FFMPEG and OpenCV versions:

ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers

opencv-python-headless        4.6.0.66

CodePudding user response：

It may be due to the fact how the libraries are implemented and the overhead of pipes.

cv2 implemented in c and uses libavcodec library (part of ffmpeg) for decoding frames:

https://github.com/opencv/opencv/blob/97c6ec6d49cb78321eafe6fa220ff80ebdc5e2f4/modules/videoio/src/cap_ffmpeg_impl.hpp#L1299

cv2 does not use ffmpeg as binary executable and uses the part of ffmpeg specifically designed for reading/decoding frames.

ffmpeg-python is a wrapper around ffmpeg executable (which in turn uses libavcodec under the hood too) that calls ffmpeg with subprocess and returns results through the pipes.

CodePudding user response：

Good stuff. Interestingly, under Windows 10 & FFmpeg 5.0.1, my results reversed:

OpenCV: 491.9 frames per second
FFmpeg-Python: 519.4 frames per second

And out of curiosity, I tried my ffmpegio to see how much overhead there is and, it came out: 507.5 frames per second (code below)

So both FFmpeg-Python and FFmpegIO scoring in the same ballpark, I'd say these numbers aren't fluke.

So, it's likely Win/Linux or 4.4/5.0 difference or both. So I retested FFmpeg-Python with FFmpeg 4.4.1: 428.1 frames per second (log reporting speed=17.9x down from x21.7. So, I believe FFmpeg devs optimized the color conversion codebase in v5 releases to match or surpass OpenCV.

import ffmpegio
import time

frames = ffmpegio.probe.video_streams_basic("BigBuckBunny.mp4")[0]['nb_frames']

print(frames)

with ffmpegio.open("BigBuckBunny.mp4",'rv') as f:
    start = time.perf_counter()
    for frame in f:
        pass
    end = time.perf_counter()

print(f"{frames/(end-start):.1f} frames per second")