Home > Blockchain >  Apache mpm event shared memory error - ffmpeg-python Error while opening encoder for output stream #
Apache mpm event shared memory error - ffmpeg-python Error while opening encoder for output stream #

Time:12-13

EDIT:

I kinda figured out, the problem needs to deal with apache mpm event... When I send my request on the first client, my script is getting executed linear. When I send my request from the second client, he is literally starting at that point in the code where the last request currently is. So that might be a problem with shared memory between those threads, but I'm not an apache professional, maybe somebody has an idea?
I used apache mpm prefork before, so every request is getting his own process and own memory etc., but there was a problem reading jpg files with that one and it worked after I changed apache to mpm event, see the following: https://github.com/python-pillow/Pillow/issues/5834#issue-comment-box

Inside my VM (running same apache version with mpm event) everything is working fine.


I've got a python script with flask running via wsgi on an apache2 webserver. Inside of that script I got following lines (458-461):

video_stream = ffmpeg.input(pathToVideo)
audio_file = ffmpeg.input(pathToAudio)
ffmpeg.concat(video_stream, audio_file, v=1, a=1).output(
     pathToOutVid).run()

So that code snippet successfully combines my video and audio file to a new video file on click on a button. My Problem starts when I am accessing the webpage from 2 clients and start that script. Both have different input audio and video files and different output files, but when they are running at the same time, one of them is passing through successfully and the other client gets the following error:

ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/path/to/movie.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
  Duration: 00:00:11.50, start: 0.000000, bitrate: 1053 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 1050 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Input #1, mp3, from '/path/to/audio.mp3':
  Metadata:
    encoder         : Lavf57.83.100
  Duration: 00:00:11.55, start: 0.025057, bitrate: 128 kb/s
    Stream #1:0: Audio: mp3, 44100 Hz, stereo, s16p, 128 kb/s
    Metadata:
      encoder         : Lavc57.10
Stream mapping:
  Stream #0:0 (h264) -> concat:in0:v0
  Stream #1:0 (mp3) -> concat:in0:a0
  concat:out:a0 -> Stream #0:0 (aac)
  concat:out:v0 -> Stream #0:1 (libx264)
Press [q] to stop, [?] for help
[libx264 @ 0x56011ee5cbc0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
Error initializing output stream 0:1 -- Error while opening encoder for output stream #0:1 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x56011ee5b260] Qavg: -nan
Conversion failed!

but executing that part on the same client without another one parallel is also working correctly with the following output:

ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/path/to/movie.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
  Duration: 00:00:11.50, start: 0.000000, bitrate: 1053 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 1050 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Input #1, mp3, from '/path/to/audio.mp3':
  Metadata:
    encoder         : Lavf57.83.100
  Duration: 00:00:11.55, start: 0.025057, bitrate: 128 kb/s
    Stream #1:0: Audio: mp3, 44100 Hz, stereo, s16p, 128 kb/s
    Metadata:
      encoder         : Lavc57.10
Stream mapping:
  Stream #0:0 (h264) -> concat:in0:v0
  Stream #1:0 (mp3) -> concat:in0:a0
  concat:out:a0 -> Stream #0:0 (aac)
  concat:out:v0 -> Stream #0:1 (libx264)
Press [q] to stop, [?] for help
[libx264 @ 0x55ae2ada7bc0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x55ae2ada7bc0] profile High, level 4.0
[libx264 @ 0x55ae2ada7bc0] 264 - core 152 r2854 e9a5903 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=9 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/path/to/movie-WITH-AUDIO.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
    Stream #0:0: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      encoder         : Lavc57.107.100 aac
    Stream #0:1: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1920x1080, q=-1--1, 30 fps, 15360 tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=   42 fps=0.0 q=0.0 size=       0kB time=00:00:01.34 bitrate=   0.0kbits/s speed=2.68x    
frame=   46 fps= 39 q=0.0 size=       0kB time=00:00:01.48 bitrate=   0.0kbits/s speed=1.25x    
frame=   57 fps= 34 q=29.0 size=       0kB time=00:00:01.85 bitrate=   0.2kbits/s speed= 1.1x    
frame=   70 fps= 30 q=29.0 size=       0kB time=00:00:02.25 bitrate=   0.2kbits/s speed=0.966x    
frame=   82 fps= 28 q=29.0 size=     256kB time=00:00:02.69 bitrate= 778.7kbits/s speed=0.927x    
frame=   92 fps= 26 q=29.0 size=     256kB time=00:00:03.01 bitrate= 694.9kbits/s speed=0.868x    
frame=  102 fps= 25 q=29.0 size=     256kB time=00:00:03.34 bitrate= 627.3kbits/s speed=0.831x    
frame=  114 fps= 25 q=29.0 size=     512kB time=00:00:03.71 bitrate=1129.1kbits/s speed=0.81x    
frame=  134 fps= 26 q=29.0 size=     512kB time=00:00:04.41 bitrate= 950.8kbits/s speed=0.864x    
frame=  152 fps= 27 q=29.0 size=     512kB time=00:00:05.01 bitrate= 836.3kbits/s speed=0.886x    
frame=  169 fps= 27 q=29.0 size=     512kB time=00:00:05.59 bitrate= 749.6kbits/s speed=0.902x    
frame=  189 fps= 28 q=29.0 size=     512kB time=00:00:06.22 bitrate= 674.1kbits/s speed=0.927x    
frame=  203 fps= 28 q=29.0 size=     768kB time=00:00:06.71 bitrate= 937.6kbits/s speed=0.912x    
frame=  215 fps= 27 q=29.0 size=    1024kB time=00:00:07.10 bitrate=1180.7kbits/s speed=0.885x    
frame=  240 fps= 28 q=29.0 size=    1024kB time=00:00:07.94 bitrate=1056.4kbits/s speed=0.921x    
frame=  258 fps= 28 q=29.0 size=    1024kB time=00:00:08.52 bitrate= 984.4kbits/s speed=0.934x    
frame=  281 fps= 29 q=29.0 size=    1024kB time=00:00:09.28 bitrate= 903.2kbits/s speed=0.965x    
frame=  302 fps= 30 q=29.0 size=    1024kB time=00:00:09.98 bitrate= 840.2kbits/s speed=0.984x    
frame=  318 fps= 30 q=29.0 size=    1280kB time=00:00:10.54 bitrate= 994.7kbits/s speed=0.989x    
frame=  335 fps= 30 q=29.0 size=    1280kB time=00:00:11.12 bitrate= 942.8kbits/s speed=0.989x    
frame=  345 fps= 27 q=-1.0 Lsize=    1765kB time=00:00:11.51 bitrate=1255.3kbits/s speed=0.887x    
video:1569kB audio:183kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.768725%
[aac @ 0x55ae2ada6260] Qavg: 1450.385
[libx264 @ 0x55ae2ada7bc0] frame I:2     Avg QP:11.26  size: 96372
[libx264 @ 0x55ae2ada7bc0] frame P:99    Avg QP:18.02  size: 11184
[libx264 @ 0x55ae2ada7bc0] frame B:244   Avg QP:17.15  size:  1253
[libx264 @ 0x55ae2ada7bc0] consecutive B-frames:  2.6%  8.1%  3.5% 85.8%
[libx264 @ 0x55ae2ada7bc0] mb I  I16..4: 54.0% 36.8%  9.2%
[libx264 @ 0x55ae2ada7bc0] mb P  I16..4:  1.7%  8.2%  0.2%  P16..4: 16.8%  2.9%  2.6%  0.0%  0.0%    skip:67.7%
[libx264 @ 0x55ae2ada7bc0] mb B  I16..4:  0.3%  0.3%  0.0%  B16..8:  8.1%  0.2%  0.1%  direct: 1.8%  skip:89.2%  L0:35.2% L1:62.5% BI: 2.3%
[libx264 @ 0x55ae2ada7bc0] 8x8 transform intra:71.5% inter:83.8%
[libx264 @ 0x55ae2ada7bc0] coded y,uvDC,uvAC intra: 18.8% 23.1% 7.1% inter: 2.4% 6.6% 0.1%
[libx264 @ 0x55ae2ada7bc0] i16 v,h,dc,p: 62% 22%  7%  8%
[libx264 @ 0x55ae2ada7bc0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 33% 16% 41%  1%  1%  1%  2%  1%  2%
[libx264 @ 0x55ae2ada7bc0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 25% 12%  6%  8%  6% 11%  6%  8%
[libx264 @ 0x55ae2ada7bc0] i8c dc,h,v,p: 72% 14% 12%  1%
[libx264 @ 0x55ae2ada7bc0] Weighted P-Frames: Y:53.5% UV:48.5%
[libx264 @ 0x55ae2ada7bc0] ref P L0: 75.9% 11.4% 10.0%  2.4%  0.3%
[libx264 @ 0x55ae2ada7bc0] ref B L0: 89.7%  9.5%  0.8%
[libx264 @ 0x55ae2ada7bc0] ref B L1: 97.9%  2.1%
[libx264 @ 0x55ae2ada7bc0] kb/s:1117.06

I tried to reproduce this error inside my virtual machine running the same Ubuntu 18.04 version, same apache version with mpm event, same python version, same ffmpeg-python version and ffmpeg version, but on my localhost VM everything is working without any problems. Now I need help because I have no idea why this encoder cannot be opened (when another is currently using it? - but no problem on my VM?)

In case it helps, I'm also using opencv and got the following part using the h264 encoder:

video = cv.VideoWriter(output, cv.VideoWriter_fourcc('a', 'v', 'c', '1'), 30, 1920, 1080))

The part of the program where this video is written usually takes very long, if that knowledge helps, but there are no problems with the encoder when multiple clients use it at this point.

I really hope somebody can help me and I appreciate the time you are investing for me, thanks for that!
If I forgot to send something or I need to give more detailed information or anything, just text me :)


Versions

  • Ubuntu 18.04 LTS
  • Apache/2.4.29
  • ffmpeg version 3.4.8-0ubuntu0.2
  • Python 3.6.9
  • Flask 2.0.2
  • opencv 3.4.1
  • ffmpeg-python 0.2.0

CodePudding user response:

Okay, for the love of God Im done... needed 4 days for that **** to find out, I had an apache mod activated (fcgid) which literally assign every single line of ur code to a single thread... so I deactivated that one and now its working without any issues...

So in short - what I did:

a2dismod fcgid
systemctl restart apache2

now its working.

Thx to @Rotem for his assistance! Sorry for that weired problem and my own stupidity...

  • Related