streaming Opencv videocapture frames using GStreamer in python for webcam-CodePudding

i am trying to stream a videocapture over network. I have used fastapi and uvicorn for this and it worked well but now i am moving to wireless network and the network can't handle the stream, im getting 2-3fps with 5 sec lag. I read that gstreamer is the best way to stream the frames, although i will need a decoder on the receiving end of the stream.

this is my sender:

Sender.py

import time
import cv2

fps = 52
frame_width = 640
frame_height = 360
flip = 0
camSet='v4l2src device=/dev/video0 ! video/x-raw,width=640,height=360 ! nvvidconv flip-method=' str(flip) ' \
        ! video/x-raw(memory:NVMM), format=I420, width=640, height=360 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert \
        ! video/x-raw, format=BGR enable-max-performance=1 ! appsink '
cam=cv2.VideoCapture(camSet,cv2.CAP_GSTREAMER)

    gst_str_rtp = " appsrc ! videoconvert ! videoscale ! video/x-raw,format=I420,width=640,height=360,framerate=52/1 !  videoconvert !\
     x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay ! \
udpsink host=0.0.0.0 port=8000"

if cam.isOpened() is not True:
    print("Cannot open camera. Exiting.")
    quit()
    
fourcc = cv2.VideoWriter_fourcc(*'H264')
out = cv2.VideoWriter(gst_str_rtp, fourcc, 52, (frame_width, frame_height), True)

while True:
    ret, frame = cam.read()

    cv2.imshow('webcam',frame)
    out.write(frame)
    cv2.moveWindow('webcam',0,0)
    if cv2.waitKey(1)==ord('q'):
        break

cam.release()
out.release()
cv2.destroyAllWindows()

I have a few pipeline examples for the out but i am not sure which even works, like these:

gst_str_rtp ="appsrc ! videoconvert ! video/x-raw,width=1280,height=720 ! queue ! x264enc ! h264parse\
     ! rtph264pay ! udpsink host=127.0.0.1 port=8000"

gst_str_rtp = "appsrc ! video/x-raw, format=I420 ! queue ! videoconvert ! \
      width=640,height=360,framerate=52/1 ! nvvidconv ! omxh264enc ! \
          video/x-h264, stream-format=byte-stream ! h264parse ! rtph264pay pt=96 config-interval=1 ! \
        udpsink host=127.0.0.1 port=8000"

my receiver is this: receiver.py

global video_frame
video_frame = None

cv2.namedWindow('stream')

# Use locks for thread-safe viewing of frames in multiple browsers
# locks prevent the program from getting user inputs from multiple sources
# lock helps syncronize the program
global thread_lock 
thread_lock = threading.Lock()

app = FastAPI()

def main():
    global video_frame
    camSet='udpsrc host=0.0.0.0 port=8000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, \
        encoding-name=(string)H264,\
         payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! appsink'
    video_capture = cv2.VideoCapture(camSet ,cv2.CAP_GSTREAMER)#initalizes our video feed
    # video_capture.set(cv2.CAP_PROP_BUFFERSIZE, 0)#sets camera buffer to 0 meaning we wont get double frames
    print("reading frames")

    while True:
        
        ret, frame = video_capture.read()

        if not ret:
            print('empty frame')
            break

        cv2.imshow(frame,'stream')
        # with thread_lock:
        #     video_frame = frame.copy()



def encodeFrame():#encode frame function takes the frames and encoding them to jpg and encodes them again to bytes so we can stream them
    global thread_lock 
    while True:
        # Acquire thread_lock to access the global video_frame object
        with thread_lock:
            global video_frame
            if video_frame is None:
                continue
            return_key, encoded_image = cv2.imencode(".jpg", video_frame)
            if not return_key:
                continue

        # Output image as a byte array
        yield(b'--frame\r\n' b'Content-Type: image/jpeg\r\n\r\n'   
            bytearray(encoded_image)   b'\r\n')


@app.get("/")
async def video_feed():#our stream function, streams the frames encodeFrame has made to the IP and port we chose
    return StreamingResponse(encodeFrame(), media_type="multipart/x-mixed-replace;boundary=frame")



if __name__ == '__main__':
    # Create a thread and attach the method that captures the image frames, to it
    process_thread = threading.Thread(target=main)

    # Start the thread
    process_thread.start()
    uvicorn.run(app, host="0.0.0.0", port=9000, access_log=True)

i am using flask and uvicorn to restream the images to my java server can read them . either my sender encoding or receiver decoding isn't working or maybe both are wrong and i am not sure which or even if i am going the right way, i am fairly new to gstreamer and pipelines. i would appreciate any help with this and suggestions that wouldn't require me to restream are also needed.

CodePudding user response：

Not sure this will solve your case, but the following may help:

There seems to be typo in the camera capture, where enable-max-performance=1 is not appropriate in video caps. This item is rather a plugin's property (probably from an encoder). It may be better to set framerate in case your camera driver provides other framerates with this resolution, otherwise you'll face a mismatch with writer fps.

camSet='v4l2src device=/dev/video0 ! video/x-raw,width=640,height=360,framerate=52/1 ! nvvidconv flip-method=' str(flip) ' ! video/x-raw(memory:NVMM), format=I420, width=640, height=360 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! queue ! appsink drop=1'

Multicast may hog wifi. Better stream to your receiver only:

... ! udpsink host=<receiver_IP> port=8000 auto-multicast=0

You would simply receive on receiver host only with:

'udpsrc port=8000 auto-multicast=0 ! application/x-rtp,media=video,encoding-name=H264 ! rtpjitterbuffer latency=300 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1'

# Variant for NVIDIA:
'udpsrc port=8000 auto-multicast=0 ! application/x-rtp,media=video,encoding-name=H264 ! rtpjitterbuffer latency=300 ! rtph264depay ! decodebin ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1'

When using gstreamer backend, note that the 4cc code is useless. You may use HW accelerated encoder such as:

gst_str_rtp = "appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),format=NV12,width=640,height=360,framerate=52/1 ! nvv4l2h264enc insert-sps-pps=1 insert-vui=1 idrinterval=30 ! h264parse ! rtph264pay ! udpsink host=<receiver_IP> port=8000 auto-multicast=0"
out = cv2.VideoWriter(gst_str_rtp, cv2.CAP_GSTREAMER, 0, float(52), (frame_width, frame_height), True)