I'm starting a project that consists of receive video from an RTSP server and showing it in a panel using QT c and GSTreamer. I want to receive every frame as a QImage object to run some features that I need. I know that we can use QTMultimedia to link the Gstreamer and view the video, but in this particular case, I would like to have QImages. I found a way (or i think i did) to get the samples from the video by calling the function "gst_app_sink_pull_sample". We get a GSTSample from it. However, i did not find any way to convert this into, for example, a JPG raw data which is easy to convert into a QImage.

I also found a way to access the data from GSTSample from here: "How to get video stream frame-by-frame from Gstreamer pipeline? (without OpenCV)" but, once again, i've no idea how can i convert this data into a QImage.

 /* Initialize GStreamer */
        gst_init (NULL, NULL);

        /* Create the elements */
        GstElement * source = gst_element_factory_make ("videotestsrc", "source"); // HERE I'M USING THE GSTREAMER SOURCE TEST
        GstElement * sink = gst_element_factory_make ("appsink", "sink");
        GstAppSink *appsink = GST_APP_SINK(sink);

        /* Create the empty pipeline */
         GstElement * pipeline = gst_pipeline_new ("test-pipeline");

        if (!pipeline || !source || !sink) {
            g_printerr ("Not all elements could be created.\n");
            return ;
        }

        /* Build the pipeline */
        gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL);
        if (gst_element_link (source, sink) != TRUE) {
            g_printerr ("Elements could not be linked.\n");
            gst_object_unref (pipeline);
            return ;
        }

        /* Modify the source's properties */
        g_object_set (source, "pattern", 0, NULL);


        /* Start playing */
        ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
        if (ret == GST_STATE_CHANGE_FAILURE) {
            g_printerr ("Unable to set the pipeline to the playing state.\n");
            gst_object_unref (pipeline);
            return;
        }


        // My question starts HERE
        GstSample *sample = nullptr;

        do{

            sample = gst_app_sink_pull_sample(appsink); // Get the frame
            if (!sample) {
                printf("sample is NULL\n");
            }else{
                
                // Get the raw data (or trying to...)
                GstBuffer * buffer = gst_sample_get_buffer(sample);
                GstMapInfo info;
                gst_buffer_map(buffer, &info, GST_MAP_READ);

                // Dumb way to check if the frame is being received properly
                QImage imageAux((const unsigned char*)info.data, 100, 100, QImage::Format_RGB16);
                imageAux.save("out.jpg"); //returns garbage

            }


        }while(sample);

CodePudding user response：

According to that topic

cv::Mat frame(cv::Size(width, height), CV_8UC4, (char *)map.data, cv::Mat::AUTOSTEP);

The actual image data begins from map.data pointer. And have CV_8UC4 format in terms of OpenCV. Construct the QImage with the next constructor QImage(uchar *data, int width, int height, int bytesPerLine, QImage::Format format, ImageCleanupFunction cleanupFunction = nullptr, void *cleanupInfo = nullptr)

You should understand what is CV_8UC4 format. And find appropriate one among QImage::Format list. And find other parameters in that 'map' to push into the constructor. In good case, when you found appropriate format in QImage::Format, push the map.data into uchar *data. In wrong case you should try to find another way to convert CV_8UC4 into something suitable with Qt's images.

Also, the author of the mentioned topic might mistake with CV_8UC4. Then there is the issue what kind of image format under map.data pointer. But the steps to get QImage can be suitable anyway, once you get the right format.

CodePudding user response：

Result from cv::imwrite

The result is not good. The image above results from saving the Mat frame() as image using cv::imwrite("some.jpg", frame); It may return an image like that: videotestsrc image

There is the code:

  GstBuffer * buffer = gst_sample_get_buffer(sample);
                GstMapInfo map;
                GstCaps *caps = gst_sample_get_caps(sample);
                GstStructure *structure = gst_caps_get_structure(caps, 0);
                const int width = g_value_get_int(gst_structure_get_value(structure, "width"));
                const int height = g_value_get_int(gst_structure_get_value(structure, "height"));

                gst_buffer_map(buffer, &map, GST_MAP_READ);

                cv::Mat frame(cv::Size(width, height),  CV_8UC3, (char *)map.data, cv::Mat::AUTO_STEP);

                cv::imwrite("some.jpg", frame);

                QImage imgIn= QImage((uchar*) frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888);
                imgIn.save("out.jpg");

The QImage is quite similar, but now I'm more focused on trying to understand what is happening with v::imwrite("some.jpg", frame); I did try several frames formats like CV_8UC4, CV_8UC1, etc...