How to convert CMSampleBuffer to OpenCV's Mat instance in swift-CodePudding

I am working on an OpenCV project where I am taking input from iPhone native camera as CMSampleBuffer now I wanted to create Mat instance that is required in OpenCV for further process

I have found some old post related with it but all are not working in current swift as those are pretty old.

Raw image data from camera like "645 PRO"

How to convert CMSampleBufferRef to IplImage (iOS)

CodePudding user response：

As far as I know openCV is written in C and you will have to interact with it using an Objc or an Objc wrapper. Hence the resources you found are solid and there are many others on working with OpenCV on iOS. A simple swifty wrapper I found online is - https://github.com/Legoless/LegoCV.

There is a full interoperability between Objc/Objc and swift and it is fairly easy to implement. See these two resources for more info - https://developer.apple.com/documentation/swift/imported_c_and_objective-c_apis/importing_objective-c_into_swift https://rderik.com/blog/understanding-objective-c-and-swift-interoperability/

CodePudding user response：

First, convert CMSampleBuffer To UIImage.

extension CMSampleBuffer {
    func asUIImage()-> UIImage? {
        guard let imageBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(self) else {
            return nil
        }
        let ciImage = CIImage(cvPixelBuffer: imageBuffer)
        return convertToUiImage(ciImage: ciImage)
    }
    
    func convertToUiImage(ciImage: CIImage) -> UIImage? {
        let context = CIContext(options: nil)
        context.clearCaches()
        guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else {
            return nil
        }
        let image = UIImage(cgImage: cgImage)
        return image
    }
}

Then you can easily convert UIImage to Mat using OpenCV ios.
OpenCVWrapper.mm

  (cv::Mat) getMatInstance : (UIImage *) uiImage {
    cv::Mat sourceImage;
    UIImageToMat(uiImage, sourceImage);
    return sourceImage;
}