Converting 16 bit depth frame from Intel Realsense D455 to OpenCV Mat in Android Java-CodePudding

I am trying to convert a DepthFrame object that I have obtained from the Intel Realsense D455 camera to an OpenCV Mat object in Java. I can get the the target depth of a pixel using DepthFrame.getDistance(x,y) but I am trying to get the whole matrix so that I can get the distance values in meters, similar to the sample code in their Github repo, which is in C .

I can convert any color image obtained from the camera stream (VideoFrame or colored DepthFrame) to a Mat since they are 8 bits per pixel using the following function:

    public static Mat VideoFrame2Mat(final VideoFrame frame) {
        Mat frameMat = new Mat(frame.getHeight(), frame.getWidth(), CV_8UC3);
        final int bufferSize = (int)(frameMat.total() * frameMat.elemSize());
        byte[] dataBuffer = new byte[bufferSize];
        frame.getData(dataBuffer);
        ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer().get(dataBuffer);
        frameMat.put(0,0, dataBuffer);
        return frameMat;
    }

However, the un-colorized DepthFrame values are 16 bits per pixel, and the above code gives an error when the CV_8UC1 is substituted with CV_16UC1. The error arises because in the Java wrapper of the OpenCV function Mat.put(row, col, data[]), there is a type check that allows only 8 bit Mats to be processed:

    // javadoc:Mat::put(row,col,data)
    public int put(int row, int col, byte[] data) {
        int t = type();
        if (data == null || data.length % CvType.channels(t) != 0)
            throw new UnsupportedOperationException(
                    "Provided data element number ("  
                            (data == null ? 0 : data.length)  
                            ") should be multiple of the Mat channels count ("  
                            CvType.channels(t)   ")");
        if (CvType.depth(t) == CvType.CV_8U || CvType.depth(t) == CvType.CV_8S) {
            return nPutB(nativeObj, row, col, data.length, data);
        }
        throw new UnsupportedOperationException("Mat data type is not compatible: "   t);
    }

Therefore I tried to use the constructor of Mat that accepts the array and wrote the following method:

    public static Mat DepthFrame2Mat(final DepthFrame frame) {
        byte[] dataBuffer = new byte[frame.getDataSize()];
        frame.getData(dataBuffer);
        ByteBuffer buffer = ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer().get(dataBuffer);

        Log.d(TAG, String.format("DepthFrame2Mat: w: %s h: %s capacity: %s remaining %s framedatasize: %s databufferlen: %s framedepth: %s type: %s " ,
                frame.getWidth(), frame.getHeight(), buffer.capacity(), buffer.remaining(), frame.getDataSize(), dataBuffer.length, frame.getBitsPerPixel(), frame.getProfile().getFormat()));
        
        return new Mat(frame.getHeight(), frame.getWidth(), CV_16UC1, buffer);
    }

But now I keep getting the error E/cv::error(): OpenCV(4.5.5) Error: Assertion failed (total() == 0 || data != NULL) in Mat, file /build/master_pack-android/opencv/modules/core/src/matrix.cpp, line 428

Using the log command seen in the function, I am checking if the data is empty or null, but it is not. Moreover, the DepthFrame bit depth and type seems to be correct, too:

D/CvHelpers: DepthFrame2Mat: w: 640 h: 480 capacity: 614400 remaining 0 framedatasize: 614400 databufferlen: 614400 framedepth: 16 type: Z16

What could be the reason of this error? Is there a better way to handle this conversion?

Note: I have checked the SO questions such as this and examples on the web, however, all of them are in C . I don't want to add JNI support just for creating a Mat.

CodePudding user response：

Even though not directly an OpenCV API solution, converting the byte array to short array in Java seems to work:

    public static Mat DepthFrame2Mat(final DepthFrame frame) {
        Mat frameMat = new Mat(frame.getHeight(), frame.getWidth(), CV_16UC1);
        final int bufferSize = (int)(frameMat.total() * frameMat.elemSize());
        byte[] dataBuffer = new byte[bufferSize];
        short[] s = new short[dataBuffer.length / 2];
        frame.getData(dataBuffer);
        ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(s);
        frameMat.put(0,0, s);
        return frameMat;
    }