Error while computing Surface Normal using Camera Intrinsic Matrix in OpenCV-CodePudding

I am trying to compute surface normals in OpenCV. Well, this should be quick and easy but I don't know why it is not working. Below is the code:

> import cv2
> img_color = cv2.imread("color.png")
> img_depth = cv2.imread("depth.png", cv2.CV_16UC1) # millimeters
> img_color.shape, img_color.dtype
  ((720, 1280, 3), dtype('uint8'))

> img_depth.shape, img_depth.dtype
  ((720, 1280), dtype('uint16'))

> k = np.array([[ fx,  0, cx ],
                [  0, fy, cy ],
                [  0,  0,  1 ]])
> k
  array([[900,   0, 640],
         [  0, 900, 360],
         [  0,   0,   1]])
  
> img_depth_m = img_depth.astype('float32') * 0.001 # meter
> rows, cols = img_depth_m.shape
> normal_estimator = cv2.rgbd.RgbdNormals_create(rows, cols, cv2.CV_32F, k, 3)
> normals = normal_estimator.apply( img_depth_m )

It throws the following error:

error  Traceback (most recent call last)
/tmp/ipykernel_19178/1208043521.py in <module>
----> 4 normals = normal_estimator.apply( img_depth_m )
error: OpenCV(4.2.0) ../contrib/modules/rgbd/src/normal.cpp:776: 
error: (-215:Assertion failed) points3d_ori.channels() == 3 in function 'operator()'

It seems that OpenCV is expecting a 3 Channel input matrix. However, I looked at the docstring in the notebook, it says:

Docstring:
apply(points[, normals]) -> normals
.   Given a set of 3d points in a depth image, compute the normals at each point.
.        * @param points a rows x cols x 3 matrix of CV_32F/CV64F or a rows x cols x 1 CV_U16S
.        * @param normals a rows x cols x 3 matrix
Type:      builtin_function_or_method

Anyway, how to compute surface normals using camera intrinsic matrix in OpenCV?

PS: I am using OpenCV v4.2.0 on Ubuntu 20.04 LTS.

CodePudding user response：

There is a function called depthTo3d in OpenCV to do convert depth image into 3D points. Please see the following code snippet:

In [1]: import cv2

In [2]: cv2.rgbd.depthTo3d?
Docstring:
depthTo3d(depth, K[, points3d[, mask]]) -> points3d
.   Converts a depth image to an organized set of 3d points.
.      * The coordinate system is x pointing left, y down and z away from the camera
.      * @param depth the depth image (if given as short int CV_U, it is assumed to be the depth in millimeters
.      *              (as done with the Microsoft Kinect), otherwise, if given as CV_32F or CV_64F, it is assumed in meters)
.      * @param K The calibration matrix
.      * @param points3d the resulting 3d points. They are of depth the same as `depth` if it is CV_32F or CV_64F, and the
.      *        depth of `K` if `depth` is of depth CV_U
.      * @param mask the mask of the points to consider (can be empty)
Type:      builtin_function_or_method

In [3]: points3d = cv2.rgbd.depthTo3d(img_depth, k)

In [4]: normal_estimator = cv2.rgbd.RgbdNormals_create(rows, cols, cv2.CV_32F, k, 3)

In [5]: normals = normal_estimator.apply( points3d )