How to project a chessboard in camera coordinates to real-world coordinates in OpenCV using rvecs an-CodePudding

The

Code:

import numpy as np import cv2 as cv import matplotlib.pyplot as plt # termination criteria criteria = (cv.TERM_CRITERIA_EPS cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) # prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0) objp = np.zeros((12*13,3), np.float32) objp[:,:2] = np.mgrid[0:12,0:13].T.reshape(-1,2) # Arrays to store object points and image points from all the images. objpoints = [] # 3d point in real world space imgpoints = [] # 2d points in image plane. img = cv.imread('image.tif') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Find the chess board corners ret, corners = cv.findChessboardCorners(gray, (12,13), None) # If found, add object points, image points (after refining them) if ret == True: objpoints.append(objp) corners2 = cv.cornerSubPix(gray,corners, (11,11), (-1,-1), criteria) imgpoints.append(corners) # Draw and display the corners cv.drawChessboardCorners(img, (12,13), corners2, ret) cv.imshow('img', img) cv.waitKey(2000) cv.destroyAllWindows() ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None) #Plot undistorted h, w = img.shape[:2] newcameramtx, roi = cv.getOptimalNewCameraMatrix(mtx, dist, (w,h), 1, (w,h)) dst = cv.undistort(img, mtx, dist, None, newcameramtx) # crop the image x, y, w, h = roi dst = dst[y:y h, x:x w] plt.figure() plt.imshow(dst) plt.savefig("undistorted.png", dpi = 300) plt.close()

Undistorted image:

The undistorted image indeed has straight lines. However, in order to test the calibration procedure I would like to further transform the image into real-world coordinates using the rvecs and tvecs outputs of cv.calibrateCamera. From the

What I have tried

Following the comment of @Christoph Rackwitz, I found

Any ideas where the problem is?

CodePudding user response：

Every camera has its own

For the input image of:

The Explanation

Import the necessary libraries:

import cv2 import numpy as np

Define a function, triangles, that will take in an array of coordinates, points, and yield lists of 3 indices of the array for triangles that will cover the area of the original array of coordinates:

def triangles(points): points = np.where(points, points, 1) subdiv = cv2.Subdiv2D((*points.min(0), *points.max(0))) for pt in points: subdiv.insert(tuple(map(int, pt))) for pts in subdiv.getTriangleList().reshape(-1, 3, 2): yield [np.where(np.all(points == pt, 1))[0][0] for pt in pts]

Define a function, crop, that will take in an image array, img, and an array of three coordinates, pts. It will return a rectangular segment of the image just large enough to fit the triangle formed by the three point, and return the array of three coordinates transferred to the top-left corner of image:

def crop(img, pts): x, y, w, h = cv2.boundingRect(pts) img_cropped = img[y: y h, x: x w] pts[:, 0] -= x pts[:, 1] -= y return img_cropped, pts

Define a function, warp, that will take in 2 image arrays, img1 and img2, and 2 arrays of coordinates, pts1 and pts2. It will utilize the triangles function defined before iterate through the triangles from the first array of coordinates, the crop function defined before to crop both images at coordinates corresponding to the triangle indices and use the cv2.warpAffine() method to warp the image at the current triangle of the iterations:

def warp(img1, img2, pts1, pts2): img2 = img2.copy() for indices in triangles(pts1): img1_cropped, triangle1 = crop(img1, pts1[indices]) img2_cropped, triangle2 = crop(img2, pts2[indices]) transform = cv2.getAffineTransform(np.float32(triangle1), np.float32(triangle2)) img2_warped = cv2.warpAffine(img1_cropped, transform, img2_cropped.shape[:2][::-1], None, cv2.INTER_LINEAR, cv2.BORDER_REFLECT_101) mask = np.zeros_like(img2_cropped) cv2.fillConvexPoly(mask, np.int32(triangle2), (1, 1, 1), 16, 0) img2_cropped *= 1 - mask img2_cropped = img2_warped * mask return img2

Read in the image of the distorted chessboard, convert it to grayscale and detect the corners of the chessboard:

img = cv2.imread("image.png") gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) criteria = (cv2.TERM_CRITERIA_EPS cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001) ret, corners = cv2.findChessboardCorners(gray, (12 ,13), None) corners2 = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)

Define an array of the destination points for each corner detected. If you plot each corner along with their corresponding index in the array, you will see that they are in this order:

So our destination array must be in that order, or we will end up with unreadable results. The x, y, w, h, r, c below will be the destination array of coordinates' top-left x, y position, each square's width & height, and the number of rows & columns of points in the board:

x, y, w, h, r, c = 15, 40, 38, 38, 12, 13 pts1 = np.int32(corners2.squeeze()) arr2 = np.tile(np.arange(c), r).reshape((r, c)) arr1 = np.tile(np.arange(r), c).reshape((c, r)) arr = np.dstack((arr1[:, ::-1] * h y, arr2.T * w x)) pts2 = arr.reshape((r * c, 2))

Finally, show the warped part of the image on a blank image:

cv2.imshow("result", warp(img, np.zeros_like(img), pts1, pts2)) cv2.waitKey(0)