Wall edge mapping and localization from RGB-D Data

Manual methods are predominantly used for sealant application in construction, which often has shortcomings such as training of skilled personnel, inconsistent application quality, and improper installation leading to premature failure of the sealant. Thus, I aim to create a monitoring system to verify the application of wall sealant is appropriate.

I have leveraged the high-resolution depth sensing capabilities of RGB-D cameras, and the development of powerful mini computation hardware to construct an autonomous sealant application system is proposed, eliminating the need for 3D models and manual human operation. The system utilizes an RGB-D camera to capture point cloud data and employs point cloud edge estimation to detect edge and corner positions, providing precise and accurate location information. This information is then used to generate a path for a robot manipulator, enabling automatic sealant application on walls.

As shown in Figure 1 the silicon application automation system consists of a vision and manipulation subsystem. The manipulation subsystem is in charge of moving the sealant applicator to the desired edge position and comprises the mobile platform, manipulator, sealant applicator, and portable computer. The vision subsystem consists of an Intel RealSense Depth Camera D435 camera and a processing unit namely the KRIA KR260 Robotics Starter Kit. Its main function is to acquire the 3D point cloud of the environment and horizontal edge extraction.

As seen from Figure 1, both subsystems are controlled using the portable computer. The portable computer selected for this application is the KRIA KR260 Robotics Starterkit which comes equipped with all required interfaces widely used in robotics vision and sensing. Additionally, it comes equipped with an active thermal solution to ensure performance even with heavy calculation as point cloud processing is a computationally heavy process also the solution will be deployed autonomously so the small form factor is greatly favored.

This project utilizes an edge detection algorithm which is built on top of the mean shift algorithm to develop an adaptive density-based threshold for edge classification. In other words, the edge cloud can be determined because a smooth surface typically will have a smaller shift in its mean when compared with an edge point. Thus, a threshold can be used to identify between edge and non-edge points.

Although the edge cloud has been extracted from the raw point cloud, they are still indistinguishable edges. Therefore, the edge cloud will need to go through further segmentation to extract the inlier edge point cloud using RANSAC.

Finally, the extracted inline point cloud is refitted using the least square method to localize the edge as shown in fig 4.

From Figure 5 the edge estimated given the edge point is shown, the line represents the estimated edge position and the blue points are the edge points. The accuracy and effectiveness of the algorithm were evaluated by comparing the predicted edge locations with the expected positions in the simulated environment. This comparison provides a quantifiable metric for assessing the reliability of the edge extraction algorithm.

Figure 5 shows the impact of Gaussian noise on the algorithm's error. It was observed that the level of noise present in the raw point cloud has a significant effect on the overall error. Overall, the proposed solution demonstrated the highest stability and accuracy when the Gaussian noise parameter σ was below 0.004. However, when σ reached 0.004, there was a noticeable increase in the overall error, although it remained stable at around 1 mm. These results highlight the algorithm's sensitivity to noise and provide valuable insights for optimizing its performance in practical sealant application scenarios.

In conclusion, the proposed method of automating sealant application using edge extraction and localization on raw point cloud data has shown promising results. The solution is highly accurate, with a precision of around 0.1 cm, and has a runtime of approximately one minute, making it an efficient solution for the construction industry. This innovation was aimed at addressing the issue of the workforce required for the sealant application task, making it more manageable for construction companies.

The demo video can be found https://drive.google.com/file/d/1Z_x266xVe77Tn2BS5vbJpBHlCrpiPIpl/view?usp=sharing