Multimodal Fusion Perception for Rock Sampling Robot in Field Environments

Rock-sampling robots are essential equipment for the intelligence of field geological exploration. However, in actual operations, robots face severe challenges such as unstructured field environments, intense illumination fluctuations, and highly variable rock lithology. Therefore, relying on a self-developed field rock-sampling robot platform, this paper proposes a multimodal fusion perception method for field environments to achieve high-precision segmentation of field rock regions and precise calculation of sampling poses. Addressing the issue of irregular terrain distribution, a multi-sensor fusion mapping method is employed to construct a 3D map with accurate geometric structures and rich texture colors. Based on this map, by integrating the geometric structures of point clouds with visual texture information, a multimodal collaborative constrained unsupervised rock segmentation algorithm is designed.This algorithm leverages physical priors to effectively overcome interference from drastic light changes and resolves the problem of insufficient annotated data caused by dynamic lithological variations. Furthermore, in light of the execution constraints of robotic sampling, an automated sampling-pose generation strategy is proposed. Through the parallel processes of local surface feature analysis of the rock region and kinematic constraint correction of the robot, the optimal sampling point is selected, enabling a precise mapping of the sampling point from the environmental space to the robotic operational space. Field experiments demonstrate that the rock segmentation accuracy reaches 89.57%, a significant improvement over the traditional region-growing method. Furthermore, the sampling-pose estimation achieves a mean position error of 0.696 cm and a normal vector error of 1.44°, which fulfills the precision requirements for autonomous rock sampling in the field.