Multimodal Fusion Perception for Rock Sampling Robot in Field Environments
-
摘要: 岩石取样机器人是地质勘探智能化的核心装备，但机器人在实际作业中面临着野外非结构化环境、剧烈光照波动以及岩性复杂多变等挑战. 因此，结合自研的野外岩石取样机器人平台，提出了一种多模态融合的野外环境感知方法，实现从野外岩石区域的高精度分割和取样位姿的精准解算. 针对野外地形分布无规律的问题，采用多传感器融合的环境建图方法，构建几何结构准确且纹理色彩丰富的三维地图. 在此地图基础上，融合点云的几何结构与视觉纹理信息，设计了多模态协同约束的无监督岩石区域分割算法，利用物理先验有效克服了野外光照条件变化剧烈的干扰，并解决了岩性动态多变导致的标注数据匮乏的问题. 结合机器人取样作业的执行约束，提出一种取样点位姿自动生成策略，通过岩石区域的局部表面特征分析与机器人运动学约束校正，选取最优岩石取样点，并实现取样点从环境空间到机器人作业空间的精准映射. 在野外实测中，岩石分割精度达89.57%，较传统区域生长法有显著提升；且取样点位姿估计位置误差为0.696 cm，法向量误差为1.44°，满足自动化岩石取样的精度需求.Abstract: Rock-sampling robots are essential equipment for the intelligence of field geological exploration. However, in actual operations, robots face severe challenges such as unstructured field environments, intense illumination fluctuations, and highly variable rock lithology. Therefore, relying on a self-developed field rock-sampling robot platform, this paper proposes a multimodal fusion perception method for field environments to achieve high-precision segmentation of field rock regions and precise calculation of sampling poses. Addressing the issue of irregular terrain distribution, a multi-sensor fusion mapping method is employed to construct a 3D map with accurate geometric structures and rich texture colors. Based on this map, by integrating the geometric structures of point clouds with visual texture information, a multimodal collaborative constrained unsupervised rock segmentation algorithm is designed.This algorithm leverages physical priors to effectively overcome interference from drastic light changes and resolves the problem of insufficient annotated data caused by dynamic lithological variations. Furthermore, in light of the execution constraints of robotic sampling, an automated sampling-pose generation strategy is proposed. Through the parallel processes of local surface feature analysis of the rock region and kinematic constraint correction of the robot, the optimal sampling point is selected, enabling a precise mapping of the sampling point from the environmental space to the robotic operational space. Field experiments demonstrate that the rock segmentation accuracy reaches 89.57%, a significant improvement over the traditional region-growing method. Furthermore, the sampling-pose estimation achieves a mean position error of 0.696 cm and a normal vector error of 1.44°, which fulfills the precision requirements for autonomous rock sampling in the field.
-
图 3 多模态协同约束的岩石区域语义分割算法
Fig. 3. Multimodal cooperative constraint-based semantic segmentation algorithm for rock regions
图 4 野外岩石区域与非岩石区域的特征分布统计
Fig. 4. Statistical distribution of feature characteristics between rock and non-rock areas in the field
表 1 分割精度对比
Table 1. Comparison of segmentation accuracy
序号 内容 分割方法 区域生长法 本文方法 S1 正确分割点数(个) 35 627 86 693 总点数（个） 96 788 96 788 正确率（%） 36.81 89.57 S2 正确分割点数（个） 32 996 79 618 总点数（个） 92 376 92 376 正确率（%） 35.72 86.19 S3 正确分割点数（个） 32 127 88 776 总点数（个） 101 540 101 540 正确率（%） 31.64 87.43 
下载: 导出CSV
表 2 取样点位姿实验误差结果
Table 2. Results of sampling point pose experiment
实验序号 位置误差（m） 姿态误差（°） 1 0.006 9 1.43 2 0.006 4 1.14 3 0.007 1 1.66 4 0.007 0 1.25 5 0.007 4 1.72
下载: 导出CSV
-
Bar, A., Lohdefink, J., Kapoor, N., et al., 2020. The Vulnerability of Semantic Segmentation Networks to Adversarial Attacks in Autonomous Driving: Enhancing Extensive Environment Sensing. IEEE Signal Processing Magazine, 38(1): 42-52. https://doi.org/10.1109/msp.2020.2983666 Cadena, C., Carlone, L., Carrillo, H., et al., 2016. Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age. IEEE Transactions on Robotics, 32(6): 1309-1332. https://doi.org/10.1109/tro.2016.2624754 Campos, C., Elvira, R., Rodriguez, J. J. G., et al., 2021. ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial, and Multimap SLAM. IEEE Transactions on Robotics, 37(6): 1874-1890. https://doi.org/10.1109/tro.2021.3075644 Engel, J., Koltun, V., Cremers, D., 2017. Direct Sparse Odometry, IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3): 611-625. El Madawi, K., Rashed, H., El Sallab, A., et al., 2019. RGB and Lidar Fusion Based 3D Semantic Segmentation for Autonomous Driving. IEEE Intelligent Transportation Systems Conference(ITSC), Piscataway, N. J., Institute of Electrical and Electronics Engineers (IEEE), USA, 7-12. Forster, C., Zhang, Z. C., Gassner, M., et al., 2016. SVO: Semidirect Visual Odometry for Monocular and Multicamera Systems. IEEE Transactions on Robotics, 33(2): 249-265. https://doi.org/10.1109/tro.2016.2623335 Gorai, A. K., Raval, S., Patel, A. K., et al., 2021. Design and Development of a Machine Vision System Using Artificial Neural Network-Based Algorithm for Automated Coal Characterization. International Journal of Coal Science & Technology, 8(4): 737-755. https://doi.org/10.1007/s40789-020-00370-9 Guo, Y. F., Xu, P., Li W., et al., 2025. 3D Perception and Reconstruction of Underground Coal Mine Roadways Based on Multi-Sensor Fusion. Earth Science. doi: 10.3799/dqkx.2025.274 (in Chinese with English abstract). Hu, Q. Y., Yang, B., Xie, L. H., et al., 2021. Learning Semantic Segmentation of Large-Scale Point Clouds with Random Sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25: 1-1. https://doi.org/10.1109/tpami.2021.3083288 Lin, J. R., Zheng, C. R., Xu, W., et al., 2021. R $^2$ LIVE: A Robust, Real-Time, LiDAR-Inertial-Visual Tightly-Coupled State Estimator and Mapping. IEEE Robotics and Automation Letters, 6(4): 7469-7476. https://doi.org/10.1109/lra.2021.3095515 Li, K. L., Li, M., Hanebeck, U. D., 2021. Towards High-Performance Solid-State-LiDAR-Inertial Odometry and Mapping. IEEE Robotics and Automation Letters, 6(3): 5167-5174. https://doi.org/10.1109/lra.2021.3070251 Liu, X., Gong, G., Hu, X., et al., 2024. Cognitive Enhancement of Robot Path Planning and Environmental Perception Based on Gmapping Algorithm Optimization, Electronics, 13(5): 818(in Chinese with English abstract). doi: 10.3390/electronics13050818 Lin, F. F., Lin, T. L., Yao, Y., et al., 2025. VPA-Net: A Visual Perception Assistance Network for 3d Lidar Semantic Segmentation. Pattern Recognition, 158: 111014. https://doi.org/10.1016/j.patcog.2024.111014 Liu, H. W., Xu, G. Q., Liu, B., et al., 2025. A Real Time LiDAR-Visual-Inertial Object Level Semantic SLAM for Forest Environments. ISPRS Journal of Photogrammetry and Remote Sensing, 219: 71-90. https://doi.org/10.1016/j.isprsjprs.2024.11.013 Mur-Artal, R., Montiel, J. M. M., Tardos, J. D., 2015. ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics, 31(5): 1147-1163. https://doi.org/10.1109/tro.2015.2463671 Pan, B. W., Sun, J. K., Leung, H. Y. T., et al., 2020. Cross-View Semantic Segmentation for Sensing Surroundings. IEEE Robotics and Automation Letters, 5(3): 4867-4873. https://doi.org/10.1109/lra.2020.3004325 Pang, Y. E., Li, X., Dong, Z. K., et al., 2024. Convolutional Neural Network-Based Model for Recognizing TBM Rock Chip Gradation. Automation in Construction, 163(1): 105414. https://doi.org/10.1016/j.autcon.2024.105414 Qin, T., Li, P. L., Shen, S. J., 2018. VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator. IEEE Transactions on Robotics, 34(4): 1004-1020. https://doi.org/10.1109/tro.2018.2853729 Sun, K., Mohta, K., Pfrommer, B., et al., 2018. Robust Stereo Visual Inertial Odometry for Fast Autonomous Flight. IEEE Robotics and Automation Letters, 3(2): 965-972. https://doi.org/10.1109/lra.2018.2793349 Shin, Y. S., Park, Y. S., Kim, A., 2020. DVL-SLAM: Sparse Depth Enhanced Direct Visual-LiDAR SLAM. Autonomous Robots, 44(2): 115-130. https://doi.org/10.1007/s10514-019-09881-0 Wang, Y. Z., Sun, S. Y., 2021. Image-Based Rock Typing Using Grain Geometry Features. Computers & Geosciences, 149: 104703. https://doi.org/10.1016/j.cageo. 2021.104703 doi: 10.1016/j.cageo.2021.104703 Wang, Y., Zhang, J., Chen, Y. H., et al., 2024. An Automated Learning Method of Semantic Segmentation for Train Autonomous Driving Environment Understanding. IEEE Transactions on Industrial Informatics, 20(4): 6913-6922. https://doi.org/10.1109/tii.2024.3353874 Xie, T., Sun, Q. H., Sun, T., et al., 2025. DVDS: A Deep Visual Dynamic Slam System. Expert Systems with Applications, 260(10): 125438. https://doi.org/10.1016/j.eswa.2024.125438 Xu, X. B., Zhang, L., Yang J., et al., 2022. A Review of Multi-Sensor Fusion SLAM Systems Based on 3D LIDAR. Remote Sensing, 14(12): 2835. https://doi.org/10.3390/rs14122835 Xu, K., Hao, Y. F., Yuan, S. H., et al., 2025. AirSLAM: An Efficient and Illumination-Robust Point-Line Visual SLAM System. IEEE Transactions on Robotics, 41(5): 1673-1692. https://doi.org/10.1109/tro.2025.3539171 Yan, X., Gao, J., Zheng, C., et al., 2022. 2DPASS: 2D Priors Assisted Semantic Segmentation on Lidar Point Clouds. European Conference on Computer Vision (ECCV), Springer, Berlin, Germany, 677-695. Yang, J. T., Kang, Z. Z., Yang, Z., et al., 2022. A Laboratory Open-Set Martian Rock Classification Method Based on Spectral Signatures. IEEE Transactions on Geoscience and Remote Sensing, 60: 1-15. https://doi.org/10.1109/tgrs.2022.3175996 Yao, X., Shi, J., Chen, J., et al., 2024. From Direct Exploration to AI Exploration: Evolution and Insight, Development and Research Centre, 15(4): 671-679 (in Chinese with English abstract). Zhang, J., Kaess, M., Singh, S., 2017. A Real-Time Method for Depth Enhanced Visual Odometry. Autonomous Robots, 41(1): 31-43. https://doi.org/10.1007/s10514-015-9525-1 Zhang, J., Singh, S., 2018. Laser-Visual-Inertial Odometry and Mapping with High Robustness and Low Drift. Journal of Field Robotics, 35(8): 1242-1264. https://doi.org/10.1002/rob.21809 Zhao, L., Zhou, H., Zhu, X. G., et al., 2023. LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation. IEEE Transactions on Multimedia, 26: 1158-1168. https://doi.org/10.1109/tmm.2023.3277281 Zhu, H., Pan, B., Wu, W., et al., 2023. A Review of Rock Mass Discontinuity Information Collection and Identification Methods, Journal of Basic Science and Engineering, 31(6): 1339-1360 (in Chinese with English abstract). Zhang, S., Shan, J. J., Liu, Y. B., 2025. Approximate Inference Particle Filtering for Mobile Robot SLAM. IEEE Transactions on Automation Science and Engineering, 22: 7967-7978. https://doi.org/10.1109/tase.2024.3475735 郭云飞, 徐鹏, 李伟, 等, 2026. 基于多传感器融合的煤矿井下巷道三维感知与重建方法. 地球科学. doi: 10.3799/dqkx.2025.274 姚晓峰, 施俊法, 陈骥, 等. 2024. 从"直接找矿"到"智能找矿": 脉络与启示, 矿产勘查, 15(4): 671-679. 朱合华, 潘柄屹, 武威, 等. 2023. 岩体结构面信息采集及识别方法研究进展, 应用基础与工程科学学报, 31(6): 1339-1360. -
点击查看大图
图(7) / 表(2)
计量
- 文章访问数: 399
- HTML全文浏览量: 49
- PDF下载量: 66
- 被引次数: 0
