Geotechnical Named Entity Recognition Based on BERT-BiGRU-CRF Model
-
摘要: 岩土工程实体识别是岩土工程文本挖掘和知识谱图的工作基础和重要前提. 针对岩土工程实体识别问题，参考《GB/T 50279-2014：岩土工程基本术语标准》等国家行业标准规范，设计和构建了一个小规模的岩土工程命名实体语料库；提出了一种岩土工程文本命名实体识别深度学习模型BERT-BiGRU-CRF（简称：GENER）：表示学习层采用BERT预训练语言模型实现了岩土工程文本特征的迁移表示学习；BiGRU上下文编码层实现对岩土工程文本上下文特征编码；CRF标签解码层解决了标签间依赖约束，生成符合标注规律的岩土工程命名实体标签序列；最后，基于岩土工程命名实体语料库，对GENER模型进行了实验分析. 在对照实验中，取得了良好效果：精确率P达到了90.94%，召回率R达到了92.88%，F1值达到了91.89%，模型训练速度提升了4.735%. 实验结果表明相比基线模型BiLSTM-CRF和其他预训练模型，GENER模型在小规模语料岩土工程命名实体识别方面效果更优，未来可推广应用到其他地质类文本命名实体识别任务.Abstract: Geotechnical engineering named entity recognition is an important prerequisite and the work foundation for geotechnical information mining and knowledge Graph. Aiming at the recognition and classification of named entities in geotechnical texts, this article first designs and constructs a named entity corpus of geotechnical engineering according to Standard for Fundamental Terms of Geotechnical Engineering (GB/T 50279-2014) and other national industry standards; and based on deep learning technologies, a named entity recognition and classification deep learning model GENER is proposed for geotechnical engineering text. In GENER, the distributed representation learning of geotechnical engineering text features is realized based on the BERT pretrained language model; the geotechnical engineering text context feature encoding is achieved based on the BiGRU context coding layer; and based on the label decoding layer of CRF, the context features are decoded to generate the label sequence of geotechnical engineering named entity. Finally, based on the geotechnical engineering corpus, the GENER model is experimentally analyzed. comparing with other deep learning models for named entity recognition based on pretrained language models, the GENER model has better performance. The precision reaches 90.94%, the recall reaches 92.88%, the F1-score reaches 91.89%and model training speed increased by 4.735% respectively.Experiments show that compared with BiLSTM-CRF and CNN-BiLSTM-CRF models, this model is more effective in small-scale corpus geotechnical engineering entity recognition.
-
Key words:
- named entity recognition /
- deep learning /
- geotechnical engineering /
- corpus /
- geological bigdata
-
图 1 岩土工程命名实体语料库构建过程
Fig. 1. Constructing process for the geotechnical named entity corpus
图 5 表示学习层预训练模型结构
Fig. 5. Structure of the pretrained language model in the representation learning layer
表 1 岩土工程命名实体类型
Table 1. Types of geotechnical named entities
实体类型 简称 说明 岩土工程名称 GEN 切坡、边坡、高切坡等岩土工程名称 岩土工程地质 GEO 岩土工程物质组成和地质结构等 工程勘测方法 SUR 工程测量、工程地质测绘、勘探、试验等岩土工程勘察方法 工程属性参数 FEA 形态特征、地质结构、岩土物理力学参数等属性参数 工程评测方法 EVA 变形破坏模式、预测变形破坏模式等 工程防护方法 DEF 岩土工程防护方案和措施 
下载: 导出CSV
表 2 岩土工程命名实体标注实例
Table 2. Examples for geotechnical named entity labels
实体示例 标注示例 （1）秭归县泄滩乡石门高切坡位于长江北岸坊家山村一组石门，距秭归新县城约70 km （1）秭/B_GEN归/M_GEN县/M_GEN泄/M_GEN滩/M_GEN乡/M_GEN石/M_GEN门/M_GEN高/M_GEN切/M_GEN坡/E_GEN位/O于/O长/O江/O北/O岸/O坊/O家/O山/O村/O-/O组/O石/O门/O，/O距/O秭/O归/O新/O县/O城/O约/O7/O0/OK/Om/O. /O （2）高切坡的物质组成主要为前震旦系（γ1）闪云斜长花岗岩，局部鲜见伟晶岩脉发育 （2）高/O切/O坡/O的/O物/O质/O组/O成/O主/O要/O为/O闪/B_GEO云/M_GEO斜/M_GEO长/M_GEO花/M_GEO岗/M_GEO岩/E_GEO，/O局/O部/O鲜/O见/O伟/B_GEO晶/M_GEO岩/M_GEO脉/E_GEO发/O育/O. /O （3）该段高切坡坡顶高程201~214 m，坡脚高程181.1 m，坡度50°~57°，坡高20~33 m （3）该/O段/O高/O切/O坡/O坡/B_FEA顶/M_FEA高/M_FEA程/E_FEA2/O0/O1/Om/O~/O2/O1/O4/Om/O，/O坡/B_FEA脚/M_FEA高/M_FEA程/E_FEA1/O8/O1/O./O1/Om/O，/O坡/B_FEA度/E_FEA5/O0/O°/O~/O5/O7/O°/O，/O坡/B_FEA高/E_FEA2/O0/O~/O3/O3/Om/O. /O （4）制定了以工程地质测绘为主，辅以少量山地和钻探工作的勘察方案 （4）制/O定/O了/O以/O工/B_SUR程/M_SUR地/M_SUR质/M_SUR测/M_SUR绘/E_SUR为/O主/O，/O辅/O以/O少/O量/O山/O地/O和/O钻/B_SUR探/E_SUR工/O作/O的/O勘/O察/O方/O案/O. /O （5）切坡的变形破坏主要表现为坡面在雨水冲蚀作用下形成的水砂流冲蚀破坏. 该段切坡的变形破坏模式为局部破坏 （5）切/O坡/O的/O变/O形/O破/O坏/O主/O要/O表/O现/O为/O坡/O面/O在/O雨/O水/O冲/O蚀/O作/O用/O下/O形/O成/O的/O水/B_EVAL砂/M_EVAL流/M_EVAL冲/M_EVAL蚀/M_EVAL破/M_EVAL坏/E_EVAL, /O该/O段/O切/O坡/O的/O变/O形/O破/O坏/O模/O式/O为/O局/B_EVAL部/M_EVAL破/M_EVAL坏/E_EVAL. /O （6）坡顶后缘设计地表截水沟，坡底设计排水沟，坡面设深排水孔等，以减少地表水入渗 （6）坡/O顶/O后/O缘/O设/O计/O地/B_DEF表/M_DEF截/M_DEF水/M_DEF沟/E_DEF，/O坡/O底/O设/O计/O排/B_DEF水/M_DEF沟/E-DEF，/O坡/O面/O设/O深/B_DEF排/B_DEF水/M_DEF孔/E-DEF等/O，/O以/O减/O少/O地/O表/O水/O入/O渗/O. /O
下载: 导出CSV
表 3 实验设置信息
Table 3. Experimental settings
环境 工具 详情 硬件 CPU Intel(R) Xeon(R) Gold 6248 GPU Tesla V100S-PCIE-32GB 内存 512GB 软件 操作系统 Centos7 开发语言 Python 3.7.2 深度学习框架 Pytorch 1.6.0
下载: 导出CSV
表 4 GENER模型超参数配置
Table 4. Hyperparameter configurations of the GENER model
参数名称 参数值 Max_Sentence_Length 128 预训练层 子嵌入向量维度 768 编码器堆叠数 内存 自注意力头数 12 上下文编码层 是否双向（Directional） True 网络层数 2 上下文特征编码维度 384 偏置（Bias） True Batch_First True Dropout 0.1 Num_workers 8
下载: 导出CSV
表 5 数据集命名实体统计分布
Table 5. Statistical distribution of named entities in the datasets
训练集 验证集 测试集 岩土工程名称（GEN） 3 312 419 472 岩土工程地质（GEO） 10 304 1 294 1 431 工程属性参数（FEA） 12 202 1 465 1 519 工程勘测方法（SUR） 2 710 333 396 工程评测方法（EVA） 4 344 536 591 工程防护方法（DEF） 4 078 522 540 总计 36 950 4 569 4 949
下载: 导出CSV
表 6 学习率与精确率P、召回率R、F1值关系表
Table 6. Relationships between learning rate and precision, recall, and F1 score
模型/学习率 le−1 5e−1 le−2 5e-2 le−3 5e−3 le-4 5e−4 le−5 5e−5 le−6 5e−6 P 74.75 29.04 80.69 76.99 85.82 84.57 83.41 85.31 67.70 91.46 5.72 62.61 BiLSTM-CRF R 74.75 29.04 80.54 76.55 85.82 84.44 83.41 84.98 67.70 89.90 5.67 62.61 F1 74.75 29.04 80.61 76.77 85.82 84.51 83.41 85.14 67.70 90.66 5.70 62.61 P 71.29 69.22 79.81 71.27 84.70 84.50 84.56 84.37 74.88 90.79 21.08 61.49 ERNIE-BiLSTM-CRF R 70.96 68.93 79.81 70.42 84.64 82.53 84.56 84.37 73.33 92.53 20.19 61.49 F1 71.12 69.07 79.81 70.84 84.67 83.50 84.56 84.37 74.10 91.64 20.63 61.49 P 71.72 14.93 88.51 71.72 91.11 88.93 86.05 88.51 80.75 90.12 71.86 71.02 BERT-BiLSTM-CRF R 71.72 0.38 88.51 71.72 91.11 88.93 86.05 88.51 80.38 91.81 69.66 70.88 F1 71.72 0.75 88.51 71.72 91.11 88.93 86.05 88.51 80.57 90.94 70.74 70.95 P 28.57 11.80 11.34 71.72 79.20 11.80 84.90 84.67 72.34 85.86 70.79 71.48 RoBERTa-BiLSTM-CRF R 84.29 11.80 11.34 71.72 79.20 11.80 84.90 84.67 72.34 91.32 70.46 70.96 F1 1.64 11.80 11.34 71.72 79.20 11.80 84.90 84.67 72.34 88.47 70.62 71.22 P 71.72 63.66 76.21 68.66 88.03 84.83 87.14 87.49 83.50 88.17 77.55 79.39 XLNET-BiLSTM-CRF R 71.72 83.44 75.86 54.98 87.93 84.83 86.97 87.32 81.46 91.47 36.93 72.91 F1 71.72 71.28 76.04 61.06 87.98 84.83 87.06 87.40 82.47 89.77 50.04 76.01 P 29.62 30.69 90.08 75.54 90.84 89.77 89.96 90.34 83.52 90.94 70.98 75.58 GENER R 29.62 30.69 90.08 71.34 90.84 89.77 89.96 90.34 83.52 92.88 70.00 75.52 F1 29.62 30.69 90.08 73.38 90.84 89.77 89.96 90.34 83.52 91.89 70.49 75.55
下载: 导出CSV
表 7 消融实验结果对比
Table 7. Results of the ablation experiment
Model P R F1 s/epoch BiGRU-CRF 91.66 91.29 91.46 1 118 BERT-CRF 90.88 90.69 90.78 9 564 GENER 91.25 93.01 92.11 11 321
下载: 导出CSV
表 8 最优参数设置
Table 8. Optimal parameter settings
模型名称 epoch 批大小 学习率 BiLSTM-CRF 100 10 5e-5 ERNIE-BiLSTM-CRF 100 10 5e-5 BERT-BiLSTM-CRF 100 20 5e-5 RoBERTa-BiLSTM-CRF 100 20 5e-5 XLNET-BiLSTM-CRF 100 10 5e-5 GENER 100 20 5e-5
下载: 导出CSV
表 9 最优参数下不同模型性能数据
Table 9. Performance of different models under optimal parameter settings
Model P R F1 s/epoch BiLSTM-CRF 91.46 89.90 90.66 1 420 ERNIE-BiLSTM-CRF 90.79 92.53 91.64 11 300 BERT-BiLSTM-CRF 89.97 92.31 91.10 11 857 RoBERTa-BiLSTM-CRF 87.57 91.79 89.61 11 806 XLNET-BiLSTM-CRF 88.17 91.47 89.77 36 562 GENER 91.25 93.01 92.11 11 321
下载: 导出CSV
表 10 GENER模型各命名实体分类精度
Table 10. The classification accuracy of each named entity in GENER model
P R F1 Support 岩土工程名称（GEN） 91.61 98.05 94.72 472 岩土工程地质（GEO） 93.89 95.05 94.47 1 431 工程属性参数（FEA） 93.36 93.16 93.26 1 519 工程勘测方法（SUR） 89.39 88.59 88.99 396 工程评测方法（EVA） 80.54 85.26 82.83 591 工程防护方法（DEF） 91.08 94.12 92.58 540 总计 91.25 93.01 92.11 4 949
下载: 导出CSV
-
[1] Bengio, Y., Courville, A., Vincent, P., 2013. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis & Machine Intelligence, 35(8): 1798-1828. https://doi.org/10.1109/TPAMI.2013.50 [2] Cho, K., Van, M., Gulcehre, C., et al., 2014. Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. ArXiv Preprint ArXiv: 1406.1078. [3] Chu, D. P., Wan, P., Li, H., et al., 2021. Geological Entity Recognition Based on ELMO-CNN-BiLSTM-CRF Model. Earth Science. 46(8): 3039-3048(in Chinese with English abstract). [4] Chung, J., Gulcehre, C., Cho, K. H., et al., 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv preprint arXiv: 1412.3555. [5] Devlin, J., Chang, M., Lee, K., et al., 2018. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. ArXiv Preprint ArXiv: 1810.04805. [6] Dong, L., Yang, N., Wang, W., et al., 2019. Unified Language Model Pre-Training for Natural Language Understanding and Generation. ArXiv Preprint ArXiv: 1905.03197. [7] Fan, R., Wang, L., Yan, J., et al., 2020. Deep Learning-Based Named Entity Recognition and Knowledge Graph Construction for Geological Hazards. ISPRS International Journal of Geo-Information, 9(1): 15. https://doi.org/10.3390/ijgi9010015 [8] Goyal, A., Gupta, V., Kumar, M., 2018. Recent Named Entity Recognition and Classification Techniques: A Systematic Review. Computer Science Review, 29: 21-43. https://doi.org/10.1016/j.cosrev.2018.06.001 [9] He, Y.X., Luo, C.W., Hu, B.Y., 2015. Geographic Entity Recognition Method Based on CRF Model And Rules Combination. Computer Application and Software. 2015, 32(1): 179(in Chinese with English abstract). [10] Lafferty, J., Mccallum, A., Pereira, F., 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the Eighteenth International Conference on Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco, 282–289. [11] Lample, G., Ballesteros, M., Subramanian, S., et al., 2016. Neural Architectures for Named Entity Recognition. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. The Association for Computational Linguistics, San Diego. https://doi.org/10.18653/v1/n16-1030 [12] Li, J., Sun, A.X., Han, J.L., et al., 2022. ASurvey on Deep Learning for Named Entity Recognition. IEEE Transactions on Knowledge and Data Engineering, 34(1): 50-70. https://doi.org/10.1109/TKDE.2020.2981314 [13] Liu, D. S., Liu, H. L., Wu, Y., et al, 2022. Genetic Features of Geo-Materials and Their Testing Metohd. Journal of Civil and Environmental Engineering, 44(04): 1-9(in Chinese with English abstract). [14] Liu, H. L., Zhang, R. H., Liu, D. S., et al., 2021. Study on the Characteristics of Physical and Mechanical Parameters of Engineering Geology Based on Data Fusion. Journal of Civil and Environmental Engineering, 1-11(in Chinese with English abstract). [15] Liu, X., Zhang, S., Wei, F., et al., 2011 Recognizing Named Entities in Tweets. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, USA, 359-367. [16] Liu, Y., Ott, M., Goyal, N., et al., 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv Preprint ArXiv: 1907.11692. [17] Marrero, M., Urbano, J., Sánchez-Cuadrado, S., et al., 2013. Named Entity Recognition: Fallacies, Challenges and Opportunities. Computer Standards & Interfaces, 35(5): 482-489. https://doi.org/10.1016/j.csi.2012.09.004 [18] Ministry of Housing and Urban Rural Development of The People's Republic of China, 2013. GB/T 50330-2013: Construction Side Slope Engineering technology Stand. Beijing: China Architecture & Building Pres(in Chinese). [19] Ministry of Housing and Urban Rural Development of The People's Republic of China, 2015. JTGT 84-2015: Terminology Standard for geotechnical investigation. Beijing: China Architecture & Building Pres(in Chinese). [20] Ministry of Water Resources of the People's Republic of China, 2014. GB/T 50279-2014: Basic Nomenclature Standard of Geotechnical Engineer. China Planning Press, Beijing(in Chinese). [21] Nadeau, D., Sekine, S., 2007. A Survey of Named Entity Recognition and Classification. Lingvisticae Investigationes, 30(1): 3-26. https://doi.org/10.1075/li.30.1.03nad [22] Qiu, Q., Xie, Z., Wu, L., et al., 2019. BiLSTM-CRF for Geological Named Entity Recognition from The Geoscience Literature. Earth Science Informatics, 12(4): 565-579. https://doi.org/10.1007/s12145-019-00390-3 [23] Qiu, Q., Xie, Z., Wu, L., et al., 2019. GNER: A Generative Model for Geological Named Entity Recognition Without Labeled Data Using Deep Learning. Earth and Space Science, 6(6): 931-946. https://doi.org/10.1029/2019EA000610 [24] Qiu, X., Sun, T., Xu, Y., et al., 2020. Pre-Trained Models for Natural Language Processing: A Survey. Science China Technological Sciences, 63(10): 1872-1897. https://doi.org/10.1007/s11431-020-1647-3 [25] Quimbaya, A. P., Múnera, A. S, , Rivera, R. A. G., et al., 2016. Named Entity Recognition over Electronic Health Records through a Combined Dictionary-Based Approach. Procedia Computer Science, 100: 55-61. https://doi.org/10.1016/j.procs.2016.09.123 [26] Ritter, A., Clark, S., Etzioni, O., 2011. Named Entity Recognition in Tweets: an Experimental Study. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, USA. [27] Rocktäschel, T., Weidlich, M., Leser, U., 2012. ChemSpot: a Hybrid System for Chemical Named Entity Recognition. Bioinformatics, 28(12): 1633-1640. https://doi.org/10.1093/bioinformatics/bts183 [28] Sharnagat, R., 2014. Named Entity Recognition: A Literature Survey. Center For Indian Language Technology, 8-20. [29] Wang, C., Ma, X., Chen, J., et al., 2018. Information Extraction and Knowledge Graph Construction from Geoscience Literature. Computers & Geosciences, 112: 112-120. https://doi.org/10.1016/j.cageo.2017.12.007 [30] Yang, J., Zhang, Y., Li, L., et al., 2018. YEDDA: A Lightweight Collaborative Text Span Annotation Tool. Proceedings of ACL 2018, System Demonstrations. Association for Computational Linguistics, Australia. https://doi.org/10.18653/v1/P18-4006 [31] Yang, Z., Dai, Z., Yang, Y., et al., 2019. Xlnet: Generalized Autoregressive Pretraining for Language Understanding. Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., NewYork. [32] Zhang, G. Y., Fu, J. Y., Ouyang, Z. Z., et al., 2020. The Importance of Space Database Establishment Based on DGSS in Big Data Environment. Earth Science. 45(9): 3451-3460(in Chinese with English abstract). [33] Zhang, S., Elhadad, N., 2013. Unsupervised Biomedical Named Entity Recognition: Experiments with Clinical and Biological Texts. Journal of biomedical informatics, 46(6): 1088-1098. https://doi.org/10.1016/j.jbi.2013.08.004 [34] Zhang, S. D., Elhadad, N., 2013. Unsupervised Biomedical Named Entity Recognition: Experiments with Clinical and Biological Texts. Journal of Biomedical Informatics, 46(6): 1088-1098. https://doi.org/10.1016/j.jbi.2013.08.004 [35] Zhang, X. Y., Ye, P., Wang, S., et al., 2018. Geological Entity Recognition Method Based on Deep Belief Networks. Acta Petrologica Sinica. 34(2): 343-351(in Chinese with English abstract). [36] Zhang, X.Y., Zhu, S. N., Zhang, C. J., 2012. Annotation of Geographical Named Entities in Chinese Text. Acta Geodaetica et Cartographica Sinica, 41(1): 115-120. (in Chinese with English abstract). [37] Zhang, Z., Han, X., Liu, Z., et al., 2019. ERNIE: Enhanced Language Representation with Informative Entities. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence. https://doi.org/10.18653/v1/P19-1139 [38] 储德平, 万波, 李红, 等, 2021. 基于ELMO-CNN-BiLSTM-CRF模型的地质实体识别. 地球科学, 46(8): 3039-3048. doi: 10.3799/dqkx.2020.309 [39] 何炎祥, 罗楚威, 胡彬尧, 2015. 基于CRF和规则相结合的地理命名实体识别方法. 计算机应用与软件, 32(1): 179. https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ201501047.htm [40] 张雪英, 叶鹏, 王曙, 等, 2018. 基于深度信念网络的地质实体识别方法. 岩石学报, 34(2): 343-351. https://www.cnki.com.cn/Article/CJFDTOTAL-YSXB201802011.htm [41] 刘汉龙, 章润红, 刘东升, 等, 2021. 基于数据融合的工程地质物理力学参数特征研究. 土木与环境工程学报(中英文), 1-11. [42] 刘东升, 刘汉龙, 吴越, 等, 2022. 岩土材料的基因特征及其测试方法研究. 土木与环境工程学报(中英文), 44(04): 1-9. https://www.cnki.com.cn/Article/CJFDTOTAL-JIAN202204001.htm [43] 张雪英, 朱少楠, 张春菊, 2012. 中文文本的地理命名实体标注. 测绘学报, 41(1): 115-120. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201201023.htm [44] 张广宇, 付俊彧, 欧阳兆灼, 等, 2020. 大数据时代基于dgss系统下空间数据库建立的重要性. 地球科学, 45(9): 3451-3460. doi: 10.3799/dqkx.2020.130 [45] 中华人民共和国水利部, 2014. GB/T 50279-2014: 岩土工程基本术语标准. 北京: 中国计划出版社. [46] 中华人民共和国住房和城乡建设部, 2013, GB/T 50330-2013: 建筑边坡工程技术规范. 北京: 中国建筑工业出版社. [47] 中华人民共和国住房和城乡建设部, 2015. JTGT 84-2015: 岩土工程勘察术语标准. 北京: 中国建筑工业出版社. -
点击查看大图
图(14) / 表(10)
计量
- 文章访问数: 42
- HTML全文浏览量: 21
- PDF下载量: 14
- 被引次数: 0
