Masked Sentence Model Based on BERT for Move Recognition in Medical Scientific Abstracts

doi:10.2478/jdis-2019-0020

Abstract

Abstract:

Purpose: Move recognition in scientific abstracts is an NLP task of classifying sentences of the abstracts into different types of language units. To improve the performance of move recognition in scientific abstracts, a novel model of move recognition is proposed that outperforms the BERT-based method.

Design/methodology/approach: Prevalent models based on BERT for sentence classification often classify sentences without considering the context of the sentences. In this paper, inspired by the BERT masked language model (MLM), we propose a novel model called the masked sentence model that integrates the content and contextual information of the sentences in move recognition. Experiments are conducted on the benchmark dataset PubMed 20K RCT in three steps. Then, we compare our model with HSLN-RNN, BERT-based and SciBERT using the same dataset.

Findings: Compared with the BERT-based and SciBERT models, the F1 score of our model outperforms them by 4.96% and 4.34%, respectively, which shows the feasibility and effectiveness of the novel model and the result of our model comes closest to the state-of-the-art results of HSLN-RNN at present.

Research limitations: The sequential features of move labels are not considered, which might be one of the reasons why HSLN-RNN has better performance. Our model is restricted to dealing with biomedical English literature because we use a dataset from PubMed, which is a typical biomedical database, to fine-tune our model.

Practical implications: The proposed model is better and simpler in identifying move structures in scientific abstracts and is worthy of text classification experiments for capturing contextual features of sentences.

Originality/value: The study proposes a masked sentence model based on BERT that considers the contextual features of the sentences in abstracts in a new way. The performance of this classification model is significantly improved by rebuilding the input layer without changing the structure of neural networks.

Key words: Move recognition, BERT, Masked sentence model, Scientific abstracts

Gaihong Yu,Gaihong Yu,Zhixiong Zhang,Zhixiong Zhang,Huan Liu,Huan Liu,Liangping Ding,Liangping Ding. "Masked Sentence Model Based on BERT for Move Recognition in Medical Scientific Abstracts." Journal of Data and Information Science, vol.4, no.4, 2019, pp.42-55. DOI: 10.2478/jdis-2019-0020

References

[1]	Amini I., Martinez D., & Molla D. (2012). Overview of the ALTA 2012 shared task. In Proceedings of the Australasian Language Technology Association Workshop 2012: ALTA 2012(pp. 124-129). Dunedin, New Zealand.
[2]	Badie K., Asadi N., & Tayefeh Mahmoudi M. (2018). Zone identification based on features with high semantic richness and combining results of separate classifiers. Journal of Information and Telecommunication, 2(4), 411-427.
[3]	Basili , R. &Pennacchiotti , M.(2010). Distributional lexical semantics: Toward uniform representation paradigms for advanced acquisition and processing tasks. Natural Language Engineering, 1(1), 1-12.
[4]	Beltagy I., Lo K., & Cohan, A. (2019). SciBERT: Pretrained contextualized embeddings for scientific text. arXiv:1903.10676v3.
[5]	Dasigi P., Burns G.A.P.C., Hovy E., & Waard A. (2017). Experiment segmentation in scientific discourse as clause-level structured prediction using recurrent neural networks. arXiv:1702.05398.
[6]	Devlin J., Chang M.W., Lee K., & Toutanova K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.
[7]	Ding L.P., Zhang Z.X., & Liu H. (2019). Research on factors affecting the SVM model performance on move recognition. Data Analysis and Konwledge Discovery,.
[8]	Firth J.R.(1930). A synopsis of linguistic theory, 1930-1955. In: Firth, J.R., Ed., Studies in Linguistic Analysis, Longmans, London, 168-205.
[9]	Fisas B., Ronzano F., & Saggion H. (2016). A multi-layered annotated corpus of scientific papers. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016).
[10]	Franck Dernoncourt & Ji Young Lee. (2017). Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts. In Proceedings of the 8th International Joint Conference on Natural Language Processing.
[11]	Gerlach M., Peixoto T.P., Altmann E.G., & Altmann E.G. (2018). A network approach to topic models. Science advances, 4(7), eaaq1360.
[12]	Hirohata K., Okazaki N., Ananiadou S., & Mitsuru. (2018). Identifying sections in scientific abstracts using conditional random fields. In Proceedings of the Third International Joint Conference on Natural Language Processing.
[13]	Ma M.B., Huang L., Xiang B., & Zhou B.W. (2015). Dependency-based convolutional neural networks for sentence embedding. arXiv:1507.01839.
[14]	Peters M.E., Neumann M., Iyyer M., et al. (2018). Deep contextualized word representations.In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. doi: 10.18653/v1/N18-1202 arXiv:1802.05365.
[15]	Radford A., Narasimhan K., Salimans T., & Sutskever Ilya (2018). Improving language understanding by generative pre-training.
[16]	Lai S.W., Xu L., Liu K., & Zhao J. (2015). Recurrent convolutional neural networks for text classification. In AAAI’15 Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2267-2273.
[17]	Swales J.M.(2004).Research genres:Explorations and applications. Cambridge: Cambridge University Press.
[18]	Taylor , W.L. (1953). “Cloze procedure”: A new tool for measuring readability. Journalism & Mass Communication Quarterly, 30(4), 415-433. doi: https://doi.org/10.1177/107769905303000401
[19]	Teufel , S. (1999). Argumentative zoning: Information extraction from scientific text. Edinburgh: University of Edinburgh.
[20]	Vaswani A., Shazeer N., Parmar N., et al. (2017). Attention is all you need. arXiv:1706.03762v5.
[21]	Yamamoto , Y. &Takagi, T. (2005). A sentence classification system for multi-document summarization in the biomedical domain. In Proceedings of International Workshop on Biomedical Data Engineering, pages 90-95.
[22]	Yoon Kim. (2014). Convolutional neural networks for sentence classification. arXiv:1408.5882.
[23]	Zhang Z., Liu H., Ding L., et al. (2019). Moves recognition in abstract of research paper based on deep learning. In Proceedings of 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL). IEEE, pages 390-391.