Success of Deep Learning Methods in Determining the End of Sentence with Pos Tag Information in Turkish Texts

Abstract

As a result of today's technological developments, written and spoken texts have increased rapidly in the digital world. However, Natural Language Processing (NLP) applications have gained great importance today. The first and most important issue to be solved in NLP applications is to determine the sentence boundary of the text correctly. Punctuation marks such as periods, exclamation points, and question marks that are generally seen at the end of sentences are not only used to determine the boundary of sentences in the text. Therefore, the disambiguation of the purpose of using punctuation marks is a problem. In previous studies, the effects of POS (Part-Of-Speech) tag information at the end of the sentence were examined and successful results were obtained with classical classifiers. In this study, different numbers of POS tag information were added to 9 rules-based attributes and experimental evaluations were carried out with deep learning methods called Long Short Term Memory(LSTM) and Bidirectional Long Short Term Memory(BiLSTM). For the experiments, the Turkish National Corpus (TNC) and the parallel corpus called SETimes were used. TNC is a 50 million-word corpus of many fields and genres covering the period 1990-2009. SETimes is a parallel corpus of 10 languages, 9 of which belong to southeast Europe and one to English. Balanced sub-datasets with 30000 samples, with and without sentence endings, randomly selected from the specified corpus, were created and these datasets were used for testing. With the experiments performed, classical classifiers such as Back Propagation Neural Network, RBF (Radial Basis Function) Network, Naive Bayes classifier, Decision Tree and Support Vector Machine; and deep learning methods such as LSTM and BiLSTM were compared. As a result, it has been observed that the success of deep learning methods is significantly better than classical classifiers.