A Comparison of Performance Metrics of Turkish Twitter Messages Using Text Representations

Transkript

1 A Comparison of Performance Metrics of Turkish Twitter Messages Using Text Representations Abdullah Ammar 1*, Tolga AYDIN Ataturk University, Department of Computer Engineering, Erzurum, Turkey * Corresponding Author: ammar.karcioglu@atauni.edu.tr Abstract With the development of technology and the spread of the internet all over the world, social media platforms have evolved over time so that people can be aware of the changes happening in the world at any moment, and that everyone can share their own thoughts. Twitter, one of the most used social media platforms around the world, has become one of the most important parts of everyday life. With twitter, users share their own feelings and thoughts to create important data sources that can be used in sentiment analysis work on the social media in the field of data mining. In this study, which is implemented in python programming language, sentiment analysis was performed by using text representations in turkish twitter messages that users shared. The aim of the study, the performance effects of Bag-of-Words(BOW) model weighted by Tf-Idf and semantic relation based Word2Vec model are compared on sentiment analysis. In this study, which applied 3 different models, in the third model, the highest accuracy percentage was obtained with 66.40% by applying Random Forest algorithm to Word2Vec model. The results obtained using the machine learning algorithms from the scikit-learn library compared the performance metrics and provided the literature contribution to turkish natural language processing studies. Keywords: Twitter, Sentiment Analysis, Text Representation, Word2Vec 433

2 1. Introduction According to twitter statistics (Aslam, 2018), with 100 million users per day and more than 500 million tweets per month, users share their feelings and thoughts with the whole world quickly and reliably. Twitter; has become one of the most important social media platforms day by day because of the number of daily users and the amount of sharing made. Thanks to its large data set, it has become an important source for researchers on sentiment analysis studies(gemci ve Peker, 2013). Unlike other text with long content, twitter messages are limited to 140 characters, so twitter is called a microblogging service (Kim et al., 2010). Over time, becoming an important social media platform of twitter, it has caused researchers to change their workspaces, thanks to its large data set. In the past, the researchers focused on the structure of twitter and performed some studies. Then they performed studies on extracting semantic information from the twitter data and contributed to the study of sentiment analysis in social media(kim et al., 2010). In the literature on sentiment analysis; a corpus was created from the comments of users with different interests and sentiment analysis was performed on this data set. Purpose of this; contribute to the field of opinion mining and develop a model on sentiment classification with a formed corpus. They performed a linguistic analysis study mainly using the n-gram method (Pak et al., 2010). In another study of twitter sentiment analysis, it has been shown that POS (Part of Speech) features may not be useful for sentiment analysis in the field of microblogging(kouloumpis et al., 2011). It has been shown that using hashtags to collect training data in the study produces successful results in positive and negative data. In addition, it has been described that the method of producing better results using better training data may depend on the characteristics of the features. As a result, it has been shown that when the features of microblog services are added, the success rate may decrease(kouloumpis et al., 2011). In some studies on semantic analysis on twitter, it has been shown how twitter messages are concentrated under which topic names by using semantic relationship from twitter contents by using PLSA and LDA methods(kim et al., 2010). They also presented a vector space model that learns text representations that find meaningful information in order to derive a semantic relationship in sentiment analysis using LSA and LDA methods(maas et al., 2011). The model's probabilistic basis gives a theoretically verified technique for the induction of the word vector 434

3 as an alternative to the overwhelming number of commonly used matrix factorization-based techniques. Topic modeling methods such as LSA, PLSA and LDA have been developed in the literature(alghamdi, 2015). However, Word2Vec (Mikolov et al., 2013a and 2013b), one of the semantic relation-based methods developed by Mikolov and his colleagues, aimed to extract meaningful information on texts. Because of the lack of natural language processing studies in the literature on Turkish language and also because there is not a lot of studies based on semantic relation in Turkish, it is aimed to perform sentiment analysis study by applying text representation methods to positive and negative labeled Turkish twitter messages. In this study, it has been shown that by using the text representation methods, good results are produced in the classification of sentiment analysis in labeled turkish twitter messages. Moreover, it has been shown that these text representations and classification algorithms will produce successful results in sentiment analysis studies. For each feature extraction method using Tf-Idf and Word2Vec (unlike LDA and PLSA methods), performans metrics were compared using Linear SVM, Logistic Regression, KNN, Decision Tree and Random Forest algorithms. The rest of this article is organized as follows. Chapter 2 describes our materials and methods. In this section, data set, preprocessing steps on data, root retrieval, text representation methods and classifiers are shown. Chapter 3 shows our findings as a result of study. Section 4 describes our study results and the study we plan to do in the future. 2. Material and Method In our study, three different models were developed using different text representation methods. All basic classifiers were trained in the same data set, but different text representation methods were applied. First model as given in the section , each machine learning algorithm was applied by feature extracting by Tf-Idf text representation method and the results are obtained. Second model as given in the section , Word2Vec text representation method was applied. However, twitter messages were represented by the average of word embeddings. In the third model; The Word2Vec method was used. However, in this model, messages were represented by the average of the weighted word embeddings. 435

4 Figure 1. The Model of System 2.1. Text Representations Text representations are very important for lexical features used in twitter sentiment analysis studies. Recent studies have shown that intensive, low-dimensional and real-valued word embedding provide competitive performance for twitter sentiment classification(ren et al., 2016) Bag of Words(BOW) The Tf-Idf (Term Frequency - Inverse Document Frequency) model is one of the popular approaches to text representation in natural language processing. Tf-Idf creates a vector space model of the text by weighting the text to show how important the word in the text is in the document (Salton, 1988). Pang and his colleagues pioneered in the field of bag-of-word(bow) where each word was represented as a vector (Ren et al., ). The BOW model has some problems, such as limited classification performance due to the high size of the text representation and the 436

5 inability to catch the semantic relation between the words. In this study, The representation adopted by this work is the bag-of-words weighted by Tf-Idf Word2Vec Word2Vec, which is one of the most commonly used models for generating word vectors, is implemented with two models described as "skip-gram (SG)" and "continuous bagof-words (CBOW)" (Nakov et al., 2016). The Word2Vec model, which succeeded in finding semantic relations between words, was developed by Mikolov and his colleagues (Mikolov et al., 2013a and 2013b). This model, which is basically a shallow artificial neural network, has an input layer, a projection layer and an output layer that allows to find words that are close to each other. In this study, default parameter values of Word2Vec model were used. A 300- dimensional "wiki-tr" corpus (Grave, 2017) with words developed by Bogazici University was used for the training of the Turkish data set. In addition, since there was no previously trained vector in the study, a 300-dimensional vector was initially chosen. Figure 2. Models of CBOW and Skip-Gram (Rong, 2014) 437

6 2.2. Data Set As shown in Figure 3, the training of our Turkish twitter data with positive and negative label and emoji based, which is 80% training data and 20% test data, was performed in itself and the results were compared. Figure 3. Distribution of Labeled Turkish Twitter Messages 2.3. Data Preprocessing Before feature extracting of our data set, we processed the data set in one of the natural language processing steps, the data preprocessing step. We removed the hashtags that begin with the "#" character, the username that begins with the character, all the digits, punctuations and image marks, and URL addresses in our data set. After completing these operations, we performed a tokenize operation. As the final stage; we removed the stop words and obtained the cleaned data set Root Retrieval One of the natural language processing processes is root retrieval processing. We have the python programming language, to root retrieval of our pre-processed data set. eveloped in 2.5. Classifiers 438

7 In this work, which is performed in the python programming language, the success metrics of the results obtained using the machine learning algorithms in the scikit-learn library(pedregosa et al., 2011) are compared. The parameter values of the selected algorithms are described below Linear SVM Linear SVM was originally formulated for binary classification(tang, 2013). The SVM classifies by constructing an N-dimensional hyperplanar that best separates the data into two categories(mitchell, 1997). The support vector machine is a supervised learning method that produces input-output mapping functions from a labeled set of training data(wang, 2005). We performed our work by selecting the default values of the algorithm Logistic Regression Logistic regression is the most common method used to model binary response data. When the response is binary, it typically takes the form of 1/0, with 1 generally indicating a success and 0 a failure. However, the actual values that 1 and 0 can take vary widely, depending on the purpose of the study(hilbe, 2011). We performed our work by selecting the default values of this algorithm Decision Tree A decision tree can be used as a model for a sequential decision problems under uncertainty. A decision tree describes graphically the decisions to be made, the events that may occur, and the outcomes associated with combinations of decisions and events(url). Decision tree models include such concepts as nodes, branches, terminal values, strategy, payoff distribution, certain equivalent, and the rollback method(url). The parameter values of our decision tree were chosen as (criterion = "gini", random_state = 100, max_depth=100, min_samples_leaf=8) KNN 439

8 A more sophisticated approach, k-nearest neighbor (KNN) classification, finds a group of k objects in the training set that are closest to the test object, and bases the assignment of a label on the predominance of a particular class in this neighborhood(wu et al., 2008). There are three key elements of this approach: a set of labeled objects, e.g., a set of stored records, a distance or similarity metric to compute distance between objects, and the value of k, the number of nearest neighbors(wu et al., 2008). The neighbor number of the algorithm in our study was choosen as Random Forest Random forest is an algorithm for classification developed by Leo Breiman that uses an -Uriarte and De Andres, 2006). Random forests are an effective tool in prediction. Because of the Law of Large Numbers they do not overfit. Injecting the right kind of randomness makes them accurate classifiers and regressors(breiman, 2001). We performed our work by selecting the default values of the algorithm. 3. Results and Discussion After applying the preprocessing and rooting operations, which are natural language processing steps, to our dataset and then we compared the machine learning algorithms after feature extracting of our data. We defined our performance metrics as accuracy and average recall. As a result of the study, we have achieved the highest accuracy percentage of 66.40% in the third model we created with Word2Vec model. We have also shown that two different Word2Vec models we have developed can increase accuracy and average recall values in most of the algorithms we implemented. agues, the highest success in the BOW model was achieved with 62.48%. However, with the BOW model we have developed, we have achieved the highest accuracy percentage of 65.18% in the BOW model by increasing their accuracy percentage by 4.32%. In addition,increasing their the highest accuracy percentage, we also achieved the highest accuracy percentage by 66,40% in our work. 440

9 Table 1. Results of The First Model Classifiers Accuracy Average Recall Tf-Idf+Linear SVM %65,18 %60,14 Tf-Idf+Logistic Reg. %64,10 %59,56 Tf-Idf+Decision Tree %59,66 %54,51 Tf-Idf+KNN Tf-Idf+Random Forest %57,04 %63,12 %55,73 %58,82 Table 2. Process Time of The First Model Classifiers Process Time(sec) Tf-Idf+Linear SVM 4,701 Tf-Idf+Logistic Reg. 12,451 Tf-Idf+Decision Tree 11,087 Tf-Idf+KNN Tf-Idf+Random Forest 8,781 33,286 As shown in Table 2, the lowest processing time was obtained with Linear SVM in the sentiment analysis study in the BOW model weighted Tf-Idf, while the highest processing time was obtained by Random Forest algorithm. In the first stage, the highest accuracy rate and lowest processing time were obtained by Linear SVM. Table 3. Results of The Second Model Classifiers Accuracy Average Recall Word2Vec+Linear SVM %51,20 %50,10 Word2Vec+Logistic Reg. %63,14 %62,72 Word2Vec+Decision Tree %66,12 %65,47 Word2Vec +KNN Word2Vec+Random Forest %65,77 %65,92 %65,15 %65,75 441

10 Table 4. Process Time of The Second Model Classifiers Process Time Word2Vec+Linear SVM 3,954 Word2Vec+Logistic Reg. 0,257 Word2Vec+Decision Tree 2,273 Word2Vec +KNN Word2Vec+Random Forest 13,654 1,383 As shown in Table 4, in our second model using word2vec, the lowest processing time was obtained with KNN. Using linear regression, we achieved a low process time and a accuracy rate close to the highest accuracy rate. Table 5. Results of The Third Model Classifiers Accuracy Average Recall Word2Vec+Linear SVM %51,28 %50,10 Word2Vec+Logistic Reg. %63,51 %62,86 Word2Vec+Decision Tree %66,18 %65,48 Word2Vec +KNN Word2Vec+Random Forest %65,92 %66,40 %65,30 %65,31 Table 6. Process Time of The Third Model Classifiers Process Time Word2Vec+Linear SVM 3,911 Word2Vec+Logistic Reg. 0,232 Word2Vec+Decision Tree 2,916 Word2Vec +KNN Word2Vec+Random Forest 13,572 1,500 As shown in Table 6, with the Word2Vec model, which was averaged by Tf-Idf-weighted word placements, the lowest processing time was obtained by Linear Regression in the third model, while the highest processing time was obtained by KNN. Using Random Forest algorithm, low process time and highest success rate were obtained. 442

11 4. Conclusions and Recommendations In this study, which is sentiment analysis study using nltk library of python programming language in Turkish natural language processing, we compared accuracy and average recall percentages by using different text representations for sentiment analysis study on twitter, which is one of the social media platforms. We have shown that better results can be achieved using the Word2Vec model, which is based on the semantic relationship, and that machine learning algorithms can be used effectively in sentiment analysis studies. We have also shown that by developing the used Word2Vec model, we can produce better results in itself. In order to achieve better results in our future studies, we aim to do the following studies; Working on a larger data set and increasing the number of labels in data. Working on data sets that are specific to a keyword. To develop a larger size corpus that can be used in Turkish natural language processing. To achieve higher accuracy by adding other algorithms more suitable to the models used. References Alghamdi, R., & Alfalqi, K. (2015). A survey of topic modeling in text mining. Int. J. Adv. Comput. Sci. Appl.(IJACSA), 6(1). Aslam, S., January 1, Twitter by the Numbers: Stats, Demographics & Fun Facts. It was taken from on 10/01/2018. Breiman, L. (2001). Random forests. Machine learning, 45(1), In Signal Processing and Communications Applications Conference (SIU), th (pp ). IEEE. -Uriarte, R., & De Andres, S. A. (2006). Gene selection and classification of microarray data using random forest. BMC bioinformatics, 7(1), 3. Gemci, F., & Peker, K. A. (2013, November). Extracting Turkish tweet topics using LDA. In Electrical and Electronics Engineering (ELECO), th International Conference on (pp ). IEEE. Grave, E, 3 May Pre-trained word vectors. It was taken from on 15/12/

12 Hilbe, J. M. (2011). Logistic regression. In International Encyclopedia of Statistical Science (pp ). Springer Berlin Heidelberg. Kim, T. Y., Min, M., Yoon, T., & Lee, J. H. (2010). Semantic Analysis of Twitter contents using PLSA, and LDA. In SCIS & ISIS SCIS & ISIS 2010 (pp ). Japan Society for Fuzzy Theory and Intelligent Informatics. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arxiv preprint arxiv: Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp ). Mitchell T. M., Machine Learning. McGraw-Hill Science, Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., & Stoyanov, V. (2016). SemEval-2016 task 4: Sentiment analysis in Twitter. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) (pp. 1-18). Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O.,... & Vanderplas, J. (2011). Scikit-learn: Machine learning in Python. Journal of machine learning research, 12(Oct), Ren, Y., Wang, R., & Ji, D. (2016). A topic-enhanced word embedding for twitter sentiment classification. Information Sciences, 369, Rong, X. (2014). word2vec parameter learning explained. arxiv preprint arxiv: Salton, Gerard, and Christopher Buckley. "Term-weighting approaches in automatic text retrieval." Information processing and management 24.5 (1988): Tang, Y. (2013). Deep learning using linear support vector machines. arxiv preprint arxiv: , 11 Jan Turkish Stemmer for Python. It was taken from on 12/01/2018. URL: Wang, L. (Ed.). (2005). Support vector machines: theory and applications (Vol. 177). Springer Science & Business Media. Wu, X., Kumar, V., Ross Quinlan, J. et al. Knowl Inf Syst (2008) 14: 1. Xie, W., Zhu, F., Jiang, J., Lim, E. P., & Wang, K. (2016). Topicsketch: Real-time bursty topic detection from twitter. IEEE Transactions on Knowledge and Data Engineering, 28(8),