Performance Evaluation of Sentiment Classification Models: A Comparative Study of NBC, SVM, and DT with SMOTE
DOI:
https://doi.org/10.30865/klik.v4i5.1827Keywords:
DT; NBC; Sentiment Classification; SMOTE; SVMAbstract
This research explores the performance of sentiment classification models, namely Naive Bayes Classifier (NBC), Decision Tree (DT), and Support Vector Machine (SVM), using the CRISP-DM methodology in the context of digital content analysis and data mining. The analysis was conducted on a SMOTE dataset in Rapidminer, yielding significant performance metrics. The NBC model achieved an accuracy of 86.98% +/- 0.96%, precision of 100.00% +/- 0.00%, recall of 78.82% +/- 1.55%, and f-measure of 88.15% +/- 0.97%, with an AUC of 0.657 +/- 0.203. Similarly, the DT model exhibited an accuracy of 93.20% +/- 0.42%, precision of 90.87% +/- 0.64%, recall of 98.88% +/- 0.31%, and f-measure of 94.70% +/- 0.31%, with an AUC of 0.918 +/- 0.006. Furthermore, the SVM model demonstrated an accuracy of 96.80% +/- 0.65%, precision of 98.99% +/- 0.28%, recall of 95.77% +/- 1.03%, and f-measure of 97.35% +/- 0.55%, with an AUC of 0.994. These findings highlight the efficacy of these models in accurately classifying sentiments within digital content, suggesting their suitability for various data mining applications. Recommendations for future research include exploring ensemble methods, continuous model updating, alternative sampling techniques, feature engineering approaches, and collaboration with domain experts to enhance real-world applicability
Downloads
References
O. B. Onyancha, “Indigenous knowledge, traditional knowledge and local knowledge: what is the difference? An informetrics perspective,” Glob. Knowledge, Mem. Commun., vol. 73, no. 3, pp. 237–257, Jan. 2024, doi: 10.1108/GKMC-01-2022-0011.
Y. Yang, X. Lin, and R. B. Anderson, “Entrepreneurship by indigenous people in Canada and Australia: diverse modes and community implications,” Int. J. Entrep. Behav. Res., vol. 30, no. 1, pp. 90–109, Jan. 2024, doi: 10.1108/IJEBR-01-2023-0085.
B. Beamer and K. C. Gleason, “Reflections on the impact of informal sector tourism on indigenous Namibian Craft processes,” Arts Mark., vol. 12, no. 1, pp. 1–16, Jan. 2022, doi: 10.1108/AAM-05-2020-0015.
C. Makate, “Local institutions and indigenous knowledge in adoption and scaling of climate-smart agricultural innovations among sub-Saharan smallholder farmers,” Int. J. Clim. Chang. Strateg. Manag., vol. 12, no. 2, pp. 270–287, Jan. 2020, doi: 10.1108/IJCCSM-07-2018-0055.
L. Koppenhafer, K. Scott, T. Weaver, and M. Mulder, “The service empowerment model: a collaborative approach to reducing vulnerability,” J. Serv. Mark., vol. 37, no. 7, pp. 911–926, Jan. 2023, doi: 10.1108/JSM-10-2022-0317.
S. Lee, Y. Chang, O. K. D. Lee, S. Ryu, and Q. Yin, “Exploring online social platform affordances for digital creators: a multi-method approach using qualitative and configurational analysis,” Ind. Manag. Data Syst., vol. 124, no. 4, pp. 1501–1530, Jan. 2024, doi: 10.1108/IMDS-12-2023-0951.
N. Gryllakis and M. Matsiola, “Digital audiovisual content in marketing and distributing cultural products during the COVID-19 pandemic in Greece,” Arts Mark., vol. 13, no. 1, pp. 4–19, Jan. 2023, doi: 10.1108/AAM-09-2021-0053.
N. Nicoli, K. Henriksen, M. Komodromos, and D. Tsagalas, “Investigating digital storytelling for the creation of positively engaging digital content,” EuroMed J. Bus., vol. 17, no. 2, pp. 157–173, Jan. 2022, doi: 10.1108/EMJB-03-2021-0036.
A. P. Kieling, R. Tezza, and G. L. Vargas, “Website stage model for Brazilian wineries: an analysis of presence in digital and mobile media,” Int. J. Wine Bus. Res., vol. 35, no. 1, pp. 45–65, Jan. 2023, doi: 10.1108/IJWBR-05-2021-0032.
C. Clune and E. McDaid, “Content moderation on social media: constructing accountability in the digital space,” Accounting, Audit. Account. J., vol. 37, no. 1, pp. 257–279, Jan. 2024, doi: 10.1108/AAAJ-11-2022-6119.
C. Chen and T. Kellison, “The clock is ticking: contexts, tensions and opportunities for addressing environmental justice in sport management,” Sport. Bus. Manag. An Int. J., vol. 13, no. 3, pp. 376–396, Jan. 2023, doi: 10.1108/sbm-08-2022-0071.
R. Colbourne, P. Moroz, C. Hall, K. Lendsay, and R. B. Anderson, “Indigenous works and two eyed seeing: mapping the case for indigenous-led research,” Qual. Res. Organ. Manag. An Int. J., vol. 15, no. 1, pp. 68–86, Jan. 2020, doi: 10.1108/QROM-04-2019-1754.
A. Kaur and W. Qian, “The state of disclosures on Aboriginal engagement: an examination of Australian mining companies,” Meditari Account. Res., vol. 29, no. 2, pp. 345–370, Jan. 2020, doi: 10.1108/MEDAR-01-2020-0702.
L. Bellato and J. M. Cheer, “Inclusive and regenerative urban tourism: capacity development perspectives,” Int. J. Tour. Cities, vol. 7, no. 4, pp. 943–961, Jan. 2021, doi: 10.1108/IJTC-08-2020-0167.
S. W. Maingi, “Safari tourism and its role in sustainable poverty eradication in East Africa: the case of Kenya,” Worldw. Hosp. Tour. Themes, vol. 13, no. 1, pp. 81–94, Jan. 2021, doi: 10.1108/WHATT-08-2020-0084.
I. Moyo and H. M. S. Cele, “Protected areas and environmental conservation in KwaZulu-Natal, South Africa: on HEIs, livelihoods and sustainable development,” Int. J. Sustain. High. Educ., vol. 22, no. 7, pp. 1536–1551, Jan. 2021, doi: 10.1108/IJSHE-05-2020-0157.
P. Scherrer, “Tourism to serve culture: the evolution of an Aboriginal tourism business model in Australia,” Tour. Rev., vol. 75, no. 4, pp. 663–680, Jan. 2020, doi: 10.1108/TR-09-2019-0364.
T. Neha et al., “Sustainable prosperity and enterprises for Maori communities in Aotearoa New Zealand: a review of the literature,” J. Enterprising Communities, vol. 15, no. 4, pp. 608–625, Jan. 2021, doi: 10.1108/JEC-07-2020-0133.
R. Obiedat et al., “Sentiment Analysis of Customers’ Reviews Using a Hybrid Evolutionary SVM-Based Approach in an Imbalanced Data Distribution,” IEEE Access, vol. 10, pp. 22260–22273, 2022, doi: 10.1109/ACCESS.2022.3149482.
Z. Wu, G. Cao, and W. Mo, “Multi-Tasking for Aspect-Based Sentiment Analysis via Constructing Auxiliary Self-Supervision ACOP Task,” IEEE Access, vol. 11, no. May, pp. 82924–82932, 2023, doi: 10.1109/ACCESS.2023.3276320.
R. Bringula, S. A. I. D. A. Ulfa, J. P. P. Miranda, and F. A. L. Atienza, “Text mining analysis on students’ expectations and anxieties towards data analytics course,” Cogent Eng., vol. 9, no. 1, 2022, doi: 10.1080/23311916.2022.2127469.
R. K. Botchway, A. B. Jibril, Z. K. Oplatková, and M. Chovancová, “Deductions from a Sub-Saharan African Bank’s Tweets: A sentiment analysis approach,” Cogent Econ. Financ., vol. 8, no. 1, 2020, doi: 10.1080/23322039.2020.1776006.
R. Harakawa, T. Ogawa, and M. Haseyama, “Extracting Hierarchical Structure of Web Video Groups Based on Sentiment-Aware Signed Network Analysis,” IEEE Access, vol. 5, pp. 16963–16973, 2017, doi: 10.1109/ACCESS.2017.2741098.
J. Chen, Q. Mao, and L. Xue, “Visual sentiment analysis with active learning,” IEEE Access, vol. 8, pp. 185899–185908, 2020, doi: 10.1109/ACCESS.2020.3024948.
H. Zhang, S. Sun, Y. Hu, J. Liu, and Y. Guo, “Sentiment Classification for Chinese Text Based on Interactive Multitask Learning,” IEEE Access, vol. 8, pp. 129626–129635, 2020, doi: 10.1109/ACCESS.2020.3007889.
C. B. Lee, H. N. Io, and H. Tang, “Sentiments and perceptions after a privacy breach incident,” Cogent Bus. Manag., vol. 9, no. 1, 2022, doi: 10.1080/23311975.2022.2050018.
V. Gupta, S. Singh, and S. S. Yadav, “The impact of media sentiments on IPO underpricing,” J. Asia Bus. Stud., vol. 16, no. 5, pp. 786–801, Jan. 2022, doi: 10.1108/JABS-10-2020-0404.
F. Caviggioli, L. Lamberti, P. Landoni, and P. Meola, “Technology adoption news and corporate reputation: sentiment analysis about the introduction of Bitcoin,” J. Prod. Brand Manag., vol. 29, no. 7, pp. 877–897, Jan. 2020, doi: 10.1108/JPBM-03-2018-1774.
W. Zheng, S. Zhang, C. Yang, and P. Hu, “Lightweight multilayer interactive attention network for aspect-based sentiment analysis,” Conn. Sci., vol. 35, no. 1, 2023, doi: 10.1080/09540091.2023.2189119.
K. Puh and M. Bagi? Babac, “Predicting sentiment and rating of tourist reviews using machine learning,” J. Hosp. Tour. Insights, vol. 6, no. 3, pp. 1188–1204, 2023, doi: 10.1108/JHTI-02-2022-0078.
F. Alattar and K. Shaalan, “A Survey on Opinion Reason Mining and Interpreting Sentiment Variations,” IEEE Access, vol. 9, pp. 39636–39655, 2021, doi: 10.1109/ACCESS.2021.3063921.
J. Wu, K. Lu, S. Su, and S. Wang, “Chinese Micro-Blog Sentiment Analysis Based on Multiple Sentiment Dictionaries and Semantic Rule Sets,” IEEE Access, vol. 7, pp. 183924–183939, 2019, doi: 10.1109/ACCESS.2019.2960655.
K. Xu, H. Zhao, and T. Liu, “Aspect-Specific Heterogeneous Graph Convolutional Network for Aspect-Based Sentiment Classification,” IEEE Access, vol. 8, pp. 139346–139355, 2020, doi: 10.1109/ACCESS.2020.3012637.
I. Z. P. Hamdan and M. Othman, “Predicting Customer Loyalty Using Machine Learning for Hotel Industry,” J. Soft Comput. Data Min., vol. 3, no. 2, pp. 31–42, 2022.
Y. A. Singgalen, “Social Network Analysis and Sentiment Classification of Extended Reality Product Content,” J. Tek. Inform. C.I.T Medicom, vol. 16, no. 1, pp. 24–34, 2024.
J. A. Syahid and D. Mahdiana, “Perbandingan algoritma untuk klasifikasi analisis sentimen terhadap Genose pada media sosial Twitter,” semanTIK, vol. 7, no. 1, pp. 9–16, 2021, doi: 10.5281/zenodo.5034916.
H. Kim and G. Qin, “Summarizing Students’ Free Responses for an Introductory Algebra-Based Physics Course Survey Using Cluster and Sentiment Analysis,” IEEE Access, vol. 11, no. July, pp. 89052–89066, 2023, doi: 10.1109/ACCESS.2023.3305260.
K. Jahanbin and M. A. Z. Chahooki, “Aspect-Based Sentiment Analysis of Twitter Influencers to Predict the Trend of Cryptocurrencies Based on Hybrid Deep Transfer Learning Models,” IEEE Access, vol. 11, no. November, pp. 121656–121670, 2023, doi: 10.1109/ACCESS.2023.3327060.
K. R. Mabokela, T. Celik, and M. Raborife, “Multilingual Sentiment Analysis for Under-Resourced Languages: A Systematic Review of the Landscape,” IEEE Access, vol. 11, no. February, pp. 15996–16020, 2023, doi: 10.1109/ACCESS.2022.3224136.
T. Lin and I. Joe, “An Adaptive Masked Attention Mechanism to Act on the Local Text in a Global Context for Aspect-Based Sentiment Analysis,” IEEE Access, vol. 11, no. May, pp. 43055–43066, 2023, doi: 10.1109/ACCESS.2023.3270927.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Performance Evaluation of Sentiment Classification Models: A Comparative Study of NBC, SVM, and DT with SMOTE
ARTICLE HISTORY
Issue
Section
Copyright (c) 2024 Yerik Afrianto Singgalen
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
