2025-03-26: A Battle of Opinions: Tools vs. Humans (and Humans vs. Humans) in Sentiment Analysis
Introduction
We analyzed the sentiment of 100 tweets using three sentiment analysis tools (TextBlob, VADER, and a RoBERTa-base model) and six human raters. To measure agreement, we calculated Cohen’s Kappa for each pair of raters (including both humans and tools) and Fleiss’ Kappa for multiple raters. The results? Let’s just say consensus was hard to find. Even the human raters struggled to agree, so we took a majority vote among them and compared it with the tools. Notably, the RoBERTa-base model showed the best alignment with human rating.
Our dataset consists of 100 tweets collected using the keyword “Site C, Khayelitsha” to study residents' perceptions of safety and security in Khayelitsha Township, South Africa, as part of the Minerva Research Initiative Grant awarded by the U.S. Department of Defense in 2022. We then collected the sentiment labels by running the sentiment analysis tools listed below and also by gathering data from six human raters. This data is available in a GitHub repository. Unfortunately, the project that made this work possible has been cancelled effective at the end of February, 2025.
The Sentiment Analysis Tools
We explored three sentiment analysis tools:
TextBlob: TextBlob is a Python library for processing textual data. The sentiment analysis feature in TextBlob returns a namedtuple, Sentiment(polarity, subjectivity), where polarity ranges from [-1.0, 1.0] and subjectivity ranges from [0.0, 1.0]. You can explore the tweet values in this notebook.
VADER: A rule-based model specifically developed for sentiment analysis of social media text. This is published in ICWSM-14 (2014) under the title “VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text”.
RoBERTa-base model: This is a model trained on ~58M tweets and fine tuned for sentiment analysis with the TweetEval benchmark in 2020. Recent updates to this model and more information can be found at TweetNLP.
Measuring Inter-Rater Agreement: Cohen’s Kappa
Cohen's kappa coefficient is a statistic that is used to measure inter-rater agreement between two raters for categorical items. The coefficient ranges from -1 to 1, with 1 indicating perfect agreement and 0 and less indicating no agreement. Values can be interpreted as: <0 (poor agreement), 0.01-0.20 (slight), 0.21-0.40 (fair), 0.41-0.60 (moderate), 0.61-0.80 (substantial), and 0.81-1.00 (almost perfect).
We first calculated the Cohen’s Kappa for each pair of raters (Table 1). The kappa values between human raters and tools vary, with some raters showing moderate agreement (e.g., Rater1 and Rater6 with a kappa of 0.5245) and others showing fair agreement (e.g., Rater1 and Rater2 with a kappa of 0.2802). The tools (TextBlob, VADER, and RoBERTa) generally have slight or fair agreement with the human raters, indicating that the tools are less consistent with human raters.
Table 1: Cohen’s Kappa - Positive, Negative, Neutral
The highest agreement between any two raters (human or tool) is highlighted in red.
Exploring Binary Sentiment Classification for Higher Agreement
We hypothesized that classifying sentiment as "negative vs. non-negative" (or even "positive vs. non-positive") might yield higher agreement than the traditional "Positive, Negative, Neutral" categories. This is because humans are generally more adept at identifying distinct positive or negative sentiments, while neutral statements tend to be more ambiguous and prone to subjective interpretation. To test our theory, we calculated the Cohen’s Kappa for Negative vs. Non-Negative (Table 2) and Positive vs. Non-Positive (Table 3).
The kappa values between raters for Negative vs. Non Negative show significantly higher agreement compared to the three-class sentiment classification in Table 1. For example, in Table 2, Rater1 and Rater6 have a kappa value of 0.8010, indicating almost perfect agreement. RoBERTa exhibits the highest agreement with the raters (for example, RoBERTa with Rater1 having a almost perfect kappa value of 0.8562), which is similar to what we saw in the three-class classification, but with noticeably stronger agreement in the binary negative vs. non-negative classification. VADER and TextBlob still show slight and fair agreement, especially VADER (for example, VADER with Rater3 having a kappa value of -0.0128 indicating poor agreement), which suggests it struggles with more nuanced sentiment identification.
Table 2: Cohen’s Kappa - Negative vs. Non Negative
The highest agreement between any two raters (human or tool) is highlighted in red.
As shown in Table 3, the kappa values for the Positive vs. Non-Positive classification also show a moderate increase in agreement compared to the three-class classification in Table 1, but the values are generally lower than the Negative vs. Non-Negative classification in Table 2. For example, Rater 1 and Rater 6 have slight agreement with a kappa value of 0.3429. However, as shown in Table 2, they have almost perfect agreement (kappa value of 0.8010) for the Negative vs. Non-Negative classification. RoBERTa continues to show better performance in both binary classifications (with kappa values like 0.5118 with Rater1), but there is still some variation in how it compares with other raters. VADER and TextBlob show weaker agreement in the positive vs. non-positive classification compared to negative vs. non-negative.
Table 3: Cohen’s Kappa - Positive vs. Non Positive
The highest agreement between any two raters (human or tool) is highlighted in red.
In summary, the binary classification approach shows improved agreement compared to the three-class system, the Positive vs. Non-Positive classification does not result in as strong an agreement as the Negative vs. Non-Negative case. This suggests that negative sentiment is more distinct and easier to identify than positive sentiment in our dataset.
Multi-Rater Agreement: Fleiss’ Kappa
Fleiss's kappa is a generalization of Cohen's kappa for more than two raters. In Table 4, we calculated Fleiss' Kappa to assess inter-rater agreement across multiple raters and tools for various sentiment classification tasks: Positive/Negative/Neutral, Negative/Non-Negative, and Positive/Non-Positive. Fleiss' Kappa, unlike Cohen's Kappa, is specifically designed for multiple raters and gives a measure of the consistency of the classifications.
Table 4: Fleiss' Kappa - Inter-rater agreement multiple raters/tools
The highest kappa value for each of the three classifications is highlighted in red.
Majority Voting: Aligning Human Raters for Better Consistency
As shown in Table 4, the individual human raters had varying interpretations of sentiment, leading to moderate inter-rater agreement at best. To address this inconsistency and improve reliability, we applied a majority voting approach, aggregating the most common sentiment label among human raters. This allowed us to establish a more stable consensus, which we then used to compare against sentiment analysis tools.
We noticed that applying majority voting among human raters significantly improved consistency, making their collective judgment more aligned with automated tools.
Table 5: Cohen’s kappa - Majority of Human Votes vs. each Tool
The highest kappa value for each of the three classifications is highlighted in bold.
RoBERTa shows the almost perfect agreement with the majority vote of human raters, particularly in Negative/Non-Negative (0.8761). It also shows substantial agreement in Positive/Negative/Neutral (0.7340) and Positive/Not Positive (0.6101). This indicates that RoBERTa is highly consistent with the human majority, especially when distinguishing between negative and non-negative sentiments, and also performs substantially in classifying positive/negative/neutral and positive/non-positive sentiment. VADER and TextBlob show fair/moderate agreement, with VADER performing moderately in Negative/Not Negative (0.4960) but still demonstrating lower kappa values compared to RoBERTa. TextBlob performs the worst, especially in Positive/Not Positive (0.2922), showing fair agreement with the majority vote of human raters.
Key Takeaways
Binary sentiment classification improves agreement
Categorizing sentiment as Negative vs. Non-Negative or Positive vs. Non-Positive resulted in higher agreement among human raters and between humans and sentiment analysis tools.
Negative vs. Non-Negative classification showed the strongest agreement, suggesting that negative sentiment is more distinct and easier to identify than positive sentiment.
RoBERTa emerges as the most reliable tool in matching human sentiment classifications, especially after applying majority voting, as it consistently shows substantial/almost perfect agreement with the majority vote across all sentiment categories. Also, RoBERTa’s consistency was particularly strong in Negative vs. Non-Negative classification (Kappa = 0.8761), indicating its ability to detect negative sentiment reliably.
VADER and TextBlob show weaker performance
VADER and TextBlob exhibit less agreement, particularly in Positive/Not Positive, indicating they might struggle with identifying positive sentiments as clearly as negative or non-negative ones.
TextBlob showed the lowest agreement across all categories, indicating that its sentiment classification approach may be less aligned with human perception.
We believe that RoBERTa is more effective than VADER and TextBlob for sentiment analysis because it applies transformer-based deep learning to understand the context of words. As opposed to VADER and TextBlob that rely on fixed rules and lexicons, RoBERTa is trained on a large dataset and fine-tuned through TweetEval, which makes it more effective at capturing richer sentiments, sarcasm, and slang. RoBERTa’s performance reminds us that the future of sentiment analysis lies in models that truly understand the complexity of human emotions in text.
Acknowledgments
The advice given by Dr. Michael Nelson and Dr. Michele Weigle has been a great help in compiling this blog post. I am grateful for their guidance and support, but the responsibility for any errors remains my own.
- - Himarsha R. Jayanetti (@HimarshaJ)
Comments
Post a Comment