"Obama never said that": Evaluating fact-checks for topical consistency and quality

Abstract

This thesis examines topical consistency between claims and fact-checks in the Birdwatch dataset published by Twitter. The dataset has tweets (the claims), notes (context-adding annotations written by Birdwatch users), and quality labels (ratings from the community of Birdwatch users). High quality notes can be thought of as potential “fact-checks” on the tweets. We find topics by clustering contextual word type embeddings (following a method introduced by Sia et al. [2020]) and evaluate two research questions: (1) Do notes that have high topic overlap with their associated tweet get better ratings? and (2) Can this topic modeling method be used to measure the helpful extra context that Birdwatch notes add to a tweet? Kullback-Leibler divergence is used to proxy topic overlap between the documents. We find that there is a statistically significant relationship between topic overlap and helpfulness but cannot establish a relationship between helpfulness and added context.