key: cord-0579380-tt32iymn
authors: Hamid, Abdullah; Shiekh, Nasrullah; Said, Naina; Ahmad, Kashif; Gul, Asma; Hassan, Laiq; Al-Fuqaha, Ala
title: Fake News Detection in Social Media using Graph Neural Networks and NLP Techniques: A COVID-19 Use-case
date: 2020-11-30
journal: nan
DOI: nan
sha: 945b8bd08eb997e837fae17f1be38ea705b3b502
doc_id: 579380
cord_uid: tt32iymn

The paper presents our solutions for the MediaEval 2020 task namely FakeNews: Corona Virus and 5G Conspiracy Multimedia Twitter-Data-Based Analysis. The task aims to analyze tweets related to COVID-19 and 5G conspiracy theories to detect misinformation spreaders. The task is composed of two sub-tasks namely (i) text-based, and (ii) structure-based fake news detection. For the first task, we propose six different solutions relying on Bag of Words (BoW) and BERT embedding. Three of the methods aim at binary classification task by differentiating in 5G conspiracy and the rest of the COVID-19 related tweets while the rest of them treat the task as ternary classification problem. In the ternary classification task, our BoW and BERT based methods obtained an F1-score of .606% and .566% on the development set, respectively. On the binary classification, the BoW and BERT based solutions obtained an average F1-score of .666% and .693%, respectively. On the other hand, for structure-based fake news detection, we rely on Graph Neural Networks (GNNs) achieving an average ROC of .95% on the development set.

In the modern world, social media is playing its part in several ways, for instance in news dissemination and information sharing, social media outlets, such as Twitter, Facebook, and Instagram, have been proved very effective [1, 6, 7, 9] . However, it also comes with several challenges, such as collecting information from several sources, detecting and filtering misinformation [4, 5, 11] . Similar to other events and pandemics, being one of the deadly pandemics in the history, COVID-19 has been the subject of discussion over social media since its emergence. Without any surprise, a lot of misinformation about the pandemic are circulated over social networks. In order to identify misinformation spreaders and filter fake news about COVID-19 and 5G conspiracy, a task namely "FakeNews: Corona Virus and 5G Conspiracy Multimedia Twitter-Data-Based Analysis" has been proposed in the benchmark MediaEval 2020 competition [8] .

This paper provides a detailed description of the methods proposed by team DCSE_UETP for the fake news detection task. The

Copyright 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). MediaEval'20, December 14-15 2020, Online task consists of two parts, namely (i) text-based misinformation detection (TMD), and (ii) structure-based misinformation detection (SMD). The first task (TMD) is based on textual analysis of COVID-19 related information shared on Twitter during January 2020 and 15th of July 2020, and aims to detect different types of conspiracy theories about COVID-19 and its vaccines, such as that "the 5G weakens the immune system and thus caused the current corona-virus pandemic etc., [8] . In the SMD task, the participants are provided with a set of graphs, each representing a sub-graph of Twitter, and corresponds to a single tweet where the vertices of the graphs represent accounts. Similar to TMD, in this task, the participants need to detect and differentiate between 5G and other COVID-19 conspiracy theories.

For the text-based analysis, we employed two different methods including a (i) Bag of Words (BoW), and a (ii) BERT model-based solution [3] . Before proceeding with the proposed methods, it is to be noted that the dataset provided for the text-based analysis is not balanced where one of the classes namely non-conspiracy contains a very high number of samples while the rest are composed of relatively fewer samples. In total, the majority class contains 4412, while the other two classes, namely 5G conspiracies, and other conspiracies, are composed of 1263 and 785 samples, respectively. In order to balance the dataset, we rely on an ensemble of different re-sampled datasets, where models are built/trained by dividing the class with a higher number of samples into n-differing parts as illustrated in Figure 1 . After training models, the results of the models are combined using two different late fusion methods including a majority voting method, and summation of the posterior probabilities. In the majority voting, since we have four models, in the case of tie we consider the accumulative probabilities/scores to assign a label to a test sample.

Before deploying BoW and BERT, text has been cleaned by removing punctuation's keys, such as commas, full-stops, emojis, URLs, and stop words. Once the text is pre-processed, we proceed with the tokenization and creation of BoW vocabulary, which is followed by generation of the feature vector for each sentence. A Naives Bayes classifier is then trained on the extracted features. On the other hand, a logistic regression model is trained on word embeddings generated via BERT. 

Graphs representation learning using Graph Neural Networks (GNNs) have been shown to be effective in various domains such as social networks, biological networks, and financial networks. GNNs aggregate the neighborhood representation within k hops and then apply a pooling such as SUM, MEAN, MAX to obtain the final representation of the node. Furthermore, GNN's can be used to learn the representation of a simple graph structures [2, 10, 12] , which then can be used to classify the graphs. For graph classification, these methods learn the representation of nodes, followed by graph READOUT method, which is aggregating the node features obtained after the final iteration of GNN.

We model this problem as a graph classification task. Following Keyule et al. [12] , we train our model using three classes of the graphs 5G Conspiracy, non-conspiracy, other-conspiracy, and learn the representation of the graphs.

For TMD, we submitted six different runs mainly relying on two approaches, namely BERT and BoW, under two late fusion schemes. Three of the runs are based on binary classification while the three deal the task as ternary classification problem. It is to be noted that the fusion schemes are used to combine the scores/output of the four individual models trained as result of the data balancing method as described earlier.

The first three runs are based on the ternary classification task, where run 1 and run 2 are based on BoW with majority voting and accumulative classification scores of the individual models. The third and final ternary run is based on BERT features, where a logistic regression model is trained on word embeddings generated by BERT. As can be seen in Table 1a , overall, better results are obtained with BoW approach under the majority voting scheme.

The last three runs are based on the binary classification task, where the first two (i.e., Run 4 and Run 5) are based on BoW with majority voting and accumulative classification based fusion methods while the final one (i.e., Run 6) is based on BERT with accumulative score based fusion scheme. As expected, the performance of all the methods is significantly higher on the binary classification task compared to ternary classification task. Similar trend has been also observed on the test set, where overall better results are obtained with BoW under majority voting scheme. 

The challenge is composed of two tasks, one aiming to analyze and detect COVID-19 related fake news using tweets' text while the other aims to analyze network structure for the possible detection of the fake news. For the first task, we mainly relied on two state-ofthe-art methods namely BoW and BERT embeddings under different fusion schemes. Overall better results are obtained with BoW under the majority voting scheme. For the SMD task, we rely on GNNs to differentiate among different conspiracy theories on COVID-19. In the current implementations, both textual and structural information are used independently, in the future we aim to enrich the structural information with the textual information for better detection of fake news.

Social media and satellites: Disaster event detection, linking and summarization

Bert: Pre-training of deep bidirectional transformers for language understanding

Unsupervised Fake News Detection: A Graphbased Approach

Graph Neural Networks with Continual Learning for Fake News Detection from Social Media

Twitter as a lifeline: Human-annotated twitter corpora for NLP of crisis-related messages

How events determine spreading patterns: information transmission via internal and external influences on social networks

FakeNews: Corona Virus and 5G Conspiracy Task at MediaEval 2020

Natural disasters detection in social media and satellite imagery: a survey

How Powerful are Graph Neural Networks?

Unsupervised fake news detection on social media: A generative approach

Hierarchical Graph Representation Learning with Differentiable Pooling