Text classification is a well-studied problem in information retrieval and data mining, with applications ranging from healthcare diagnosis to the detection of COVID-19-related fake news. Social media platforms generate vast volumes of user-generated text, and the rapid spread of misinformation during the pandemic has made automated fake news detection an important task. In this study, the authors apply pretrained transformer-based transfer learning models and large language models, namely BERT-base, ALBERT-base, XLM-RoBERTa-base, DistilBERT, GPT-2, XLNet, and T5, to the COVID-19 English Tweets dataset, which contains 6,420 tweets labeled as real or fake.
To further improve performance, the authors apply six late-fusion optimization techniques: weighted fusion, particle swarm optimization (PSO) fusion, the Nelder-Mead algorithm, Powell's method, the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, and trust-region constrained optimization. Models are evaluated using accuracy, precision, recall, and F1-score, and confusion-matrix heatmaps are used to visualize the predictions of each model. GPT-2 achieved the best performance, with an accuracy of 0.9631, precision of 0.9612, recall of 0.9610, and F1-score of 0.9600, while Nelder-Mead fusion achieved the best result among the fusion methods with an accuracy of 0.9610. These results highlight the value of LLMs for COVID-19 fake news detection. The authors conclude by outlining future directions, including multiclass classification, multilingual datasets, and real-time social media streams.
Model Selection and Transfer Learning Foundations
The study evaluates seven model architectures. BERT-base and ALBERT-base are widely used encoder-only models. XLM-RoBERTa-base extends multilingual capabilities. DistilBERT offers a distilled, lightweight variant. GPT-2 is an autoregressive decoder model. XLNet employs permutation-language-modeling objectives. T5 casts text tasks as a text-to-text format. All models are pretrained and finetuned on the labeled tweet dataset. The activation dimension across vision and language settings in the companion paper is 768, but here the focus is exclusively on text classification.
Late-Fusion Optimization and Ensembling
Six fusion techniques are examined. Weighted fusion combines model scores with fixed or learned weights. Particle swarm optimization (PSO) fusion treats weight selection as an optimization problem over particle trajectories. The Nelder-Mead algorithm performs derivative-free search in the weight space. Powell's method also avoids gradient computation through directional exploration. The BFGS algorithm approximates the Hessian to guide convergence. Trust-region constrained optimization limits steps to a region where the model is well-approximated. Each technique produces a combined prediction from the individual model outputs. The Nelder-Mead approach achieves the highest accuracy among fusion methods at 0.9610, narrowly trailing the single-model GPT-2 result.
Empirical Results and Metric Summary
Performance metrics across all models and fusion approaches are reported using accuracy, precision, recall, and F1-score. GPT-2 alone achieves an accuracy of 0.9631, precision of 0.9612, recall of 0.9610, and F1-score of 0.9600. These scores place GPT-2 above all other individual models and above all fusion-combined results except that the Nelder-Mead fusion reaches an accuracy of 0.9610, which is competitive but does not surpass the single-model GPT-2 result. Confusion-matrix heatmaps visualize true-positive, false-positive, false-negative, and true-negative counts for each approach. The dataset size of 6,420 tweets provides a moderate-scale testbed; results may vary with larger or more imbalanced datasets.
Sparse Probing and Representation Quality
The authors do not report sparse-probing experiments in this paper. The focus remains on end-to-end classification metrics. Readers interested in whether individual latent atoms capture class-relevant information may look to the companion work on sparse dictionary learning for related analysis.
Limitations and acknowledged constraints
The evaluation uses a binary real-or-fake labeling scheme. Multiclass classification, such as distinguishing between different types of misinformation, is noted as a future direction. The English-only tweet dataset limits multilingual generalization. Real-time social media streams introduce additional challenges with velocity and distribution shift. The 6,420-tweet dataset, while substantial for a proof-of-concept study, may not capture the full variability of pandemic-era misinformation.
Practical implications for developer workflows
Developers seeking to apply LLMs to fake news detection can start from the model pool examined in this study. GPT-2 provides a strong out-of-the-box baseline with the reported metrics. Fusion via Nelder-Mead optimization offers a simple, gradient-free method to combine multiple model predictions without requiring additional training. The code and dataset are publicly available, enabling reproducible experimentation. Developers working on multilingual or multiclass extensions can adapt the same evaluation framework.