Tweet Sentiment Extraction
F1 0.72 on Kaggle · BERT/RoBERTa ensemble for tweet sentiment span extraction with text normalization pipeline
Oct 2023
0.72311
F1 Score
Problem
Extract the precise text span from a tweet that supports its sentiment label — a token-level localization task where a single off-by-one token prediction tanks the F1 score. Tweets are noisy: URLs, HTML entities, inconsistent casing, and slang all degrade tokenization and model accuracy.
Approach
Built a tweet normalization pipeline (HTML/URL stripping, punctuation standardization, casing rules) — reducing tokenization noise and directly improving model accuracy on the clean signal.
Trained a BERT-based span extraction model with start/end logits as the primary baseline, establishing a strong single-model reference point.
Boosted test-set performance by ensembling RoBERTa and DistilBERT predictions via averaged logits — capturing complementary signal across model architectures.
Results
Achieved F1 0.72311 on the Kaggle competition test set through multi-model ensembling.
Demonstrated that tweet normalization preprocessing directly lifts accuracy by cleaning tokenization noise before training.
Validated ensemble approach: RoBERTa + DistilBERT averaged logits outperformed the single-model BERT baseline on held-out data.
Architecture Overview
Tweet normalization
BERT tokenizer
Span extraction head (start/end logits)
Ensemble (RoBERTa + DistilBERT)
Majority vote / avg logits
Predicted span
How to Run
Install: pip install transformers torch datasetsPreprocess data: python preprocess.py --data tweet_data.csvTrain: python train.py --model bert-base-uncased --epochs 5Ensemble inference: python ensemble_infer.py --models bert roberta distilbert