All Projects
LLM/RAG

Tweet Sentiment Extraction

F1 0.72 on Kaggle · BERT/RoBERTa ensemble for tweet sentiment span extraction with text normalization pipeline

Oct 2023

0.72311

F1 Score

Problem

Extract the precise text span from a tweet that supports its sentiment label — a token-level localization task where a single off-by-one token prediction tanks the F1 score. Tweets are noisy: URLs, HTML entities, inconsistent casing, and slang all degrade tokenization and model accuracy.

Approach

1

Built a tweet normalization pipeline (HTML/URL stripping, punctuation standardization, casing rules) — reducing tokenization noise and directly improving model accuracy on the clean signal.

2

Trained a BERT-based span extraction model with start/end logits as the primary baseline, establishing a strong single-model reference point.

3

Boosted test-set performance by ensembling RoBERTa and DistilBERT predictions via averaged logits — capturing complementary signal across model architectures.

Results

Achieved F1 0.72311 on the Kaggle competition test set through multi-model ensembling.

Demonstrated that tweet normalization preprocessing directly lifts accuracy by cleaning tokenization noise before training.

Validated ensemble approach: RoBERTa + DistilBERT averaged logits outperformed the single-model BERT baseline on held-out data.

Architecture Overview

Tweet normalization

→

BERT tokenizer

→

Span extraction head (start/end logits)

→

Ensemble (RoBERTa + DistilBERT)

→

Majority vote / avg logits

→

Predicted span

How to Run

01Install: pip install transformers torch datasets
02Preprocess data: python preprocess.py --data tweet_data.csv
03Train: python train.py --model bert-base-uncased --epochs 5
04Ensemble inference: python ensemble_infer.py --models bert roberta distilbert

Tech Stack

PythonPyTorchTransformersBERTRoBERTaDistilBERT