Natural Language Processing Filter.
Building a model to filter and categorize movie reviews based on sentiment
using a labeled dataset of IMDB reviews tagged by polarity.
CLIENT:
Film Junky Union
DATE:
May 31, 2025
PROJECT:
Exploration and Preprocessing
I began by exploring the dataset to check for duplicates, missing values, and class imbalances, none were found.
Through exploratory data analysis (EDA), I discovered interesting trends: while the number of films increased yearly, the number of reviews did not grow proportionally, and no review data was available beyond 2009. Visualizations included yearly review counts, reviews per movie, and polarity distributions across training and test sets, which were found to be consistent.
Text Processing & Model Development
Text data was cleaned, tokenized, and lemmatized using NLTK and spaCy, then transformed into numerical features via TF-IDF vectorization.
I first used a Dummy Classifier as a sanity check, then trained multiple models:
- Model 1: NLTK + TF-IDF + Logistic Regression → F1 score: 0.88
- Model 2: spaCy + TF-IDF + Logistic Regression → F1 score: 0.88
- Model 3: spaCy + TF-IDF + LGBMClassifier → F1 score: 0.85
- Model 4: BERT fine-tuning → Lower performance due to dataset size
