NLP: Amazon Sentiment, from Machine Learning to Transformers

Half a million reviews, mostly happy.

The corpus is the Amazon Fine Food Reviews dataset: 568,454 reviews of 74,258 products written between 1999 and 2012, each with a star rating. The ratings are heavily skewed toward five stars:

I turned the ratings into a binary sentiment: 1 and 2 stars negative, 4 and 5 positive, 3 stars left out as neutral. That gives 82,037 negative reviews against 443,777 positive ones, so I undersampled the positive class to a balanced study set of 164,074 reviews, 82,037 per class. The rating defines the label; the models only ever see the text.

Keep the words that carry sentiment.

All models share one custom cleaning function, the one the demo at the top runs on sample reviews. It strips HTML, URLs and emails, lowercases the text and expands contractions, removes numbers and punctuation, then uses spaCy to lemmatize and keep only nouns, verbs, adjectives and adverbs.

• Stopwords, tuned for sentiment. I took 146 sentiment words such as great, awful and recommend out of the standard stopword list, and added 193 domain words such as amazon, package and coffee that say nothing about sentiment.
• Negations survive. "not", "no" and "never" are joined to the next word, so not good becomes the single token not_good instead of a misleading good.

negation_tokens = {"not", "never", "no"}exceptions_token_text = {'not_have', 'not_as'}i = 0while i < len(doc):    token = doc[i]    # 'no', 'never' or 'not' is joined to the next token with an underscore: not_bad, no_talking ...    if token.lower_ in negation_tokens and i + 1 < len(doc):        next_token = doc[i + 1]        token_text = f"{token.text}_{next_token.text}"        if token_text not in exceptions_token_text:            filtered_sentence.append(token_text)        i += 2    else:        # keep adjectives, adverbs, nouns and verbs that are not stopwords, as lemmas        if (token.pos_ in ['ADJ', 'ADV', 'NOUN', 'VERB']                and token.lemma_ not in stop_words                and not token.is_punct                and not token.is_space                and not token.is_digit):            filtered_sentence.append(token.lemma_)        i += 1

A strong baseline first.

The classical models use a three-step scikit-learn pipeline: TF-IDF on unigrams and bigrams, chi-square feature selection keeping the 20,000 most informative features, then the classifier. Everything is fitted inside five-fold cross-validation, so no information leaks from the validation folds.

A linear SVM on well-cleaned TF-IDF features already gets nine reviews out of ten right. Any deep model has to beat that.

A small network that reads in context.

TF-IDF ignores word order. My network combines local phrases and long-range context: Word2Vec embeddings trained on the reviews themselves (128 dimensions), a convolution that picks up short phrases, and two bidirectional LSTMs that read the review in both directions.

Trained with RMSprop, L2 regularization, learning-rate reduction and early stopping, the network has 2,660,353 parameters and reached 92.63% accuracy on 16,408 test reviews, with a ROC AUC of 0.98.

Then a transformer, for comparison.

Finally I fine-tuned Hugging Face's distilbert-base-uncased, a distilled version of BERT with about 66 million parameters, for binary classification: 128-token inputs with attention masks, AdamW at a learning rate of 5 × 10⁻⁵, a linear scheduler and gradient clipping, trained in PyTorch on Apple Metal.

The same answer at a fraction of the size.

Each model's own held-out evaluation, as saved in the notebooks; the partitions differ, as the last column shows.

ROC AUC climbs from 0.96 to 0.98 across the four generations. The most interesting result is the last step: my network matches DistilBERT's ROC AUC with 25 times fewer parameters, and comes within a point of its accuracy. A well-designed small model, trained on the right representation, can be enough, and far cheaper to run.