Topic Classification of Reuters Newswires

The problem

46 topic classes, single-label, with pronounced imbalance. The largest class contains 35.2% of the training examples; the smallest contains 10. Each newswire is represented as a multi-hot word vector, which discards word order and term frequency and is the assignment's constraint rather than a design choice.

Why macro F1 leads

The metric hierarchy is led by macro F1, not accuracy, and this decision determines everything that follows.

Macro F1 weights every class equally. Accuracy weights every sample equally, which on an imbalanced corpus means it is dominated by a handful of common topics. A model can look strong on accuracy while being nearly useless on most of the classes it is nominally predicting.

The floor makes this concrete: uniform chance is 0.0217 accuracy, while a majority-class baseline reaches 0.3517 validation accuracy at 0.0113 macro F1. Thirty-five percent accuracy comes from predicting one class forever.

Protocol

The supplied Keras test split is untouched during development. Capacity, Dropout and vocabulary size are screened one factor at a time on one reproducible 80:20 stratified split and one seed. Those results are screening decisions, not claims of stable differences.

Class weighting directly targets the imbalance problem, so the unweighted, square-root balanced and fully balanced schemes are compared across five training seeds. The two leading schemes are then tested across five independent stratified validation splits. A four-point learning curve is also repeated over five seeds.

The test split is opened only after the architecture, weighting scheme and epoch count are fixed. A fresh model is trained once on all 8,982 supplied training newswires before final evaluation.

The main finding: class weighting

Across five seeds on the fixed validation split, unweighted training reaches mean macro F1 0.5217 ± 0.0473. Square-root balanced weights raise it to 0.6219 ± 0.0135, while mean accuracy also rises from 0.8130 to 0.8283.

Fully balanced weighting reaches 0.6247 macro F1, only 0.0028 higher and inside the declared tie threshold. Square-root weighting wins the tie-break with lower validation loss and higher accuracy. Across five independent validation splits, the two weighting schemes remain effectively tied: 0.6122 for square-root weighting and 0.6114 for full balancing.

Results

Selected configuration: 128 and 128 hidden units, no Dropout, a 5,000-word vocabulary, square-root balanced class weights and nine epochs, trained on all training data.

The class-stratified macro-F1 interval is [0.5763, 0.6619]. It conditions on the observed support of every class; the unstratified interval instead resamples the observed class mixture. Neither includes retraining variability.

What the final gap means

0.8072 accuracy. 0.6280 macro F1. Balanced accuracy 0.6054.

Four fifths of newswires are classified correctly, but equal-weight class performance remains substantially lower. Class weighting narrows the problem rather than eliminating it: weighted F1 is 0.8034, while macro F1 is 0.6280.

The per-class analysis also resists simple conclusions from tiny supports. One class scores 1.000 F1 on a single test example, and support explains only part of the variation in class F1. Reporting accuracy alone would still tell only half the story.

Negative finding: dropout hurt, monotonically

Raising the Dropout rate lowers validation macro F1 at every step — from 0.5298 at rate 0.0 to 0.4898, 0.4577 and 0.3792 — while narrowing the train-validation accuracy gap from 0.1431 to 0.1214.

Dropout modestly reduces the fit difference but weakens the primary measure and raises validation loss. The selected configuration therefore uses no Dropout. The result applies to the tested width and does not rule out an interaction with another architecture.

More labelled data still helps

Mean validation macro F1 rises from 0.3995 at one quarter of the development data to 0.5139, 0.5655 and 0.6219 as more data are added. The final paired increase is 0.0564 ± 0.0137 and is positive for all five seeds. The curve has not flattened, especially on the equal-class measure, so collecting more labelled newswires remains the clearest next step.

Uncertainty

Bootstrap intervals accompany the headline figures rather than bare point estimates. The report separates an unstratified bootstrap, whose rare-class support varies between resamples, from a class-stratified bootstrap that preserves the observed support. Neither interval includes model retraining or model-selection uncertainty.

Scope

• Multi-hot representation, which discards order and frequency by design. Exact conflicting vector collisions account for only 4.4% of the observed test errors.
• Dense layers only, per the study scope.
• Five seeds and five splits, useful sensitivity checks but still small samples of training and partition variability.
• Sequential screening, so interactions among capacity, Dropout and vocabulary are not estimated.
• One historical corpus, so the results may not transfer to current news language.

Where I would take this next

• Collect more examples for low-support topics, because the learning curve is still rising.
• Repeat capacity and vocabulary screens across several seeds, promoting them from screening decisions to stable comparisons.
• Test a sequence representation, to measure what the order-free encoding costs.
• Estimate factor interactions, especially between Dropout and network width.