Topic Classification of Reuters Newswires with Densely Connected Networks

A systematic application of the universal deep learning workflow

Dataset: Reuters newswire topic classification, 46 single-label classes

Author: Omar Ashraf Mohammed

Module: CM3015: Machine Learning and Neural Networks

---

This notebook applies the universal deep learning workflow in Section 4.5 of Deep Learning with Python (Chollet, 2017). Every predictive model is a Keras Sequential network restricted to Dense and Dropout layers. The final assessed artefact is the HTML export of this executed notebook.

Abstract

This study applies the universal deep learning workflow to the imbalanced 46-class Reuters newswire dataset. Newswires are represented by multi-hot word vectors and classified with Keras Sequential networks restricted to Dense and Dropout layers. A stratified validation partition supports controlled experiments on capacity, Dropout and vocabulary size. Class weighting is then evaluated over five training seeds and five independent validation splits. Square-root balanced weights raise mean validation macro F1 from 0.5217 to 0.6219 and reduce its seed standard deviation from 0.0473 to 0.0135. The selected model has two 128-unit hidden layers, no Dropout, a 5,000-word vocabulary and nine training epochs. On the untouched test split it achieves accuracy 0.8072 with a 95 percent bootstrap interval of [0.7912, 0.8241], macro F1 0.6280 with interval [0.5484, 0.6506], and weighted F1 0.8034. A learning curve remains positive through the full development partition, with a mean final macro F1 increase of 0.0564 across five paired runs. Class weighting therefore improves equal-weight class performance without reducing overall accuracy in these experiments. The main limitation is that multi-hot encoding discards word order and frequency, and the conclusions apply only to the tested Dense model family.

1. Introduction and scope

1.1 Aim and context

The aim is to determine how capacity, Dropout, vocabulary size and class weighting affect a densely connected classifier for Reuters newswires. Reuters is a useful test of multiclass learning because 46 topics are distributed unevenly and several classes contain few examples. Chollet's worked Reuters example establishes a relevant Dense-network reference for the same representation and reports test accuracy of approximately 0.797 (Chollet, 2017, Chapter 3). This investigation extends that example through controlled model comparison, rare-class metrics and repeated estimates of variability.

1.2 Scope

Every neural model is a Keras Sequential model containing only Dense and Dropout layers. The supplied test split is reserved for one final evaluation. All analysis, code and interpretation are contained in this notebook.

1.3 Research questions

• RQ1. Does a small Dense network exceed a majority-class baseline on validation macro F1?
• RQ2. How does unregularised capacity affect fit and overfitting?
• RQ3. Does Dropout improve validation macro F1?
• RQ4. Does a 10,000-word vocabulary improve on a 5,000-word vocabulary?
• RQ5. Does class weighting improve rare-class performance without an unacceptable loss of overall accuracy?
• RQ6. Is performance still improving as the amount of labelled training data increases?

Background

A Dense layer applies a learned affine transformation, (z = Wx + b), followed by an activation function. Each hidden unit can therefore combine evidence from every input feature. ReLU, (\max(0,z)), introduces non-linearity and permits stacked Dense layers to learn decision boundaries that a single linear transformation cannot express. The final layer has 46 units with softmax activation. Softmax converts the output scores into non-negative class probabilities that sum to one, so the largest probability supplies the predicted topic.

Categorical cross-entropy is the negative logarithm of the probability assigned to the correct class. Backpropagation uses the chain rule to calculate how each weight contributed to this loss. RMSprop then updates the weights using a moving average of squared gradients, which controls the scale of each update (Hinton, Srivastava and Swersky, 2012). Validation loss is used to select an epoch because it evaluates the complete predicted probability distribution.

Greater capacity can reduce training error but also fit patterns that do not generalise. Early stopping retains the weights from the minimum validation loss. Dropout supplies a second form of regularisation by randomly setting a proportion of hidden activations to zero during training (Srivastava et al., 2014). This discourages reliance on particular activation paths. At evaluation time all units are active and their outputs are scaled consistently by Keras.

Reuters is strongly imbalanced. Standard cross-entropy gives each newswire equal influence, so frequent classes contribute most of the training objective. Balanced class weights assign class (c) the weight (n/(K n_c)), where (n) is the training size, (K) the number of classes and (n_c) the support of class (c). A square-root scheme moderates these weights. Both schemes preserve the model architecture and operate through Keras training weights. Macro F1 gives each class equal importance during selection, while weighted F1 and accuracy describe performance under the observed class distribution (He and Garcia, 2009).

Environment and global controls

The next cells record the software environment, fix random seeds and request deterministic TensorFlow operations. Dataset exploration is limited to information needed to justify the evaluation measures and experimental choices.

# Core scientific stack. Only Dense and Dropout are permitted as model layers;
# NumPy, pandas, Matplotlib, seaborn, and scikit-learn support data handling,
# metrics, and figures, and Keras callbacks act as training controls.
import os

# Quieten TensorFlow low-level C++ logs before import so the report output stays
# clean; this affects logging only, not computation.
os.environ.setdefault("TF_CPP_MIN_LOG_LEVEL", "3")

import ast
import platform
from collections import Counter, defaultdict
import time
import random
import warnings

import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns

import sklearn
from sklearn.model_selection import train_test_split
from sklearn.metrics import (
    f1_score,
    precision_score,
    recall_score,
    accuracy_score,
    balanced_accuracy_score,
    cohen_kappa_score,
    matthews_corrcoef,
    top_k_accuracy_score,
    roc_auc_score,
    log_loss,
    classification_report,
    confusion_matrix,
)
from sklearn.preprocessing import label_binarize

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.callbacks import EarlyStopping

# Silence known-harmless warnings so the report output stays clean: the Keras
# input_shape notice, since the brief requires input_shape on the first Dense
# layer, and scikit-learn notices raised when a bootstrap resample of the
# imbalanced label set omits a rare class. That omission is a real property of
# resampling an imbalanced set, and its effect on the macro F1 interval is
# measured in Section 3 and is not assumed away.
from sklearn.exceptions import UndefinedMetricWarning
warnings.filterwarnings("ignore", category=UndefinedMetricWarning)
warnings.filterwarnings("ignore", message="Do not pass an", category=UserWarning)
warnings.filterwarnings("ignore", message="y_pred contains classes not in y_true")
# Raise the TensorFlow logger to ERROR so tf.function retracing notices during
# the many short predict calls do not clutter the report output.
tf.get_logger().setLevel("ERROR")

# Version and hardware provenance for reproducibility.
versions = pd.DataFrame(
    {
        "component": [
            "Python", "TensorFlow", "Keras", "NumPy", "pandas",
            "scikit-learn", "Matplotlib", "seaborn",
        ],
        "version": [
            platform.python_version(), tf.__version__, keras.__version__,
            np.__version__, pd.__version__, sklearn.__version__,
            matplotlib.__version__, sns.__version__,
        ],
    }
)

gpus = tf.config.list_physical_devices("GPU")
hardware = f"{platform.system()} {platform.machine()}, GPUs visible to TensorFlow: {len(gpus)}"
print(hardware)
versions

Darwin arm64, GPUs visible to TensorFlow: 0

	component	version
0	Python	3.12.0
1	TensorFlow	2.18.0
2	Keras	3.6.0
3	NumPy	2.1.2
4	pandas	2.2.2
5	scikit-learn	1.5.2
6	Matplotlib	3.9.2
7	seaborn	0.13.2

# Controls used throughout the investigation.
RANDOM_STATE = 42
REUTERS_SEED = 113
VOCAB_MAIN = 10000
VOCAB_SMALL = 5000
NUM_CLASSES = 46
BATCH_SIZE = 512
MAX_EPOCHS = 30
SEEDS = [7, 42, 113, 2026, 31415]
TIE_THRESHOLD = 0.005

EARLY_STOPPING = dict(
    monitor="val_loss",
    patience=4,
    min_delta=0.001,
    restore_best_weights=True,
)

os.environ["PYTHONHASHSEED"] = "0"
try:
    tf.config.experimental.enable_op_determinism()
except Exception:
    pass


def set_all_seeds(seed):
    random.seed(seed)
    np.random.seed(seed)
    keras.utils.set_random_seed(seed)


sns.set_theme(style="whitegrid", palette="colorblind")
plt.rcParams.update({
    "figure.dpi": 110,
    "font.size": 11,
    "axes.titlesize": 12,
    "axes.labelsize": 11,
    "legend.fontsize": 10,
    "figure.autolayout": True,
})
PALETTE = sns.color_palette("colorblind")

pd.set_option("display.max_rows", 120)
pd.set_option("display.max_colwidth", None)
print("Global controls set.")

Global controls set.

2. The universal deep learning workflow

This section applies the seven steps of the universal workflow (Chollet, 2017, Section 4.5) to the Reuters task. Each step states its purpose before the code and interprets the outcome after the result.

2.1 Step 1: Defining the problem and assembling the dataset

The task is single-label multiclass classification. One newswire is an observation, its word indices form the input, and one of 46 topic identifiers is the target. The model returns a probability distribution over these classes. Supervised learning assumes that labels are sufficiently reliable and that future observations resemble the historical training distribution.

Reuters is loaded twice with the same shuffle seed so each vocabulary experiment uses Keras's corresponding out-of-vocabulary handling. The labels and split membership are asserted to be identical.

# Load matching 10,000-word and 5,000-word versions of Reuters.
(train_data, train_labels), (test_data, test_labels) = keras.datasets.reuters.load_data(
    num_words=VOCAB_MAIN,
    seed=REUTERS_SEED,
)
(train_data_small, train_labels_small), (test_data_small, _) = (
    keras.datasets.reuters.load_data(num_words=VOCAB_SMALL, seed=REUTERS_SEED)
)

train_labels = np.asarray(train_labels, dtype="int64")
test_labels = np.asarray(test_labels, dtype="int64")
train_labels_small = np.asarray(train_labels_small, dtype="int64")

assert np.array_equal(train_labels, train_labels_small)
assert set(np.unique(train_labels)) == set(range(NUM_CLASSES))
print(f"Training newswires: {len(train_data)}   Test newswires: {len(test_data)}")

Training newswires: 8982   Test newswires: 2246

# Compact dataset summary. Sequence lengths describe raw newswire lengths before
# multi-hot encoding, which motivates the fixed-length representation in Step 4.
train_lengths = np.array([len(seq) for seq in train_data])
class_counts = pd.Series(train_labels).value_counts().sort_index()

summary = pd.DataFrame(
    {
        "quantity": [
            "Training examples", "Test examples", "Number of classes",
            "Vocabulary limit", "Median newswire length (tokens)",
            "Mean newswire length (tokens)", "Maximum newswire length (tokens)",
            "Largest class share", "Smallest class support",
        ],
        "value": [
            len(train_data), len(test_data), NUM_CLASSES, VOCAB_MAIN,
            int(np.median(train_lengths)), round(float(train_lengths.mean()), 1),
            int(train_lengths.max()),
            f"{class_counts.max() / len(train_labels):.1%} (class {int(class_counts.idxmax())})",
            f"{int(class_counts.min())} (class {int(class_counts.idxmin())})",
        ],
    }
)
summary

	quantity	value
0	Training examples	8982
1	Test examples	2246
2	Number of classes	46
3	Vocabulary limit	10000
4	Median newswire length (tokens)	95
5	Mean newswire length (tokens)	145.5
6	Maximum newswire length (tokens)	2376
7	Largest class share	35.2% (class 3)
8	Smallest class support	10 (class 35)

Interpretation. The training partition contains 8982 newswires and the reserved test partition 2246. Newswire lengths are short and right skewed, with a median of 95 tokens. Class support is markedly uneven: the largest class holds 35.2% of the training examples while the smallest holds only 10. This imbalance is the central reason for selecting macro F1 as the primary measure in Step 2, since a classifier can reach moderate accuracy by favouring frequent topics while failing the rare ones.

2.2 Step 2: Choosing a measure of success

Categorical cross-entropy is the optimisation loss and selects the training epoch. Accuracy measures the overall correct proportion. Macro F1 gives every class equal weight and is the primary model-selection measure because Reuters is imbalanced. Weighted F1 describes performance under the observed class frequencies.

Models are ranked by validation macro F1. Values within 0.005 are treated as practically tied, with lower validation cross-entropy and then fewer parameters used as tie-breaks. This is a decision rule and does not establish statistical equivalence. Balanced accuracy, top-k accuracy and calibration are reserved for final diagnosis.

# The metric hierarchy is fixed here so that later selection is transparent.
metric_plan = pd.DataFrame(
    {
        "measure": [
            "Categorical cross-entropy", "Macro F1", "Accuracy",
            "Weighted F1", "Generalisation gap", "Parameter count",
            "Balanced accuracy", "Cohen's kappa",
            "Matthews correlation coefficient", "Top-3 and top-5 accuracy",
            "Macro one-vs-rest ROC AUC", "Expected calibration error",
        ],
        "role": [
            "Optimisation objective and epoch selection",
            "Primary model-selection measure",
            "Secondary, interpretable overall measure",
            "Secondary, support-weighted quality",
            "Diagnostic for overfitting",
            "Tie-break and efficiency",
            "Final diagnostic, imbalance-robust recall",
            "Final diagnostic, chance-corrected agreement",
            "Final diagnostic, balanced multiclass correlation",
            "Final diagnostic, ranked-shortlist quality",
            "Final diagnostic, threshold-independent separability",
            "Final diagnostic, confidence calibration",
        ],
    }
)
metric_plan

	measure	role
0	Categorical cross-entropy	Optimisation objective and epoch selection
1	Macro F1	Primary model-selection measure
2	Accuracy	Secondary, interpretable overall measure
3	Weighted F1	Secondary, support-weighted quality
4	Generalisation gap	Diagnostic for overfitting
5	Parameter count	Tie-break and efficiency
6	Balanced accuracy	Final diagnostic, imbalance-robust recall
7	Cohen's kappa	Final diagnostic, chance-corrected agreement
8	Matthews correlation coefficient	Final diagnostic, balanced multiclass correlation
9	Top-3 and top-5 accuracy	Final diagnostic, ranked-shortlist quality
10	Macro one-vs-rest ROC AUC	Final diagnostic, threshold-independent separability
11	Expected calibration error	Final diagnostic, confidence calibration

The diagnostic functions are defined once and reused during final evaluation. Expected calibration error divides predictions into ten confidence intervals and compares mean confidence with observed accuracy (Guo et al., 2017). Macro one-vs-rest ROC AUC is retained as a secondary diagnostic and is not used for selection.

LABELS = list(range(NUM_CLASSES))


def macro_ovr_auc(y_true, y_proba):
    # One-vs-rest ROC AUC averaged over classes that contain both a positive and
    # a negative test case; classes without both cannot yield a defined AUC.
    binary = label_binarize(y_true, classes=LABELS)
    scores = []
    for k in LABELS:
        column = binary[:, k]
        if 0 < column.sum() < len(column):
            scores.append(roc_auc_score(column, y_proba[:, k]))
    return float(np.mean(scores)), len(scores)


def expected_calibration_error(confidence, correct, n_bins=10):
    # Partition predictions into equal-width confidence bins and accumulate the
    # support-weighted gap between mean confidence and empirical accuracy.
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    total = len(confidence)
    ece = 0.0
    rows = []
    for b in range(n_bins):
        low, high = edges[b], edges[b + 1]
        in_bin = (confidence > low) & (confidence <= high)
        if b == 0:
            in_bin = (confidence >= low) & (confidence <= high)
        count = int(in_bin.sum())
        if count == 0:
            rows.append({"bin_low": low, "bin_high": high, "count": 0,
                         "accuracy": np.nan, "confidence": np.nan})
            continue
        acc = float(correct[in_bin].mean())
        conf = float(confidence[in_bin].mean())
        ece += (count / total) * abs(acc - conf)
        rows.append({"bin_low": low, "bin_high": high, "count": count,
                     "accuracy": acc, "confidence": conf})
    return ece, pd.DataFrame(rows)


def full_metric_suite(y_true, y_pred, y_proba):
    # Scalar diagnostic suite reported at the final evaluation. Macro and
    # weighted variants use zero_division=0 so absent classes score zero.
    confidence = y_proba.max(axis=1)
    correct = (y_pred == y_true).astype(float)
    ece, _ = expected_calibration_error(confidence, correct)
    auc, auc_classes = macro_ovr_auc(y_true, y_proba)
    return {
        "Cross-entropy": log_loss(y_true, y_proba, labels=LABELS),
        "Accuracy": accuracy_score(y_true, y_pred),
        "Balanced accuracy": balanced_accuracy_score(y_true, y_pred),
        "Top-3 accuracy": top_k_accuracy_score(y_true, y_proba, k=3, labels=LABELS),
        "Top-5 accuracy": top_k_accuracy_score(y_true, y_proba, k=5, labels=LABELS),
        "Macro precision": precision_score(y_true, y_pred, average="macro",
                                           labels=LABELS, zero_division=0),
        "Macro recall": recall_score(y_true, y_pred, average="macro",
                                     labels=LABELS, zero_division=0),
        "Macro F1": f1_score(y_true, y_pred, average="macro",
                             labels=LABELS, zero_division=0),
        "Weighted precision": precision_score(y_true, y_pred, average="weighted",
                                              labels=LABELS, zero_division=0),
        "Weighted recall": recall_score(y_true, y_pred, average="weighted",
                                        labels=LABELS, zero_division=0),
        "Weighted F1": f1_score(y_true, y_pred, average="weighted",
                                labels=LABELS, zero_division=0),
        "Cohen's kappa": cohen_kappa_score(y_true, y_pred),
        "Matthews corrcoef": matthews_corrcoef(y_true, y_pred),
        f"Macro OVR ROC AUC ({auc_classes} classes)": auc,
        "Expected calibration error": ece,
    }

2.3 Step 3: Deciding on an evaluation protocol

The supplied test split is not used during model development. A reproducible stratified 80:20 split of the supplied training data creates development and validation partitions. Candidate comparisons use identical rows. Capacity, Dropout and vocabulary are screened on this controlled split. All three class-weighting schemes are then repeated across five training seeds. The two strongest schemes are also compared across five independent stratified splits, which tests sensitivity to partition choice.

Repeated use of validation data can favour its particular examples. Seed and split repetitions quantify parts of this uncertainty, although they do not replace evaluation on the untouched test set.

# One reproducible stratified 80:20 development and validation split.
# Integer targets are retained for scikit-learn metrics; one-hot targets are
# created in Step 4 for categorical cross-entropy.
dev_idx, val_idx = train_test_split(
    np.arange(len(train_labels)),
    test_size=0.20,
    random_state=RANDOM_STATE,
    stratify=train_labels,
)

# Disjointness and coverage assertions guard against leakage between partitions.
assert set(dev_idx).isdisjoint(set(val_idx))
assert len(dev_idx) + len(val_idx) == len(train_labels)

y_dev_int = train_labels[dev_idx]
y_val_int = train_labels[val_idx]

split_table = pd.DataFrame(
    {
        "partition": ["Development", "Validation", "Test (reserved)"],
        "examples": [len(dev_idx), len(val_idx), len(test_labels)],
        "share of training": [
            f"{len(dev_idx) / len(train_labels):.0%}",
            f"{len(val_idx) / len(train_labels):.0%}",
            "held out",
        ],
    }
)
split_table

	partition	examples	share of training
0	Development	7185	80%
1	Validation	1797	20%
2	Test (reserved)	2246	held out

2.4 Step 4: Preparing the data

Each newswire becomes a fixed-length binary vector whose entries record whether a vocabulary item appears. This representation is compatible with Dense layers and follows Chollet's Reuters example. It discards word order and term frequency. Targets are one-hot encoded for categorical cross-entropy, while integer labels are retained for scikit-learn metrics.

def multi_hot_encode(sequences, dimension):
    # Encode word presence in a float32 matrix.
    if dimension <= 0:
        raise ValueError("dimension must be positive")
    result = np.zeros((len(sequences), dimension), dtype="float32")
    for row, sequence in enumerate(sequences):
        valid = [token for token in sequence if 0 <= token < dimension]
        result[row, valid] = 1.0
    return result


def sequences_for_vocab(dimension):
    if dimension == VOCAB_MAIN:
        return train_data, test_data
    if dimension == VOCAB_SMALL:
        return train_data_small, test_data_small
    raise ValueError(f"Unsupported vocabulary size: {dimension}")


def prepare_dev_val(dimension):
    training_sequences, _ = sequences_for_vocab(dimension)
    x_development = multi_hot_encode(
        [training_sequences[i] for i in dev_idx], dimension
    )
    x_validation = multi_hot_encode(
        [training_sequences[i] for i in val_idx], dimension
    )
    return x_development, x_validation


x_dev, x_val = prepare_dev_val(VOCAB_MAIN)
y_dev = keras.utils.to_categorical(y_dev_int, NUM_CLASSES)
y_val = keras.utils.to_categorical(y_val_int, NUM_CLASSES)

assert x_dev.dtype == np.float32 and x_val.dtype == np.float32
assert x_dev.shape[1] == VOCAB_MAIN and y_dev.shape[1] == NUM_CLASSES
assert set(np.unique(x_dev)).issubset({0.0, 1.0})
assert np.isfinite(x_dev).all()
assert np.isclose(y_dev.sum(axis=1), 1).all()
print(f"x_dev {x_dev.shape} {x_dev.dtype}   y_dev {y_dev.shape}")
print(f"x_val {x_val.shape}   validation targets {y_val.shape}")

x_dev (7185, 10000) float32   y_dev (7185, 46)
x_val (1797, 10000)   validation targets (1797, 46)

# Figure 1: class support in the development partition. This is the first figure
# in the report and appears only after the problem, measures, protocol, and data
# preparation have been explained. It motivates the macro F1 emphasis.
dev_counts = pd.Series(y_dev_int).value_counts().sort_index()

fig, ax = plt.subplots(figsize=(9, 3.2))
ax.bar(dev_counts.index, dev_counts.values, color=PALETTE[0])
ax.set_xlabel("Class identifier")
ax.set_ylabel("Development examples")
ax.set_title("Figure 1. Class support in the development partition")
ax.set_xlim(-1, NUM_CLASSES)
plt.show()
plt.close(fig)

Interpretation. Figure 1 confirms a long-tailed class distribution. A small number of topics dominate the development partition while many topics have limited support. Under this imbalance, accuracy is an incomplete measure, which is why macro F1 leads the selection hierarchy. The class labels are shown as integer identifiers because the installed Keras Reuters API does not supply official topic names, and inventing names is out of scope.

2.5 Step 5: Developing a model that performs above a baseline

Uniform chance and a majority-class rule establish elementary reference levels. The neural baseline has one 16-unit ReLU hidden layer and a 46-unit softmax output. A shared model factory ensures that all candidates use the same optimiser, loss and permitted layer family.

def build_model(input_dim, hidden_units, dropout_rate, num_classes, seed):
    # Sequential model of Dense and Dropout layers only. The first Dense layer
    # receives input_shape, so a separate Input layer is unnecessary. Dropout is
    # placed after every hidden Dense layer and never after the output.
    set_all_seeds(seed)
    model = keras.Sequential(name="reuters_mlp")
    for position, units in enumerate(hidden_units):
        if position == 0:
            model.add(layers.Dense(units, activation="relu", input_shape=(input_dim,)))
        else:
            model.add(layers.Dense(units, activation="relu"))
        if dropout_rate > 0.0:
            model.add(layers.Dropout(dropout_rate))
    model.add(layers.Dense(num_classes, activation="softmax"))

    # Permitted-layer audit: every layer must be Dense or Dropout.
    for layer in model.layers:
        assert isinstance(layer, (layers.Dense, layers.Dropout)), type(layer)

    model.compile(optimizer="rmsprop", loss="categorical_crossentropy",
                  metrics=["accuracy"])
    return model

def make_class_weights(labels_int, scheme):
    # Recalculate weights from the training rows used by each fit.
    if scheme == "none":
        return None
    counts = np.bincount(labels_int, minlength=NUM_CLASSES).astype(float)
    if np.any(counts == 0):
        raise ValueError("Every class must be present in the training partition")
    balanced = len(labels_int) / (NUM_CLASSES * counts)
    if scheme == "sqrt_balanced":
        weights = np.sqrt(balanced)
    elif scheme == "balanced":
        weights = balanced
    else:
        raise ValueError(f"Unknown class-weight scheme: {scheme}")
    weights = weights / np.mean(weights[labels_int])
    return {class_id: float(weight) for class_id, weight in enumerate(weights)}


def run_experiment(
    name,
    hidden_units,
    dropout_rate,
    vocab_size,
    seed,
    features,
    class_weight_scheme="none",
    use_early_stopping=True,
    max_epochs=MAX_EPOCHS,
    keep_history=False,
):
    x_tr, y_tr, x_va, y_va, y_va_int = features
    y_tr_int = y_tr.argmax(axis=1)
    keras.backend.clear_session()
    set_all_seeds(seed)

    model = build_model(
        x_tr.shape[1], hidden_units, dropout_rate, NUM_CLASSES, seed
    )
    early_stopper = EarlyStopping(**EARLY_STOPPING)
    callbacks = [early_stopper] if use_early_stopping else []
    class_weights = make_class_weights(y_tr_int, class_weight_scheme)

    start = time.perf_counter()
    history = model.fit(
        x_tr,
        y_tr,
        validation_data=(x_va, y_va),
        epochs=max_epochs,
        batch_size=BATCH_SIZE,
        class_weight=class_weights,
        callbacks=callbacks,
        verbose=0,
    )
    elapsed = time.perf_counter() - start

    if use_early_stopping:
        best_epoch = int(early_stopper.best_epoch) + 1
    else:
        best_epoch = int(np.argmin(history.history["val_loss"])) + 1

    # Evaluate both partitions with Dropout disabled and restored weights active.
    train_loss, train_acc = model.evaluate(
        x_tr, y_tr, batch_size=BATCH_SIZE, verbose=0
    )
    val_loss, val_acc = model.evaluate(
        x_va, y_va, batch_size=BATCH_SIZE, verbose=0
    )
    proba = model.predict(x_va, batch_size=BATCH_SIZE, verbose=0)
    preds = proba.argmax(axis=1)
    macro = f1_score(
        y_va_int,
        preds,
        average="macro",
        labels=LABELS,
        zero_division=0,
    )
    weighted = f1_score(
        y_va_int,
        preds,
        average="weighted",
        labels=LABELS,
        zero_division=0,
    )

    record = {
        "name": name,
        "hidden_units": str(list(hidden_units)),
        "dropout": dropout_rate,
        "vocab": vocab_size,
        "class_weight": class_weight_scheme,
        "seed": seed,
        "params": model.count_params(),
        "epochs_run": len(history.history["val_loss"]),
        "best_epoch": best_epoch,
        "train_loss": round(float(train_loss), 4),
        "val_loss": round(float(val_loss), 4),
        "train_acc": round(float(train_acc), 4),
        "val_acc": round(float(val_acc), 4),
        "val_macro_f1": round(float(macro), 4),
        "val_weighted_f1": round(float(weighted), 4),
        "acc_gap": round(float(train_acc - val_acc), 4),
        "seconds": round(elapsed, 1),
    }
    hist = history.history if keep_history else None
    keras.backend.clear_session()
    return record, hist


def parse_units(units_field):
    return ast.literal_eval(units_field)


FIT_COUNTER = {"n": 0}


def counted_run(*args, **kwargs):
    record, hist = run_experiment(*args, **kwargs)
    FIT_COUNTER["n"] += 1
    return record, hist


features_main = (x_dev, y_dev, x_val, y_val, y_val_int)
print("Model factory and experiment runner ready.")

Model factory and experiment runner ready.

# Baselines on the validation partition.
uniform_acc = 1.0 / NUM_CLASSES
majority_class = int(pd.Series(y_dev_int).value_counts().idxmax())
majority_pred = np.full_like(y_val_int, majority_class)
majority_acc = accuracy_score(y_val_int, majority_pred)
majority_macro = f1_score(y_val_int, majority_pred, average="macro",
                          labels=list(range(NUM_CLASSES)), zero_division=0)

baseline_table = pd.DataFrame(
    {
        "baseline": ["Uniform chance", "Majority class"],
        "val_accuracy": [round(uniform_acc, 4), round(majority_acc, 4)],
        "val_macro_f1": ["n/a", round(majority_macro, 4)],
    }
)
baseline_table

	baseline	val_accuracy	val_macro_f1
0	Uniform chance	0.0217	n/a
1	Majority class	0.3517	0.0113

# Small densely connected neural baseline: one hidden layer of 16 units. This is
# also the small_16 point in the Step 7 capacity phase, so its record is reused.
baseline_record, baseline_hist = counted_run(
    "small_16", [16], 0.0, VOCAB_MAIN, RANDOM_STATE,
    features_main, use_early_stopping=True, keep_history=True,
)
pd.DataFrame([baseline_record])[
    ["name", "params", "best_epoch", "val_loss", "val_acc", "val_macro_f1"]
]

	name	params	best_epoch	val_loss	val_acc	val_macro_f1
0	small_16	160798	30	0.8871	0.8058	0.4855

Interpretation of RQ1. Uniform chance gives accuracy 0.0217. Predicting the majority class gives accuracy 0.3517 and macro F1 0.0113. The small neural model reaches validation accuracy 0.8058 and macro F1 0.4855. It therefore learns useful topic structure and clearly exceeds the elementary baselines. The test set is not used here.

2.6 Step 6: Scaling up and establishing overfitting

To expose overfitting, a deliberately high-capacity network of three 256-unit hidden layers is trained without Dropout for the full 30 epochs with early stopping disabled. The purpose is diagnostic: the complete training curves reveal the onset and development of the training and validation divergence. This run is not automatically the selected model.

# High-capacity probe: three hidden layers of 256 units, no Dropout, no early
# stopping, so the full divergence is visible in the curves.
probe_record, probe_hist = counted_run(
    "probe_256_256_256", [256, 256, 256], 0.0, VOCAB_MAIN, RANDOM_STATE,
    features_main, use_early_stopping=False, max_epochs=30, keep_history=True,
)
probe_best_epoch = probe_record["best_epoch"]
pd.DataFrame([probe_record])[
    ["name", "params", "epochs_run", "best_epoch", "train_loss", "val_loss",
     "train_acc", "val_acc"]
]

	name	params	epochs_run	best_epoch	train_loss	val_loss	train_acc	val_acc
0	probe_256_256_256	2703662	30	7	0.0699	1.055	0.9729	0.8114

# Figure 2: training and validation curves for the high-capacity probe.
epochs_axis = range(1, len(probe_hist["loss"]) + 1)
fig, axes = plt.subplots(1, 2, figsize=(11, 3.8))

axes[0].plot(epochs_axis, probe_hist["loss"], color=PALETTE[0], label="Training")
axes[0].plot(epochs_axis, probe_hist["val_loss"], color=PALETTE[3], label="Validation")
axes[0].axvline(probe_best_epoch, color="grey", linestyle="--", linewidth=1,
                label=f"Min val loss (epoch {probe_best_epoch})")
axes[0].set_xlabel("Epoch"); axes[0].set_ylabel("Categorical cross-entropy")
axes[0].set_title("Loss"); axes[0].legend()

axes[1].plot(epochs_axis, probe_hist["accuracy"], color=PALETTE[0], label="Training")
axes[1].plot(epochs_axis, probe_hist["val_accuracy"], color=PALETTE[3], label="Validation")
axes[1].axvline(probe_best_epoch, color="grey", linestyle="--", linewidth=1)
axes[1].set_xlabel("Epoch"); axes[1].set_ylabel("Accuracy")
axes[1].set_title("Accuracy"); axes[1].legend()

fig.suptitle("Figure 2. High-capacity probe training and validation curves", y=1.03)
plt.show()
plt.close(fig)

Interpretation of RQ2. Training loss continues towards zero and validation loss reaches its minimum at epoch 7 before rising. Evaluation of the final weights gives training accuracy 0.9729 and validation accuracy 0.8114, a gap of 0.1615. The network has enough capacity to fit the development data closely, and later epochs reduce validation performance. This supplies the evidence for early stopping in the candidate experiments.

2.7 Step 7: Regularising the model and tuning hyperparameters

Four controlled phases examine capacity, Dropout, vocabulary size and class weighting. The first three screen one factor at a time on a fixed split and seed. Class weighting is then evaluated across all five training seeds because it directly addresses the selected macro F1 objective. The two strongest weighting schemes are carried into the repeated-split study.

2.7.1 Phase 1: Capacity

Five architectures span narrow to wide and shallow to deep, including a deliberate bottleneck whose second layer has fewer units than the number of classes. All use no Dropout and the main vocabulary, with early stopping on validation loss.

# Phase 1 capacity configurations. The small_16 baseline record is reused.
phase1_configs = [
    ("small_16", [16]),
    ("reference_64_64", [64, 64]),
    ("wide_128_128", [128, 128]),
    ("deep_64_64_64", [64, 64, 64]),
    ("bottleneck_64_4", [64, 4]),
]

phase1_records = [baseline_record]
phase1_histories = {"small_16": baseline_hist}
for name, units in phase1_configs:
    if name == "small_16":
        continue
    rec, hist = counted_run(name, units, 0.0, VOCAB_MAIN, RANDOM_STATE,
                            features_main, use_early_stopping=True, keep_history=True)
    phase1_records.append(rec)
    phase1_histories[name] = hist

phase1_df = pd.DataFrame(phase1_records)
phase1_view = phase1_df[["name", "hidden_units", "params", "best_epoch",
                         "val_loss", "val_acc", "val_macro_f1", "acc_gap",
                         "seconds"]]
phase1_view

	name	hidden_units	params	best_epoch	val_loss	val_acc	val_macro_f1	acc_gap	seconds
0	small_16	[16]	160798	30	0.8871	0.8058	0.4855	0.1420	3.4
1	reference_64_64	[64, 64]	647214	11	0.8899	0.8075	0.4867	0.1383	3.0
2	wide_128_128	[128, 128]	1302574	9	0.8577	0.8141	0.5298	0.1431	3.5
3	deep_64_64_64	[64, 64, 64]	651374	10	1.0042	0.7963	0.4071	0.1381	2.9
4	bottleneck_64_4	[64, 4]	640554	21	1.2937	0.6956	0.1343	0.0870	4.6

def select_best(df):
    # Apply the declared rule: highest macro F1, ties within TIE_THRESHOLD
    # resolved by lower validation loss then fewer parameters.
    top = df.sort_values("val_macro_f1", ascending=False).iloc[0]
    contenders = df[df["val_macro_f1"] >= top["val_macro_f1"] - TIE_THRESHOLD]
    contenders = contenders.sort_values(["val_loss", "params"],
                                        ascending=[True, True])
    return contenders.iloc[0]


phase1_best = select_best(phase1_df)
best_units = parse_units(phase1_best["hidden_units"])
top_two = phase1_df["val_macro_f1"].nlargest(2)
phase1_margin = round(float(top_two.iloc[0] - top_two.iloc[1]), 4)
print(f"Phase 1 selection: {phase1_best['name']} "
      f"(macro F1 {phase1_best['val_macro_f1']}, params {phase1_best['params']})")
print(f"Margin over the runner-up: {phase1_margin}")

Phase 1 selection: wide_128_128 (macro F1 0.5298, params 1302574)
Margin over the runner-up: 0.0431

Interpretation. The four-unit bottleneck performs poorly, which is consistent with excessive compression before a 46-class output. The 128-128 model gives the highest validation macro F1, 0.5298, and the lowest validation loss among the useful candidates. Its 0.0431 macro F1 margin over the 64-64 model comes from one seed and split, so it is treated as a screening result.

2.7.2 Phase 2: Dropout

Dropout rates of 0.0, 0.2, 0.35 and 0.5 are compared on the selected capacity. Training and validation metrics are both evaluated after training with Dropout disabled, making their generalisation gap directly comparable.

# Phase 2 varies Dropout on the Phase 1 winner, all else fixed.
dropout_rates = [0.0, 0.2, 0.35, 0.5]
phase2_records, phase2_histories = [], {}
for rate in dropout_rates:
    name = f"drop_{rate}"
    rec, hist = counted_run(name, best_units, rate, VOCAB_MAIN, RANDOM_STATE,
                            features_main, use_early_stopping=True, keep_history=True)
    phase2_records.append(rec)
    phase2_histories[rate] = hist

phase2_df = pd.DataFrame(phase2_records)
phase2_view = phase2_df[["name", "dropout", "params", "best_epoch", "val_loss",
                         "val_acc", "val_macro_f1", "acc_gap", "seconds"]]
phase2_view

	name	dropout	params	best_epoch	val_loss	val_acc	val_macro_f1	acc_gap	seconds
0	drop_0.0	0.00	1302574	9	0.8577	0.8141	0.5298	0.1431	3.5
1	drop_0.2	0.20	1302574	9	0.8672	0.8158	0.4898	0.1291	3.7
2	drop_0.35	0.35	1302574	11	0.8943	0.8164	0.4577	0.1257	4.3
3	drop_0.5	0.50	1302574	14	0.9322	0.8058	0.3792	0.1214	4.8

phase2_best = select_best(phase2_df)
best_dropout = float(phase2_best["dropout"])
print(f"Phase 2 selection: dropout {best_dropout} "
      f"(macro F1 {phase2_best['val_macro_f1']}, gap {phase2_best['acc_gap']})")

Phase 2 selection: dropout 0.0 (macro F1 0.5298, gap 0.1431)

# Figure 3: Dropout effect on validation macro F1 and the accuracy gap.
fig, ax1 = plt.subplots(figsize=(7, 3.6))
ax1.plot(phase2_df["dropout"], phase2_df["val_macro_f1"], marker="o",
         color=PALETTE[0], label="Validation macro F1")
ax1.set_xlabel("Dropout rate"); ax1.set_ylabel("Validation macro F1", color=PALETTE[0])
ax1.tick_params(axis="y", labelcolor=PALETTE[0])

ax2 = ax1.twinx()
ax2.plot(phase2_df["dropout"], phase2_df["acc_gap"], marker="s",
         color=PALETTE[3], label="Accuracy gap")
ax2.set_ylabel("Train minus validation accuracy", color=PALETTE[3])
ax2.tick_params(axis="y", labelcolor=PALETTE[3])
ax2.grid(False)

fig.suptitle("Figure 3. Effect of Dropout on validation macro F1 and generalisation gap")
plt.show()
plt.close(fig)

Interpretation of RQ3. Post-training evaluation places both partitions in inference mode. The accuracy gap falls from 0.1431 at Dropout 0.0 to 0.1214 at rate 0.5, showing a modest reduction in the fit difference. Validation macro F1 falls at every tested rate, from 0.5298 to 0.4898, 0.4577 and 0.3792. Validation loss also rises from 0.8577 to 0.9322. Dropout therefore weakens the primary measure in this configuration, and rate 0.0 is selected. This result does not exclude an interaction between Dropout and another network width.

2.7.3 Phase 3: Vocabulary size

The selected architecture and Dropout setting are compared using Reuters loaded separately at 5,000 and 10,000 words. The loader seed and labels remain identical, while Keras applies the correct out-of-vocabulary handling for each vocabulary limit.

# Phase 3 varies vocabulary size with matching Keras representations.
x_dev_s, x_val_s = prepare_dev_val(VOCAB_SMALL)
features_small = (x_dev_s, y_dev, x_val_s, y_val, y_val_int)

phase3_records = []
for vocab, feats in [(VOCAB_SMALL, features_small), (VOCAB_MAIN, features_main)]:
    name = f"vocab_{vocab}"
    rec, _ = counted_run(
        name,
        best_units,
        best_dropout,
        vocab,
        RANDOM_STATE,
        feats,
        use_early_stopping=True,
        keep_history=False,
    )
    phase3_records.append(rec)

phase3_df = pd.DataFrame(phase3_records)
phase3_view = phase3_df[[
    "name",
    "vocab",
    "params",
    "best_epoch",
    "val_loss",
    "val_acc",
    "val_macro_f1",
    "acc_gap",
    "seconds",
]]
phase3_view

	name	vocab	params	best_epoch	val_loss	val_acc	val_macro_f1	acc_gap	seconds
0	vocab_5000	5000	662574	10	0.8594	0.8119	0.5359	0.1437	2.4
1	vocab_10000	10000	1302574	9	0.8577	0.8141	0.5298	0.1431	3.5

phase3_best = select_best(phase3_df)
best_vocab = int(phase3_best["vocab"])
print(f"Phase 3 selection: vocabulary {best_vocab} "
      f"(macro F1 {phase3_best['val_macro_f1']})")

Phase 3 selection: vocabulary 5000 (macro F1 0.5359)

Interpretation of RQ4. The 5,000-word representation reaches validation macro F1 0.5359, compared with 0.5298 for 10,000 words. The difference is 0.0061 on one seed and split. The smaller representation also reduces the parameter count from 1,302,574 to 662,574. The declared rule selects 5,000 words on the primary measure. The result is treated as a screening decision because it does not establish a stable difference between vocabulary sizes.

2.7.4 Phase 4: Class weighting and training-seed stability

Three training objectives are compared on the selected architecture and vocabulary: unweighted cross-entropy, square-root balanced weights and fully balanced inverse-frequency weights. Weights are calculated from the development labels used by each fit and normalised to a mean sample weight of one. Every scheme is repeated across five seeds. This phase addresses the imbalance directly and avoids selecting a weighting scheme from one stochastic run.

# Inspect the range of weights before fitting the three schemes.
weight_schemes = ["none", "sqrt_balanced", "balanced"]


def feature_bundle_for_vocab(vocab):
    if vocab == VOCAB_MAIN:
        return features_main
    if vocab == VOCAB_SMALL:
        return features_small
    raise ValueError(f"Unsupported vocabulary size: {vocab}")


weight_rows = []
for scheme in weight_schemes:
    weights = make_class_weights(y_dev_int, scheme)
    weight_rows.append({
        "scheme": scheme,
        "minimum_weight": 1.0 if weights is None else min(weights.values()),
        "maximum_weight": 1.0 if weights is None else max(weights.values()),
    })
pd.DataFrame(weight_rows).round(3)

	scheme	minimum_weight	maximum_weight
0	none	1.000	1.000
1	sqrt_balanced	0.357	6.342
2	balanced	0.062	19.524

# Repeat every class-weighting scheme across five training seeds.
stability_rows = []
selected_features = feature_bundle_for_vocab(best_vocab)
for scheme in weight_schemes:
    for seed in SEEDS:
        rec, _ = counted_run(
            f"weight_{scheme}",
            best_units,
            best_dropout,
            best_vocab,
            seed,
            selected_features,
            class_weight_scheme=scheme,
            use_early_stopping=True,
            keep_history=False,
        )
        stability_rows.append(rec)

stability_df = pd.DataFrame(stability_rows)
stability_df[[
    "name",
    "class_weight",
    "seed",
    "best_epoch",
    "val_loss",
    "val_acc",
    "val_macro_f1",
    "val_weighted_f1",
]]

	name	class_weight	seed	best_epoch	val_loss	val_acc	val_macro_f1	val_weighted_f1
0	weight_none	none	7	8	0.9001	0.8041	0.4616	0.7847
1	weight_none	none	42	10	0.8594	0.8119	0.5359	0.7973
2	weight_none	none	113	11	0.8679	0.8175	0.5912	0.8089
3	weight_none	none	2026	9	0.8596	0.8147	0.5129	0.7997
4	weight_none	none	31415	9	0.8579	0.8169	0.5068	0.8013
5	weight_sqrt_balanced	sqrt_balanced	7	8	0.8086	0.8247	0.6008	0.8177
6	weight_sqrt_balanced	sqrt_balanced	42	9	0.8042	0.8286	0.6351	0.8223
7	weight_sqrt_balanced	sqrt_balanced	113	9	0.8006	0.8280	0.6201	0.8211
8	weight_sqrt_balanced	sqrt_balanced	2026	9	0.8042	0.8303	0.6213	0.8224
9	weight_sqrt_balanced	sqrt_balanced	31415	8	0.8081	0.8297	0.6321	0.8242
10	weight_balanced	balanced	7	13	0.8686	0.8242	0.6181	0.8211
11	weight_balanced	balanced	42	12	0.8571	0.8236	0.6245	0.8187
12	weight_balanced	balanced	113	15	0.8733	0.8186	0.6144	0.8152
13	weight_balanced	balanced	2026	13	0.8400	0.8297	0.6546	0.8256
14	weight_balanced	balanced	31415	15	0.8842	0.8136	0.6118	0.8100

def summarise(group):
    return pd.Series({
        "hidden_units": group["hidden_units"].iloc[0],
        "dropout": group["dropout"].iloc[0],
        "vocab": group["vocab"].iloc[0],
        "class_weight": group["class_weight"].iloc[0],
        "params": int(group["params"].iloc[0]),
        "macro_f1_mean": round(group["val_macro_f1"].mean(), 4),
        "macro_f1_std": round(group["val_macro_f1"].std(ddof=1), 4),
        "acc_mean": round(group["val_acc"].mean(), 4),
        "acc_std": round(group["val_acc"].std(ddof=1), 4),
        "weighted_f1_mean": round(group["val_weighted_f1"].mean(), 4),
        "loss_mean": round(group["val_loss"].mean(), 4),
        "loss_std": round(group["val_loss"].std(ddof=1), 4),
        "median_best_epoch": int(np.median(group["best_epoch"])),
    })


stability_summary = (
    stability_df.groupby("name", sort=False)
    .apply(summarise, include_groups=False)
    .reset_index()
)
stability_summary

	name	hidden_units	dropout	vocab	class_weight	params	macro_f1_mean	macro_f1_std	acc_mean	acc_std	weighted_f1_mean	loss_mean	loss_std	median_best_epoch
0	weight_none	[128, 128]	0.0	5000	none	662574	0.5217	0.0473	0.8130	0.0054	0.7984	0.8690	0.0178	9
1	weight_sqrt_balanced	[128, 128]	0.0	5000	sqrt_balanced	662574	0.6219	0.0135	0.8283	0.0022	0.8215	0.8051	0.0033	9
2	weight_balanced	[128, 128]	0.0	5000	balanced	662574	0.6247	0.0174	0.8219	0.0061	0.8181	0.8646	0.0169	13

# Select from five-seed means, then retain the two strongest for split analysis.
ranked = stability_summary.sort_values("macro_f1_mean", ascending=False)
lead = ranked.iloc[0]
close = ranked[ranked["macro_f1_mean"] >= lead["macro_f1_mean"] - TIE_THRESHOLD]
close = close.sort_values(["loss_mean", "params"], ascending=[True, True])
final_row = close.iloc[0]
finalists = ranked.head(2).copy()

final_units = parse_units(final_row["hidden_units"])
final_dropout = float(final_row["dropout"])
final_vocab = int(final_row["vocab"])
final_class_weight = str(final_row["class_weight"])
final_epochs = int(final_row["median_best_epoch"])
print(
    f"Selected configuration: units={final_units}, dropout={final_dropout}, "
    f"vocab={final_vocab}, class_weight={final_class_weight}, "
    f"epochs={final_epochs}"
)

Selected configuration: units=[128, 128], dropout=0.0, vocab=5000, class_weight=sqrt_balanced, epochs=9

Interpretation of RQ5. Unweighted training gives mean validation macro F1 0.5217 with standard deviation 0.0473 across five seeds. Square-root balanced weights increase the mean to 0.6219 and reduce the standard deviation to 0.0135. Fully balanced weights give 0.6247 with standard deviation 0.0174. The last two means differ by 0.0028 and fall inside the declared tie threshold. Square-root weighting has lower mean validation loss, 0.8051 compared with 0.8646, and higher mean accuracy, 0.8283 compared with 0.8219, so it wins the tie-break. The improvement over unweighted training is 0.1002 in macro F1 and 0.0153 in accuracy. Weighting improves equal-weight class performance without sacrificing aggregate accuracy on the fixed validation split.

2.7.5 Sensitivity to the validation split

Training-seed repetition holds the data partition fixed. The two strongest weighting schemes are now evaluated over five fresh stratified 80:20 splits with the training seed held at 42. Class weights are recalculated from each development partition. The paired difference on each split shows whether the ranking survives changes in partition membership. The test set remains untouched.

SPLIT_SEEDS = SEEDS


def split_feature_bundle(x_all, seed):
    d_idx, v_idx = train_test_split(
        np.arange(len(train_labels)),
        test_size=0.20,
        random_state=seed,
        stratify=train_labels,
    )
    assert set(d_idx).isdisjoint(set(v_idx))
    return (
        x_all[d_idx],
        keras.utils.to_categorical(train_labels[d_idx], NUM_CLASSES),
        x_all[v_idx],
        keras.utils.to_categorical(train_labels[v_idx], NUM_CLASSES),
        train_labels[v_idx],
    )


split_rows = []
for vocab in sorted({int(v) for v in finalists["vocab"]}):
    training_sequences, _ = sequences_for_vocab(vocab)
    x_all_vocab = multi_hot_encode(training_sequences, vocab)
    for split_seed in SPLIT_SEEDS:
        bundle = split_feature_bundle(x_all_vocab, split_seed)
        for _, cfg in finalists[finalists["vocab"] == vocab].iterrows():
            rec, _ = counted_run(
                cfg["name"],
                parse_units(cfg["hidden_units"]),
                float(cfg["dropout"]),
                vocab,
                RANDOM_STATE,
                bundle,
                class_weight_scheme=str(cfg["class_weight"]),
                use_early_stopping=True,
                keep_history=False,
            )
            rec["split_seed"] = split_seed
            split_rows.append(rec)
        del bundle
    del x_all_vocab

split_df = pd.DataFrame(split_rows)
split_wide = split_df.pivot(
    index="split_seed", columns="name", values="val_macro_f1"
).round(4)
split_wide

name	weight_balanced	weight_sqrt_balanced
split_seed
7	0.6334	0.6124
42	0.6245	0.6351
113	0.6031	0.6186
2026	0.5840	0.5911
31415	0.6118	0.6040

split_summary = (
    split_df.groupby("name")
    .agg(
        macro_f1_mean=("val_macro_f1", "mean"),
        macro_f1_std=("val_macro_f1", "std"),
        acc_mean=("val_acc", "mean"),
        acc_std=("val_acc", "std"),
        loss_mean=("val_loss", "mean"),
        loss_std=("val_loss", "std"),
    )
    .round(4)
    .reset_index()
)

lead_name = str(final_row["name"])
other_name = [name for name in split_wide.columns if name != lead_name][0]
paired_diff = split_wide[lead_name] - split_wide[other_name]
split_range = float(split_wide[lead_name].max() - split_wide[lead_name].min())
seed_range = float(
    stability_df.loc[
        stability_df["name"] == lead_name, "val_macro_f1"
    ].max()
    - stability_df.loc[
        stability_df["name"] == lead_name, "val_macro_f1"
    ].min()
)

print(
    f"Range for {lead_name}: {split_range:.4f} across splits and "
    f"{seed_range:.4f} across training seeds"
)
print(
    f"Paired difference ({lead_name} minus {other_name}): "
    f"mean {paired_diff.mean():.4f}, standard deviation "
    f"{paired_diff.std():.4f}, favourable on "
    f"{int((paired_diff > 0).sum())} of {len(paired_diff)} splits"
)
split_summary

Range for weight_sqrt_balanced: 0.0440 across splits and 0.0343 across training seeds
Paired difference (weight_sqrt_balanced minus weight_balanced): mean 0.0009, standard deviation 0.0150, favourable on 3 of 5 splits

	name	macro_f1_mean	macro_f1_std	acc_mean	acc_std	loss_mean	loss_std
0	weight_balanced	0.6114	0.0192	0.8128	0.0106	0.9366	0.0474
1	weight_sqrt_balanced	0.6122	0.0164	0.8179	0.0070	0.8618	0.0362

Interpretation. Across five validation splits, square-root weighting gives mean macro F1 0.6122 with standard deviation 0.0164. Fully balanced weighting gives 0.6114 with standard deviation 0.0192. The paired square-root minus balanced difference is 0.0009 with standard deviation 0.0150, and square-root weighting leads on three of the five splits. The schemes cannot be separated on macro F1 at this resolution. Square-root weighting retains the lower validation loss and higher accuracy, so the declared tie-break supports the selected scheme. The comparison also shows that the large improvement over the unweighted mean is not dependent on one training seed.

2.7.6 Learning curve

The selected weighted model is trained on nested stratified fractions of the development partition. Each fraction is repeated over five seeds, and class weights are recalculated from the rows present in that subset. The fixed validation set permits direct comparison without accessing the test data.

def nested_fraction_indices(labels_int, fractions, seed):
    order = np.arange(len(labels_int))
    rng = np.random.default_rng(seed)
    per_class = {
        class_id: rng.permutation(order[labels_int == class_id])
        for class_id in range(NUM_CLASSES)
    }
    subsets = {}
    for fraction in fractions:
        chosen = []
        for class_id in range(NUM_CLASSES):
            take = max(1, int(round(len(per_class[class_id]) * fraction)))
            chosen.extend(per_class[class_id][:take])
        subsets[fraction] = np.array(sorted(chosen))
    return subsets


fractions = [0.25, 0.5, 0.75, 1.0]
final_feats = feature_bundle_for_vocab(final_vocab)
x_dev_final, y_dev_final = final_feats[0], final_feats[1]

learning_rows = []
for seed in SEEDS:
    subset_idx = nested_fraction_indices(y_dev_int, fractions, seed)
    for fraction in fractions:
        idx = subset_idx[fraction]
        feats = (
            x_dev_final[idx],
            y_dev_final[idx],
            final_feats[2],
            final_feats[3],
            final_feats[4],
        )
        rec, _ = counted_run(
            f"frac_{fraction}",
            final_units,
            final_dropout,
            final_vocab,
            seed,
            feats,
            class_weight_scheme=final_class_weight,
            use_early_stopping=True,
        )
        rec["fraction"] = fraction
        rec["dev_examples"] = len(idx)
        learning_rows.append(rec)

learning_df = pd.DataFrame(learning_rows)
learning_summary = (
    learning_df.groupby(["fraction", "dev_examples"])
    .agg(
        macro_f1_mean=("val_macro_f1", "mean"),
        macro_f1_std=("val_macro_f1", "std"),
        acc_mean=("val_acc", "mean"),
        acc_std=("val_acc", "std"),
        loss_mean=("val_loss", "mean"),
    )
    .round(4)
    .reset_index()
)
learning_summary["macro_f1_step"] = (
    learning_summary["macro_f1_mean"].diff().round(4)
)
learning_summary["acc_step"] = learning_summary["acc_mean"].diff().round(4)

learning_wide = learning_df.pivot(
    index="seed", columns="fraction", values="val_macro_f1"
).round(4)
print("Validation macro F1 by seed and fraction:")
print(learning_wide.to_string())

final_step = learning_wide[1.0] - learning_wide[0.75]
step_se = final_step.std(ddof=1) / np.sqrt(len(final_step))
print(f"\nFinal paired step: {final_step.round(4).tolist()}")
print(
    f"Mean {final_step.mean():.4f}, standard deviation "
    f"{final_step.std(ddof=1):.4f}, positive for "
    f"{int((final_step > 0).sum())} of {len(final_step)} seeds, "
    f"mean divided by standard error {final_step.mean() / step_se:.2f}"
)
learning_summary

Validation macro F1 by seed and fraction:
fraction    0.25    0.50    0.75    1.00
seed                                    
7         0.4293  0.5245  0.5365  0.6008
42        0.3722  0.5031  0.5711  0.6351
113       0.4171  0.5137  0.5617  0.6201
2026      0.3964  0.5093  0.5891  0.6213
31415     0.3823  0.5191  0.5691  0.6321

Final paired step: [0.0643, 0.064, 0.0584, 0.0322, 0.063]
Mean 0.0564, standard deviation 0.0137, positive for 5 of 5 seeds, mean divided by standard error 9.19

	fraction	dev_examples	macro_f1_mean	macro_f1_std	acc_mean	acc_std	loss_mean	macro_f1_step	acc_step
0	0.25	1797	0.3995	0.0237	0.7763	0.0060	1.0217	NaN	NaN
1	0.50	3597	0.5139	0.0083	0.8027	0.0056	0.9141	0.1144	0.0264
2	0.75	5388	0.5655	0.0191	0.8155	0.0033	0.8495	0.0516	0.0128
3	1.00	7185	0.6219	0.0135	0.8283	0.0022	0.8051	0.0564	0.0128

# Figure 4: learning curve of validation macro F1 against development size. The
# band is plus or minus one standard deviation over the five seeds, so the
# change between neighbouring fractions can be judged against seed variation.
fig, ax = plt.subplots(figsize=(7, 3.8))
sizes = learning_summary["dev_examples"]
mean_f1 = learning_summary["macro_f1_mean"]
sd_f1 = learning_summary["macro_f1_std"]
ax.plot(sizes, mean_f1, marker="o", color=PALETTE[0], label="Mean over five seeds")
ax.fill_between(sizes, mean_f1 - sd_f1, mean_f1 + sd_f1, color=PALETTE[0],
                alpha=0.20, label="Plus or minus one standard deviation")
ax.set_xlabel("Development examples used")
ax.set_ylabel("Validation macro F1")
ax.set_title("Figure 4. Learning curve for the selected configuration")
ax.legend(loc="lower right")
plt.show()
plt.close(fig)

Interpretation of RQ6. Mean validation macro F1 rises from 0.3995 at one quarter of the development data to 0.5139, 0.5655 and 0.6219 as further data are added. The final paired increase is 0.0564 with standard deviation 0.0137 and is positive for all five seeds. Accuracy also rises from 0.7763 to 0.8283, with smaller increments in the second half of the curve. These results provide consistent evidence that additional labelled data would improve the selected model, especially under the equal-class macro F1 measure. Extrapolation beyond the observed sample sizes remains uncertain.

3. Final model and untouched test evaluation

The architecture, class-weighting scheme and epoch count are now fixed. A fresh model is trained once on every supplied training newswire. Test probabilities are generated after this fit and no tuning follows.

# Record the complete final specification before test access.
final_arch = pd.DataFrame({
    "property": [
        "Hidden units",
        "Dropout rate",
        "Vocabulary size",
        "Class weighting",
        "Output layer",
        "Loss",
        "Optimizer",
        "Training epochs",
        "Epoch rule",
    ],
    "value": [
        str(final_units),
        final_dropout,
        final_vocab,
        final_class_weight,
        "Dense(46, softmax)",
        "categorical_crossentropy",
        "rmsprop",
        final_epochs,
        "Median best epoch across five training seeds",
    ],
})
final_arch

	property	value
0	Hidden units	[128, 128]
1	Dropout rate	0.0
2	Vocabulary size	5000
3	Class weighting	sqrt_balanced
4	Output layer	Dense(46, softmax)
5	Loss	categorical_crossentropy
6	Optimizer	rmsprop
7	Training epochs	9
8	Epoch rule	Median best epoch across five training seeds

# Train once on all supplied training examples with the selected weights.
final_train_sequences, final_test_sequences = sequences_for_vocab(final_vocab)
x_train_full = multi_hot_encode(final_train_sequences, final_vocab)
x_test_full = multi_hot_encode(final_test_sequences, final_vocab)
y_train_full = keras.utils.to_categorical(train_labels, NUM_CLASSES)
final_weights = make_class_weights(train_labels, final_class_weight)

assert x_train_full.shape[1] == final_vocab
assert np.isfinite(x_train_full).all()

keras.backend.clear_session()
set_all_seeds(RANDOM_STATE)
final_model = build_model(
    final_vocab, final_units, final_dropout, NUM_CLASSES, RANDOM_STATE
)
final_model.fit(
    x_train_full,
    y_train_full,
    epochs=final_epochs,
    batch_size=BATCH_SIZE,
    class_weight=final_weights,
    verbose=0,
)
FIT_COUNTER["n"] += 1
print(f"Final model trained for {final_epochs} epochs on {len(x_train_full)} examples.")

Final model trained for 9 epochs on 8982 examples.

# Single test evaluation. Probability rows are checked to sum to one before use.
test_proba = final_model.predict(x_test_full, batch_size=BATCH_SIZE, verbose=0)
assert np.allclose(test_proba.sum(axis=1), 1.0, atol=1e-4)
test_pred = test_proba.argmax(axis=1)

# Full diagnostic suite on the untouched test split.
suite = full_metric_suite(test_labels, test_pred, test_proba)

# The four required headline measures, retained by name for the abstract.
test_loss = suite["Cross-entropy"]
test_acc = suite["Accuracy"]
test_macro = suite["Macro F1"]
test_weighted = suite["Weighted F1"]
test_bal_acc = suite["Balanced accuracy"]
test_kappa = suite["Cohen's kappa"]
test_mcc = suite["Matthews corrcoef"]
test_top3 = suite["Top-3 accuracy"]
test_top5 = suite["Top-5 accuracy"]
test_ece = suite["Expected calibration error"]
test_auc = [v for k, v in suite.items() if k.startswith("Macro OVR")][0]

test_metrics = pd.DataFrame(
    {"metric": list(suite.keys()),
     "value": [round(v, 4) for v in suite.values()]}
)
test_metrics

	metric	value
0	Cross-entropy	0.9068
1	Accuracy	0.8072
2	Balanced accuracy	0.6054
3	Top-3 accuracy	0.9159
4	Top-5 accuracy	0.9403
5	Macro precision	0.7060
6	Macro recall	0.6054
7	Macro F1	0.6280
8	Weighted precision	0.8109
9	Weighted recall	0.8072
10	Weighted F1	0.8034
11	Cohen's kappa	0.7626
12	Matthews corrcoef	0.7629
13	Macro OVR ROC AUC (46 classes)	0.9525
14	Expected calibration error	0.0566

# Percentile bootstrap intervals resample fixed prediction-label pairs.
def bootstrap_intervals(y_true, y_pred, n_resamples=2000, seed=RANDOM_STATE):
    rng = np.random.default_rng(seed)
    n = len(y_true)
    accs = np.empty(n_resamples)
    macros = np.empty(n_resamples)
    bals = np.empty(n_resamples)
    for draw in range(n_resamples):
        sample = rng.integers(0, n, n)
        observed = y_true[sample]
        predicted = y_pred[sample]
        accs[draw] = accuracy_score(observed, predicted)
        macros[draw] = f1_score(
            observed,
            predicted,
            average="macro",
            labels=LABELS,
            zero_division=0,
        )
        bals[draw] = recall_score(
            observed,
            predicted,
            average="macro",
            labels=LABELS,
            zero_division=0,
        )

    def interval(values):
        return (
            round(float(np.percentile(values, 2.5)), 4),
            round(float(np.percentile(values, 97.5)), 4),
        )

    return interval(accs), interval(macros), interval(bals)


acc_ci, macro_ci, bal_ci = bootstrap_intervals(test_labels, test_pred)
ci_table = pd.DataFrame({
    "metric": ["Accuracy", "Macro F1", "Balanced accuracy"],
    "point_estimate": [
        round(test_acc, 4),
        round(test_macro, 4),
        round(test_bal_acc, 4),
    ],
    "ci_lower_95": [acc_ci[0], macro_ci[0], bal_ci[0]],
    "ci_upper_95": [acc_ci[1], macro_ci[1], bal_ci[1]],
})
ci_table

	metric	point_estimate	ci_lower_95	ci_upper_95
0	Accuracy	0.8072	0.7912	0.8241
1	Macro F1	0.6280	0.5484	0.6506
2	Balanced accuracy	0.6054	0.5390	0.6449

# A stratified bootstrap conditions on the observed support of each class.
def stratified_bootstrap_intervals(
    y_true, y_pred, n_resamples=2000, seed=RANDOM_STATE
):
    rng = np.random.default_rng(seed)
    index_by_class = [np.flatnonzero(y_true == class_id) for class_id in LABELS]
    accs = np.empty(n_resamples)
    macros = np.empty(n_resamples)
    for draw in range(n_resamples):
        sample = np.concatenate([
            rng.choice(indices, len(indices), replace=True)
            for indices in index_by_class
            if len(indices) > 0
        ])
        accs[draw] = accuracy_score(y_true[sample], y_pred[sample])
        macros[draw] = f1_score(
            y_true[sample],
            y_pred[sample],
            average="macro",
            labels=LABELS,
            zero_division=0,
        )

    def interval(values):
        return (
            round(float(np.percentile(values, 2.5)), 4),
            round(float(np.percentile(values, 97.5)), 4),
        )

    return interval(accs), interval(macros)


strat_acc_ci, strat_macro_ci = stratified_bootstrap_intervals(
    test_labels, test_pred
)
interval_comparison = pd.DataFrame({
    "metric": ["Accuracy", "Accuracy", "Macro F1", "Macro F1"],
    "scheme": ["Unstratified", "Class-stratified"] * 2,
    "point_estimate": [round(test_acc, 4)] * 2 + [round(test_macro, 4)] * 2,
    "ci_lower_95": [
        acc_ci[0],
        strat_acc_ci[0],
        macro_ci[0],
        strat_macro_ci[0],
    ],
    "ci_upper_95": [
        acc_ci[1],
        strat_acc_ci[1],
        macro_ci[1],
        strat_macro_ci[1],
    ],
})
interval_comparison

	metric	scheme	point_estimate	ci_lower_95	ci_upper_95
0	Accuracy	Unstratified	0.8072	0.7912	0.8241
1	Accuracy	Class-stratified	0.8072	0.7921	0.8210
2	Macro F1	Unstratified	0.6280	0.5484	0.6506
3	Macro F1	Class-stratified	0.6280	0.5763	0.6619

Interpretation. The selected model achieves test accuracy 0.8072 with an unstratified 95 percent bootstrap interval of [0.7912, 0.8241]. Macro F1 is 0.6280 with interval [0.5484, 0.6506], weighted F1 is 0.8034 and balanced accuracy is 0.6054. The difference between macro and weighted F1 shows that infrequent topics remain harder than the classes that dominate the corpus. Top-3 accuracy is 0.9159 and expected calibration error is 0.0566.

The unstratified bootstrap resamples the observed mixture of classes. Its macro F1 interval is affected when rare classes disappear from a resample. The class-stratified interval, [0.5763, 0.6619], conditions on the observed support of every class. These intervals answer different sampling questions and neither includes retraining variability.

Chollet's comparable 64-64 Dense example reports test accuracy of approximately 0.797 (Chollet, 2017, Chapter 3). The present accuracy is approximately 0.010 higher. Differences in class weighting, software version and training protocol prevent a direct claim of model superiority. The main experimental contribution is the measured improvement in validation macro F1 produced by class weighting.

4. Interpretation and error analysis

The final predictions are examined by class support, frequent confusion pairs and confidence. Integer identifiers are retained because the Keras dataset does not provide authoritative topic names.

# Per-class precision, recall, F1, and support on the test split.
report_dict = classification_report(
    test_labels, test_pred, labels=list(range(NUM_CLASSES)),
    output_dict=True, zero_division=0,
)
per_class = pd.DataFrame(
    [
        {
            "class": c,
            "precision": round(report_dict[str(c)]["precision"], 3),
            "recall": round(report_dict[str(c)]["recall"], 3),
            "f1": round(report_dict[str(c)]["f1-score"], 3),
            "support": int(report_dict[str(c)]["support"]),
        }
        for c in range(NUM_CLASSES)
    ]
)
per_class.sort_values("support", ascending=False).head(10)

	class	precision	recall	f1	support
3	3	0.940	0.924	0.932	813
4	4	0.847	0.878	0.862	474
19	19	0.624	0.737	0.676	133
1	1	0.738	0.752	0.745	105
16	16	0.686	0.818	0.747	99
11	11	0.574	0.795	0.667	83
20	20	0.618	0.486	0.544	70
8	8	0.730	0.711	0.720	38
13	13	0.600	0.568	0.583	37
25	25	0.821	0.742	0.780	31

# Raw rankings are shown with support so small classes are not overinterpreted.
scored = per_class[per_class["support"] > 0].copy()
strongest = scored.sort_values(
    ["f1", "support"], ascending=[False, False]
).head(5)
weakest = scored.sort_values(
    ["f1", "support"], ascending=[True, False]
).head(5)
extremes = pd.concat([
    strongest.assign(group="strongest"),
    weakest.assign(group="weakest"),
])
extremes[["group", "class", "precision", "recall", "f1", "support"]]

	group	class	precision	recall	f1	support
45	strongest	45	1.000	1.000	1.000	1
32	strongest	32	1.000	0.900	0.947	10
3	strongest	3	0.940	0.924	0.932	813
10	strongest	10	0.900	0.900	0.900	30
44	strongest	44	1.000	0.800	0.889	5
22	weakest	22	0.000	0.000	0.000	7
5	weakest	5	0.333	0.200	0.250	5
15	weakest	15	0.500	0.222	0.308	9
14	weakest	14	0.250	0.500	0.333	2
12	weakest	12	0.500	0.308	0.381	13

# A support threshold limits the effect of one error on class recall to 0.05.
MIN_SUPPORT = 20
well_supported = per_class[per_class["support"] >= MIN_SUPPORT].copy()
qualified = pd.concat([
    well_supported.sort_values("f1", ascending=False)
    .head(5)
    .assign(group="strongest"),
    well_supported.sort_values("f1", ascending=True)
    .head(5)
    .assign(group="weakest"),
])

scored_classes = per_class[per_class["support"] > 0]
rho_all = scored_classes[["support", "f1"]].corr(
    method="spearman"
).iloc[0, 1]
rho_qualified = well_supported[["support", "f1"]].corr(
    method="spearman"
).iloc[0, 1]

print(
    f"{len(well_supported)} of {NUM_CLASSES} classes reach support "
    f"{MIN_SUPPORT}; they contain {int(well_supported['support'].sum())} "
    f"of {len(test_labels)} test cases."
)
print(
    f"Spearman support-F1 correlation: {rho_all:.3f} for all scored "
    f"classes and {rho_qualified:.3f} for supported classes."
)
qualified[["group", "class", "precision", "recall", "f1", "support"]]

15 of 46 classes reach support 20; they contain 2005 of 2246 test cases.
Spearman support-F1 correlation: 0.264 for all scored classes and 0.291 for supported classes.

	group	class	precision	recall	f1	support
3	strongest	3	0.940	0.924	0.932	813
10	strongest	10	0.900	0.900	0.900	30
4	strongest	4	0.847	0.878	0.862	474
25	strongest	25	0.821	0.742	0.780	31
9	strongest	9	0.818	0.720	0.766	25
20	weakest	20	0.618	0.486	0.544	70
13	weakest	13	0.600	0.568	0.583	37
2	weakest	2	0.571	0.800	0.667	20
11	weakest	11	0.574	0.795	0.667	83
18	weakest	18	0.750	0.600	0.667	20

# Most frequent off-diagonal confusion pairs across all classes.
cm = confusion_matrix(test_labels, test_pred, labels=list(range(NUM_CLASSES)))
pairs = []
for i in range(NUM_CLASSES):
    for j in range(NUM_CLASSES):
        if i != j and cm[i, j] > 0:
            pairs.append({"true_class": i, "predicted_class": j, "count": int(cm[i, j])})
confusion_pairs = (pd.DataFrame(pairs)
                   .sort_values("count", ascending=False)
                   .head(10)
                   .reset_index(drop=True))
confusion_pairs

	true_class	predicted_class	count
0	3	4	35
1	4	3	20
2	20	19	15
3	19	11	10
4	13	16	9
5	4	16	8
6	4	19	8
7	19	20	7
8	20	3	6
9	16	3	6

# Figure 5: row-normalised confusion matrix for the most supported classes,
# selected by a documented rule: the twelve classes with the largest test
# support. The matrix is computed over all 46 classes first, rows are then
# normalised by the complete row total, and a final column collects predictions
# that fall outside the twelve. Restricting the matrix before normalising would
# drop those predictions and inflate the diagonal, so every row here sums to one
# and no error is hidden by the restriction.
cm_full = confusion_matrix(test_labels, test_pred, labels=LABELS)
top_support = sorted(per_class.sort_values("support", ascending=False)
                              .head(12)["class"].tolist())

row_totals = cm_full[top_support].sum(axis=1, keepdims=True).clip(min=1)
block = cm_full[np.ix_(top_support, top_support)]
outside = cm_full[top_support].sum(axis=1, keepdims=True) - block.sum(axis=1, keepdims=True)
cm_norm = np.hstack([block, outside]) / row_totals
assert np.allclose(cm_norm.sum(axis=1), 1.0)

fig, ax = plt.subplots(figsize=(8.4, 6))
sns.heatmap(cm_norm, annot=True, fmt=".2f", cmap="Blues", cbar=True,
            xticklabels=[str(c) for c in top_support] + ["other"],
            yticklabels=top_support, ax=ax, annot_kws={"size": 8})
ax.set_xlabel("Predicted class"); ax.set_ylabel("True class")
ax.set_title("Figure 5. Row-normalised confusion matrix for the twelve most supported classes")
plt.show()
plt.close(fig)

# Share of each row that leaves the displayed block, which the restricted
# matrix would otherwise conceal.
outside_share = pd.DataFrame({
    "class": top_support,
    "support": cm_full[top_support].sum(axis=1),
    "outside_share": np.round((outside / row_totals).ravel(), 3),
})
outside_share

	class	support	outside_share
0	1	105	0.057
1	3	813	0.007
2	4	474	0.017
3	8	38	0.079
4	10	30	0.067
5	11	83	0.060
6	13	37	0.027
7	16	99	0.010
8	19	133	0.030
9	20	70	0.029
10	21	27	0.111
11	25	31	0.032

# Figure 6: predicted confidence for correct and incorrect test predictions.
confidence = test_proba.max(axis=1)
correct_mask = test_pred == test_labels
conf_df = pd.DataFrame({
    "confidence": confidence,
    "outcome": np.where(correct_mask, "correct", "incorrect"),
})

fig, ax = plt.subplots(figsize=(7.5, 3.6))
sns.boxplot(data=conf_df, x="outcome", y="confidence",
            hue="outcome", palette=[PALETTE[2], PALETTE[3]], legend=False, ax=ax)
ax.set_xlabel("Prediction outcome"); ax.set_ylabel("Predicted class probability")
ax.set_title("Figure 6. Prediction confidence for correct and incorrect predictions")
plt.show()
plt.close(fig)

conf_summary = conf_df.groupby("outcome")["confidence"].agg(["mean", "median"]).round(3)
conf_summary

	mean	median
outcome
correct	0.852	0.918
incorrect	0.630	0.647

# Reliability is plotted at the observed mean confidence in each bin.
ece_value, reliability = expected_calibration_error(
    test_proba.max(axis=1),
    (test_pred == test_labels).astype(float),
)
plotted = reliability.dropna(subset=["accuracy"])

fig, ax = plt.subplots(figsize=(5.6, 4.6))
ax.plot(
    [0, 1],
    [0, 1],
    color="grey",
    linestyle="--",
    linewidth=1,
    label="Perfect calibration",
)
ax.plot(
    plotted["confidence"],
    plotted["accuracy"],
    marker="o",
    color=PALETTE[0],
    label="Observed accuracy",
)
ax.bar(
    plotted["confidence"],
    plotted["count"] / plotted["count"].sum(),
    width=0.07,
    color=PALETTE[0],
    alpha=0.25,
    label="Share of predictions",
)
ax.set_xlabel("Mean predicted confidence")
ax.set_ylabel("Empirical accuracy")
ax.set_title(
    f"Figure 7. Reliability diagram (expected calibration error {ece_value:.3f})"
)
ax.set_xlim(0, 1)
ax.set_ylim(0, 1)
ax.legend(loc="upper left")
plt.show()
plt.close(fig)
reliability.round(3)

	bin_low	bin_high	count	accuracy	confidence
0	0.0	0.1	4	0.750	0.085
1	0.1	0.2	38	0.289	0.158
2	0.2	0.3	56	0.429	0.255
3	0.3	0.4	77	0.416	0.345
4	0.4	0.5	107	0.533	0.455
5	0.5	0.6	110	0.564	0.550
6	0.6	0.7	134	0.701	0.656
7	0.7	0.8	189	0.825	0.750
8	0.8	0.9	418	0.878	0.858
9	0.9	1.0	1113	0.905	0.964

Interpretation. Raw F1 rankings are unreliable for the smallest topics. Class 45 scores 1.000 on one case and class 44 scores 0.889 on five cases. Among the 15 classes with at least 20 test examples, class 3 has the highest F1 at 0.932 on 813 cases. Class 20 has the lowest at 0.544 on 70 cases. Support and F1 have Spearman correlation 0.264 across all scored classes and 0.291 within the supported subset, so support explains only part of the variation.

The largest numeric confusion is class 3 predicted as class 4 in 35 cases, followed by class 4 predicted as class 3 in 20 cases. No semantic interpretation is attached because authoritative topic names are unavailable. Correct predictions have median confidence 0.918, compared with 0.647 for errors. Expected calibration error is 0.0566. In the highest confidence interval, mean confidence is 0.964 and empirical accuracy is 0.905, which shows overconfidence among the most certain predictions.

4.1 Representation collision diagnostic

Multi-hot vectors lose order and frequency. Exact duplicates with different labels are indistinguishable to any classifier using this representation. The next calculation measures the empirical minimum error forced by these exact collisions. It does not measure ambiguity between similar, non-identical vectors.

# Size of the error that the representation forces. Newswires sharing a
# multi-hot vector are indistinguishable to any model built on this input, so
# the best available rule is to predict the majority label within each group of
# identical vectors. What that rule still gets wrong is the floor the
# representation imposes, and it can be compared with the observed error rate.
def representation_error_floor(sequences, labels, dimension):
    groups = defaultdict(list)
    for sequence, label in zip(sequences, labels):
        groups[frozenset(t for t in sequence if 0 <= t < dimension)].append(int(label))
    forced = sum(len(v) - Counter(v).most_common(1)[0][1] for v in groups.values())
    return forced, len(groups)


floor_rows = []
for scope, seqs, labs in [("Test split", test_data, test_labels),
                          ("Training split", train_data, train_labels)]:
    forced, distinct = representation_error_floor(seqs, labs, final_vocab)
    floor_rows.append({"scope": scope, "examples": len(labs),
                       "distinct_vectors": distinct, "forced_errors": forced,
                       "error_floor": round(forced / len(labs), 4)})

floor_table = pd.DataFrame(floor_rows)
test_error_rate = 1 - test_acc
test_floor = float(floor_table.loc[floor_table["scope"] == "Test split",
                                   "error_floor"].iloc[0])
print(f"Observed test error rate {test_error_rate:.4f}; the representation "
      f"forces {test_floor:.4f} of it, a share of {test_floor / test_error_rate:.1%}")
floor_table

Observed test error rate 0.1928; the representation forces 0.0085 of it, a share of 4.4%

	scope	examples	distinct_vectors	forced_errors	error_floor
0	Test split	2246	2213	19	0.0085
1	Training split	8982	8444	283	0.0315

Interpretation. Nineteen test newswires belong to an exact multi-hot collision whose labels disagree. This gives an empirical forced-error rate of 0.0085, compared with the model's observed error rate of 0.1928. Exact collisions account for 4.4 percent of the observed error. This calculation rules out exact vector duplication as the principal explanation for model error. It does not measure information lost through word order, term frequency or similarity between non-identical vectors.

5. Limitations and reproducibility

• Multi-hot encoding discards word order and frequency. The collision diagnostic measures only identical vectors and does not establish a performance ceiling.
• Rare classes yield unstable per-class metrics. Macro F1 remains more variable than accuracy even after class weighting.
• Capacity, Dropout and vocabulary are screened sequentially, so interactions between them are not estimated.
• Five seeds and five splits provide useful sensitivity checks but remain small samples of training and partition variability.
• The ordinary bootstrap resamples the empirical class mixture. The class-stratified version conditions on the observed class counts. Neither interval includes model retraining or model-selection uncertainty (Efron and Tibshirani, 1993).
• Reuters is historical, so performance may not transfer to current news language.
• Results apply to Dense networks and the tested representation. They do not define an upper bound for Reuters classification.

Python and library versions, hardware, seeds and deterministic settings are recorded above. A clean-kernel execution is required before the HTML export is submitted.

6. Conclusion

The study follows the seven-stage deep learning workflow and keeps the supplied test partition outside model development. A small Dense model clearly exceeds chance and majority-class baselines. The high-capacity probe exposes overfitting after epoch 7. The controlled screens select two 128-unit hidden layers, no Dropout and a 5,000-word vocabulary. Vocabulary and capacity results come from one seed and split and remain screening decisions.

Class weighting provides the most substantial result. Across five seeds, square-root balanced weighting raises mean validation macro F1 from 0.5217 to 0.6219 and mean accuracy from 0.8130 to 0.8283. Its macro F1 is practically tied with full balancing, and its lower validation loss resolves the selection. The repeated-split comparison confirms that the two weighting schemes remain close. The final model uses square-root weights and nine epochs. It reaches test accuracy 0.8072, macro F1 0.6280 and weighted F1 0.8034.

The learning curve rises through the full development partition in all five paired runs, supporting the collection of additional labelled newswires. Future work within the permitted scope should allocate more examples to low-support topics and repeat the capacity and vocabulary screens across several seeds. The result is specific to the Reuters corpus, its multi-hot representation and the permitted Dense model family.

References

Abadi, M. et al. (2016) 'TensorFlow: a system for large-scale machine learning', in Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation. Savannah, GA: USENIX Association, pp. 265-283.

Boughorbel, S., Jarray, F. and El-Anbari, M. (2017) 'Optimal classifier for imbalanced data using Matthews correlation coefficient metric', PLoS ONE, 12(6), e0177678.

Chollet, F. (2017) Deep Learning with Python. 1st edn. Shelter Island, NY: Manning Publications.

Efron, B. and Tibshirani, R.J. (1993) An Introduction to the Bootstrap. New York: Chapman and Hall.

Fawcett, T. (2006) 'An introduction to ROC analysis', Pattern Recognition Letters, 27(8), pp. 861-874.

Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q. (2017) 'On calibration of modern neural networks', in Proceedings of the 34th International Conference on Machine Learning, 70, pp. 1321-1330.

He, H. and Garcia, E.A. (2009) 'Learning from imbalanced data', IEEE Transactions on Knowledge and Data Engineering, 21(9), pp. 1263-1284.

Hinton, G., Srivastava, N. and Swersky, K. (2012) Neural networks for machine learning, lecture 6a: overview of mini-batch gradient descent. University of Toronto.

Keras (2024) Reuters newswire classification dataset. Available at: https://keras.io/api/datasets/reuters/ (Accessed: 25 August 2026).

Pedregosa, F. et al. (2011) 'Scikit-learn: machine learning in Python', Journal of Machine Learning Research, 12, pp. 2825-2830.

Srivastava, N. et al. (2014) 'Dropout: a simple way to prevent neural networks from overfitting', Journal of Machine Learning Research, 15, pp. 1929-1958.

TensorFlow (2024) tf.keras.layers.Dense. Available at: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense (Accessed: 25 August 2026).

Appendix A. Code provenance and use of generative AI

The vectorisation idea and reference Dense architecture are adapted from Chollet (2017). Standard TensorFlow, Keras and scikit-learn APIs provide model fitting and evaluation measures. Generative AI assisted with project planning, code development, notebook revision and prose editing. The author is responsible for checking the implementation, executing the experiments, verifying the reported values and complying with the institution's assessment rules.

The table below distinguishes direct adaptation, standard library use and AI-assisted analytical code. It does not label AI-produced material as wholly original.

provenance = pd.DataFrame({
    "element": [
        "Multi-hot vectorisation",
        "64-64 Reuters reference architecture",
        "Universal workflow structure",
        "Dense model assembly and fitting",
        "Metrics, splits and tabulation",
        "Experiment runner and controlled phases",
        "Bootstrap, calibration and collision diagnostics",
        "Report planning and prose revision",
    ],
    "provenance": [
        "Adapted from Chollet (2017), Chapter 3",
        "Adapted from Chollet (2017), Chapter 3",
        "Based on Chollet (2017), Section 4.5",
        "Standard Keras API use with AI-assisted development",
        "Standard scikit-learn and pandas API use",
        "AI-assisted development, reviewed and executed by the author",
        "AI-assisted development using NumPy and scikit-learn",
        "Generative AI assistance with author review",
    ],
    "location": [
        "Section 2.4",
        "Section 2.7.1",
        "Section 2",
        "Sections 2.5 to 3",
        "Sections 2.2 to 4",
        "Sections 2.7 to 3",
        "Sections 3 and 4",
        "Throughout",
    ],
})
provenance

	element	provenance	location
0	Multi-hot vectorisation	Adapted from Chollet (2017), Chapter 3	Section 2.4
1	64-64 Reuters reference architecture	Adapted from Chollet (2017), Chapter 3	Section 2.7.1
2	Universal workflow structure	Based on Chollet (2017), Section 4.5	Section 2
3	Dense model assembly and fitting	Standard Keras API use with AI-assisted development	Sections 2.5 to 3
4	Metrics, splits and tabulation	Standard scikit-learn and pandas API use	Sections 2.2 to 4
5	Experiment runner and controlled phases	AI-assisted development, reviewed and executed by the author	Sections 2.7 to 3
6	Bootstrap, calibration and collision diagnostics	AI-assisted development using NumPy and scikit-learn	Sections 3 and 4
7	Report planning and prose revision	Generative AI assistance with author review	Throughout