Leakage-Aware and Resource-Conscious Arabic News Classification: Static Linear Models and Prior-Adjusted N-Gram Fusion

Abstract

A headline accuracy score can give an overly reassuring account of Arabic news classification. It says little about duplicate leakage, errors in minority topics, the quality of confidence estimates, or the resources needed to train and use a model. We therefore re-evaluated Version 2 of the Ultimate Arabic News Dataset under a protocol designed around these concerns. The source file contains 196,279 articles in ten topics. Before creating any partition, we used exact hashes and MinHash grouping to find duplicate and near-duplicate articles. A group-preserving 70/15/15 split then produced 134,318 training documents, 28,786 validation documents, and a locked test set of 28,785 documents, with no duplicate group shared across partitions. Eleven conventional classifiers were compared using the same combined word/character TF-IDF representation. We also revisited a six-expert n-gram ensemble, testing the original learned soft gate against temperature-calibrated averaging, confidence- and entropy-based weighting, global and class-specific convex mixtures, out-of-fold probabilistic stacking, top-2 routing, and prior correction. Fusion choices were made from validation Macro-F1 only. LinearSVC gave the strongest locked-test result (Macro-F1=0.9290; accuracy=0.9587), while SGDClassifier was close (Macro-F1=0.9277) and used less time and memory. The best ensemble, a prior-adjusted global convex mixture, reached Macro-F1=0.9163 on validation and 0.9201 on test. Compared with the original soft gate, its mean improvement was +0.0114 (95% CI: +0.0070 to +0.0160); compared with equal averaging, the improvement was +0.0065 (95% CI: +0.0022 to +0.0108). These gains did not overturn the ranking of the two leading linear models. The largest class-specific change was for Society: precision rose from 0.4876 to 0.6000 and F1 from 0.6202 to 0.6630. A separate train-only analysis also found markedly better log loss, expected calibration error, and Brier score after temperature scaling for LinearSVC and RidgeClassifier. The evidence therefore favors a tuned sparse linear classifier as the general benchmark for this corpus, while showing that calibration and prior correction can repair particular ensemble errors.