Beyond Anomaly Detection: A Next-Step Prediction Framework for Log-Based Anomaly Detection with Structured, Actionable Episode Reports

Abstract

Large-scale systems generate logs for all events occurring in a computing system. The most significant open challenge is to extract actionable security intelligence from the millions of logs generated. The training phase of previous papers such as DeepLog, LogAnomaly and LogBERT do not include any anomaly label, but the ground truth labels are reintroduced in the decision phase of the pipeline. These methods just identify anomalies but never go beyond what an anomaly represents, how severe it is, or which events triggered them. This paper proposes an end-to-end pipeline which can overcome all of the above limitations simultaneously. Only the normal logs are used to train a Long Short-Term Memory (LSTM) next-step predictor, and the anomaly logs are selected solely by the prediction error without any other parameter. Five label-free threshold methods are computed from the anomaly score distribution, eliminating the need for labelling at the thresholding step. The pipeline deploys either Otsu binarisation or KDE saddle based purely on score geometry, and ground truth labels are used only afterward, for measuring performance. Per-step contributions of each flagged window's anomaly score reveals if the anomaly is a sharp spike, a sustained error, or a moderate mix of both. The windows are assigned to one of five subsystems by keyword rules, are linked into temporally coherent episodes, and are aggregated by a majority-vote spreading signal which separates truly spreading faults from repeated single-location faults. This is then translated into natural language generation by means of a fixed action, followed by a natural language generation based on a template. The pipeline demonstrates competitive performance on a 4.6-million-entry BlueGene/L corpus, and has interpretable, actionable detection performance, when compared to DeepLog, LogAnomaly, and LogBERT.