Why RecSys Metrics Lie: How Preprocessing Nuances Break Reproducibility and What to Do About It

In machine learning, it is common to blame the neural network architecture, the loss function, or the negative sampling method when a published research result cannot be reproduced. However, in recommender systems (RecSys), the root of the problem often lies in subtle data preparation details. Researchers from Sber AI Lab and the AIRI Institute have shown that two teams can use what is formally the same dataset and splitting strategy (such as a global temporal split) yet obtain fundamentally different metrics due to hidden filtering nuances.

To address this issue, the developers built and open-sourced SplitLight, a tool for auditing datasets and their splitting strategies, presented at the SIGIR 2026 conference.

Subtleties That Turn Benchmarks Upside Down

The team's experiments with the popular SASRec model clearly demonstrated the profound impact of preprocessing rules on final metrics:

• Timestamp collisions: when a user's actions share the exact same timestamp, their sequence order becomes arbitrary. A simple change in the sorting of such records on the MovieLens-1M dataset dropped the NDCG@10 metric from 0.1861 to 0.1685 without any changes to the neural network architecture.
• Repeated interactions: on the Dressipi dataset, keeping consecutive repetitions of the same item artificially inflated NDCG@10 from 0.1343 to 0.2168. The model simply learned to recommend the item the user had just clicked on, while performance on genuinely new items dropped to 0.1225.
• Cold start: retaining items in the test set that were never seen during training on the same Dressipi dataset reduced algorithm performance from 0.1343 to 0.0965.

Hidden Pitfalls of Popular Datasets

Auditing common benchmarks using SplitLight revealed unexpected characteristics in public data:

• Temporal collisions: in MovieLens-1M, about 53% of events have duplicate timestamps within a single user's history, and in the Beauty dataset, such interactions account for around 39%.
• Masked real activity: while the MovieLens-20M dataset formally spans 20 years, the median activity time of an individual user within it is only about one hour.
• Temporal leakage: under the popular leave-one-out split scheme on the Diginetica dataset, around 89% of targets were affected by future information due to mixed global time intervals.
• Unexpected cold start: using a global time split (0.9 quantile), the proportion of cold users in the test split reached 100% for Diginetica and Dressipi, and 82.6% for MovieLens-20M. Under these conditions, the algorithm is evaluated solely on its ability to handle newcomers.

What SplitLight Can Do

The package focuses on interaction log formats with a minimal schema of fields: user_id, item_id, and timestamp in CSV or Parquet formats. The framework is available as a Python API for integration into ML pipelines, ready-to-use Jupyter notebooks, and a Streamlit-based interface for visual auditing without writing code.

Before launching resource-intensive training, the tool allows users to evaluate:

• distribution shifts before and after filtering (session lengths, density, popularity);
• the scale of repeated and consecutive interactions;
• the presence of temporal data leakage between train, input, and target;
• the proportion of cold entities in test and validation sets;
• statistical distribution shifts (using the Kolmogorov–Smirnov test).

“To achieve reproducibility, it is not enough to simply name the splitting scheme. It is essential to specify the exact rules for handling duplicates, resolving timestamp collisions, the fraction of filtered items, and the structure of input history and target pairs,” the study's authors note.

The project's source code is published on GitHub under the permissive MIT license.