LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models

Daniele Cipollone1,2, Sergey Titov2, Maliheh Izadi1, Egor Bogomolov2, Arie van Deursen1
1Delft University of Technology, 2JetBrains Research
NeurIPS 2026 · Evaluations & Datasets Track
Paper arXiv Code Zenodo Dataset Leaderboard
Overview of the three tasks: API Calling completes a real call site, API Identification names an API from a redacted description, Signature Recall writes a versioned signature. Below, the four API Calling prompt levels L0 to L3.

LibEvoBench tests whether code models know an API as it exists in a specific library version, across 29 releases of PyTorch, NumPy and SciPy.

Abstract

Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one. However, current LLMs are trained on temporally mixed corpora and lack explicit mechanisms for such version-specific reasoning, leading to anachronistic errors: calling APIs as they exist in a different library version.

To systematically evaluate this phenomenon, we introduce LibEvoBench, a multi-task benchmark spanning multiple versions of widely used Python libraries, along with a new metric, the Software Evolution Understanding Score (SEUS), to measure models’ consistency when working with evolving APIs. Our results show that state-of-the-art models are largely version-oblivious: performance degrades for evolving APIs, while for stable APIs it remains the same across versions. Moreover, simply specifying the target version provides no benefit, while relevant documentation significantly boosts models’ accuracy. These findings highlight a systematic limitation of current training paradigms and motivate new approaches for temporally grounded knowledge in code generation.

TL;DR
  • Even SOTA models lag on version-specific knowledge: a persistent 5–10 point gap between stable and evolving APIs that neither scale nor training recency closes.
  • Models don't store the version as a dimension of library knowledge: giving the target version yields no gain, while giving documentation adds +10–20 points.
  • Failures are temporal confusion, not fabrication: models fall back on version-agnostic, high-frequency symbols that hide the gap.

Leaderboard

13 models, evaluated by the authors at zero temperature with no thinking budget.

    Loading the leaderboard…

    The benchmark

    For every tracked release, the public API surface is rebuilt from the official Sphinx inventory and from runtime introspection, and an API counts as present only when both agree. The resulting compatibility matrices split APIs into stable ones, whose signature never changes, and evolving ones, which are added, removed or modified within the covered releases.

    LibraryVersionsAPI-CAPI-ISR
    PyTorch1016,44417,03625,656
    NumPy109,2759,38511,386
    SciPy93,94813,24418,052
    Total2929,66739,66555,094

    Every wrong answer is classified by matrix lookup, with no manual coding: invalid (a real symbol at that version, but the wrong one), anachronistic (exists in another release, not the requested one) or unknown (never existed in any tracked release). Models are ranked by the Software Evolution Understanding Score (SEUS), which averages the harmonic mean of stable and evolving scores and subtracts a stability penalty and an anachronism penalty.

    Three line charts (OpenAI GPT, Sonnet and Gemini, Qwen) of mean score against normalized library version. Dashed stable-API lines stay flat; solid evolving-API lines fall toward the latest versions.

    Mean score across API-C@L1, API-I and SR over normalized library version. Stable APIs dashed, evolving APIs solid.

    Three line charts of API Calling exact match across prompt levels L0 to L3. Scores rise from L0 to L1, stay flat from L1 to L2, and rise again at L3.

    API Calling accuracy by prompt level. The docstring drives the L0→L1 gain; the version constraint at L2 adds nothing.

    Get started

    The benchmark is on Hugging Face, one configuration per library and table. Configuration names follow <library>__<table>, with library torch, numpy or scipy.

    TableContents
    benchmark_t1API Calling (API-C) instances, prompt levels L0–L3
    benchmark_t2API Identification (API-I) instances
    benchmark_t3Signature Recall (SR) instances
    api_comp_matrixWhich APIs exist in which release
    sign_comp_matrixThe signature of each API in each release
    api_dictionaryThe tracked APIs
     
    from datasets import load_dataset

# API Calling instances for PyTorch
ds = load_dataset("JetBrains/LibEvoBench", "torch__benchmark_t1", split="train")

    The code repository has the pipeline that builds the matrices and tasks, the evaluation scripts, and the model outputs behind the results in the paper.

    BibTeX

    @inproceedings{cipollone2026libevobench,
  title     = {{LibEvoBench}: Probing Temporal Knowledge Stratification
               in Code Generation Models},
  author    = {Cipollone, Daniele and Titov, Sergey and Izadi, Maliheh
               and Bogomolov, Egor and van Deursen, Arie},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}