rsi_eval

rsi_eval is a Python library and command-line tool that tests self-improving AI agent methods against standard agent benchmarks for researchers and developers.

What it is good for

  • Testing whether a self-improving agent method actually gets better
  • Running one agent setup against many coding and reasoning benchmarks
  • Comparing agent versions using scores, token usage, and verifier output
  • Saving and resuming agent evaluation experiments

At a glance

Who it is for
Setup difficulty
Developer ยท Requires programming

Level 5 of 5. You need to write code to use it.

Cost
Free and open source
Type
Framework or SDK
Tasks
Works with
Docker, Harbor, Codex CLI, Terminal-Bench, SWE-bench, LiveCodeBench

How to get started with rsi_eval

  1. Clone the repository from GitHub and create a Python 3.12 virtual environment.
  2. Install the SDK and CLI from the checkout, then run the one-task preview, which needs no Docker or model calls.
  3. Install the Harbor extra to run real benchmark evaluations.
  4. Install Docker and confirm it works with docker info.
  5. Edit examples/evaluate.json, then run rsi-eval check --config examples/evaluate.json before running an evaluation.

From the project's own documentation. If a step does not match what you see, the official pages have the latest instructions.

Before you try it

  • Requires Python 3.12 or newer.
  • Real evaluations need Docker and Harbor 0.18.0 on your PATH.
  • Some benchmarks need an NVIDIA GPU, native Linux amd64, or separate judge credentials.
  • APEX data is released for evaluation only, not for training or parameter fitting.

Momentum

Rank this week
Not ranked (last #14, week 45, 2026)
Best rank
#14 week 45, 2026
GitHub stars
470

GitHub numbers as of Nov 16, 2026. Updated weekly.

292 388 483 Nov 2, 2026 Nov 16, 2026
GitHub stars for rsi_eval: 305 on Nov 2, 2026, 470 on Nov 16, 2026. Source: GitHub.

Ranking history

First ranked in week 45, 2026.

Questions

Is rsi_eval free?
Yes. It is free and open source (Apache-2.0 license).
Do I need to know how to code to use rsi_eval?
Yes. You need to write code to use it.
What does rsi_eval work with?
According to its own documentation, it works with Docker, Harbor, Codex CLI, Terminal-Bench, SWE-bench, and LiveCodeBench.
How do I get started with rsi_eval?
1. Clone the repository from GitHub and create a Python 3.12 virtual environment. 2. Install the SDK and CLI from the checkout, then run the one-task preview, which needs no Docker or model calls. 3. Install the Harbor extra to run real benchmark evaluations. 4. Install Docker and confirm it works with docker info. 5. Edit examples/evaluate.json, then run rsi-eval check --config examples/evaluate.json before running an evaluation.
What should I know before trying rsi_eval?
Requires Python 3.12 or newer. Real evaluations need Docker and Harbor 0.18.0 on your PATH. Some benchmarks need an NVIDIA GPU, native Linux amd64, or separate judge credentials. APEX data is released for evaluation only, not for training or parameter fitting.
What is rsi_eval good for?
Testing whether a self-improving agent method actually gets better, running one agent setup against many coding and reasoning benchmarks, comparing agent versions using scores, token usage, and verifier output, and saving and resuming agent evaluation experiments.

Similar agents to rsi_eval

  • RSI-Anything

    Helps agents learn from feedback and improve, rather than measuring such methods across benchmarks.

  • harness2

    Automatically refines prompts, tools, and skills around an agent, rather than evaluating improvement methods.

  • Opik

    Records, tests, and monitors LLM apps and agents in general, not focused on self-improvement benchmarks.

  • OpenEvolve

    Repeatedly rewrites and scores code to find better algorithms, rather than providing a benchmark evaluation interface.

  • RD-Agent

    Automates data-driven AI research and development instead of evaluating improvement methods.

  • Remory

    Remory is a memory add-on that helps coding agents keep working after their long chat history is summarized, for developers running open models.

Last updated Nov 16, 2026. Tracked since Nov 2, 2026. Spotted an error? Tell us.