rsi_eval
rsi_eval is a Python library and command-line tool that tests self-improving AI agent methods against standard agent benchmarks for researchers and developers.
What it is good for
- Testing whether a self-improving agent method actually gets better
- Running one agent setup against many coding and reasoning benchmarks
- Comparing agent versions using scores, token usage, and verifier output
- Saving and resuming agent evaluation experiments
At a glance
- Who it is for
- Setup difficulty
- Developer ยท Requires programming
Level 5 of 5. You need to write code to use it.
- Cost
- Free and open source
- Type
- Framework or SDK
- Tasks
- Works with
- Docker, Harbor, Codex CLI, Terminal-Bench, SWE-bench, LiveCodeBench
How to get started with rsi_eval
- Clone the repository from GitHub and create a Python 3.12 virtual environment.
- Install the SDK and CLI from the checkout, then run the one-task preview, which needs no Docker or model calls.
- Install the Harbor extra to run real benchmark evaluations.
- Install Docker and confirm it works with
docker info. - Edit
examples/evaluate.json, then runrsi-eval check --config examples/evaluate.jsonbefore running an evaluation.
From the project's own documentation. If a step does not match what you see, the official pages have the latest instructions.
Before you try it
- Requires Python 3.12 or newer.
- Real evaluations need Docker and Harbor 0.18.0 on your PATH.
- Some benchmarks need an NVIDIA GPU, native Linux amd64, or separate judge credentials.
- APEX data is released for evaluation only, not for training or parameter fitting.
Momentum
- Rank this week
- Not ranked (last #14, week 45, 2026)
- Best rank
- #14 week 45, 2026
- GitHub stars
- 470
GitHub numbers as of Nov 16, 2026. Updated weekly.
Ranking history
Official links
Questions
- Is rsi_eval free?
- Yes. It is free and open source (Apache-2.0 license).
- Do I need to know how to code to use rsi_eval?
- Yes. You need to write code to use it.
- What does rsi_eval work with?
- According to its own documentation, it works with Docker, Harbor, Codex CLI, Terminal-Bench, SWE-bench, and LiveCodeBench.
- How do I get started with rsi_eval?
- 1. Clone the repository from GitHub and create a Python 3.12 virtual environment. 2. Install the SDK and CLI from the checkout, then run the one-task preview, which needs no Docker or model calls. 3. Install the Harbor extra to run real benchmark evaluations. 4. Install Docker and confirm it works with docker info. 5. Edit examples/evaluate.json, then run rsi-eval check --config examples/evaluate.json before running an evaluation.
- What should I know before trying rsi_eval?
- Requires Python 3.12 or newer. Real evaluations need Docker and Harbor 0.18.0 on your PATH. Some benchmarks need an NVIDIA GPU, native Linux amd64, or separate judge credentials. APEX data is released for evaluation only, not for training or parameter fitting.
- What is rsi_eval good for?
- Testing whether a self-improving agent method actually gets better, running one agent setup against many coding and reasoning benchmarks, comparing agent versions using scores, token usage, and verifier output, and saving and resuming agent evaluation experiments.
Similar agents to rsi_eval
- RSI-Anything
Helps agents learn from feedback and improve, rather than measuring such methods across benchmarks.
- harness2
Automatically refines prompts, tools, and skills around an agent, rather than evaluating improvement methods.
- Opik
Records, tests, and monitors LLM apps and agents in general, not focused on self-improvement benchmarks.
- OpenEvolve
Repeatedly rewrites and scores code to find better algorithms, rather than providing a benchmark evaluation interface.
- RD-Agent
Automates data-driven AI research and development instead of evaluating improvement methods.
- Remory
Remory is a memory add-on that helps coding agents keep working after their long chat history is summarized, for developers running open models.