FAQ

Common questions about intent-bench methodology and design.

How do you score intent fulfillment without it getting subjective?

We don't score it subjectively -- and that is a deliberate design choice. intent-bench never asks a human (or an LLM) to judge whether code "fulfills intent." Instead, it reduces the question to automated, deterministic signals that leave no room for interpretation.

Binary test outcomes, not rubrics

Every experiment ships with a test suite. The harness runs the test command, parses the output with multi-framework regex (Go, Python, Node, Rust, TAP, and others), and classifies the result as PASS, PARTIAL, FAIL, ERROR, or TIMEOUT. No human reads the output -- it is pure string matching against standard test framework formats. Either the tests pass or they don't.

A/B blinding removes evaluator bias

Both the control and treatment conditions receive the identical prompt (verified by SHA-256 hash). The treatment condition additionally has intent artifacts seeded into the working directory, but the prompt never mentions them. The agent discovers these artifacts through natural exploration -- it is blind to its assignment. The test harness is also blind: it only reads pass/fail counts.

Process metrics are formula-based

Beyond pass/fail, intent-bench measures process quality through a knowledge entropy score (0-10) composed of four dimensions: productivity (fraction of tool calls producing lasting output), rework (fraction of edits targeting already-edited files), conventionality (standard project structure checks), and test coherence (test pass rate). Each dimension is computed from artifact counts or file system state -- no human judgment about code quality or architecture.

Statistical rigor, not opinion

Completion rates are compared with Fisher's exact test. Token efficiency uses the Mann-Whitney U test (non-parametric, no normality assumption). Confidence intervals are bootstrapped with 10,000 iterations at a fixed seed for reproducibility. Variance reduction is computed from coefficient of variation. All of this is mechanical -- the analysis script produces the same numbers regardless of who runs it.

In short: the benchmark asks "did the tests pass and how efficiently did the agent get there?" -- never "did the code look good?" That is what keeps subjectivity out.

What metrics does intent-bench report?

Each experiment produces several metrics:

Completion rate -- fraction of runs where all tests pass
Token efficiency -- total tokens consumed for passing runs, with control/treatment ratio
Knowledge entropy -- composite score (0-10) measuring process waste
Planning-to-execution ratio -- tokens spent before vs. after first code edit
Backtrack rate -- fraction of turns re-reading previously read files
Variance reduction -- whether the treatment reduces outcome variability

Why not use LLM-as-judge?

LLM-as-judge introduces a second model's biases and is not deterministic across runs. intent-bench is designed for reproducibility: same inputs, same analysis, same numbers. Automated test suites provide a ground truth that does not shift between evaluations.

Can I add my own experiments?

Yes. Create an experiment YAML in experiments/, a prompt in prompts/, and fixture data for each treatment. The prompt must not reference any treatment tool (this is enforced by automated validation). See the How to Use guide for details.

Can I add my own treatment?

Yes. Treatments are shell scripts with a standard interface: treatments/<name>.sh <setup|validate> <workdir> <experiment> <fixture_dir>. Create your treatment script, add fixture data, and reference it in an experiment YAML.

How many runs do you need for significance?

The analysis requires at least 5 runs per condition before publishing any claim. For token efficiency (a continuous metric), more runs yield tighter confidence intervals. The bootstrap approach works well with moderate sample sizes.