Running the Eval Harness

The eval harness has one entry point, npm run evals, plus a handful of flags that compose. This guide walks through the modes you’ll actually use.

The four common modes

1. Fast local sanity check (no LLM)

npm run evals --workspace=packages/evals -- --deterministic-only

Runs the ~35 library-level tests in under a second. Catches:

• Compression ratio regressions
• Dedup behavior changes
• Budget enforcement breaks
• Memory segmentation, temporal filtering, subgraph extraction
• Conflict detection rules

Use as a pre-commit hook or a fast PR signal. Does not exercise any LLM-bound code path.

2. Full local evaluation (Ollama)

npm run evals --workspace=packages/evals

Runs both tracks against a local Ollama model. Free but slower (~30s), and the judge is weaker than a frontier model, so it’s useful for “does this even compile” but not for production gating.

3. Full CI evaluation (GPT-4o, multi-sample)

OPENAI_API_KEY=sk-... npm run evals:ci --workspace=packages/evals

Same as local but:

• Uses gpt-4o as the judge
• 3 samples per semantic test (instead of 1)
• Higher concurrency (8 vs 2)
• Hides the progress bar (CI-friendly output)

The CI mode is what should run in scheduled regression checks. Cost is bounded by the per-test estimate; you’ll see a warning if the projection exceeds $5.

4. Baselined CI evaluation

OPENAI_API_KEY=sk-... \
  npm run evals:ci --workspace=packages/evals -- --baseline

Adds baseline comparison on top of mode 3:

• Loads golden/baselines/main-latest.json
• Compares current drift to the prior snapshot
• Exits with code 2 if any suite regressed by more than the noise floor (1pp by default, configurable)
• Overwrites the baseline only if the run passed both the absolute gate and the relative comparison

See Drift & Baselines for what counts as a regression.

CLI reference

Flag	Type	Default	What it does
--mode	local | ci	local	Picks provider, concurrency, sample defaults
--suite	suite name	3 core suites	Restricts to one suite. Default runs context-engine, memory, orchestrator; integration is accepted but drives no goldens (see note).
--samples	int	1 local / 3 ci	Number of independent semantic samples per test
--sut-model	string	claude-sonnet-4-6	Model driving the system-under-test (SUT) on the semantic track
--provider	anthropic | openai | ollama	ollama local / openai ci	Overrides the judge provider the mode would pick
--deterministic-only	flag	false	Skip the semantic track entirely
--baseline	flag	false	Load + compare + persist golden/baselines/main-latest.json
--baseline-noise-floor	float	1	Minimum pp delta to flag as a regression. Kept below the drift ceiling so sub-ceiling regressions are caught
--commit	string	(auto)	Short git SHA stamped onto a new baseline snapshot

Note

--suite integration is a valid flag, but no golden dataset ships for it. The manifest only carries goldens for orchestrator, memory, and context-engine (18 trajectories each), and integration is excluded from the default suite set. Selecting it runs nothing against goldens until you record integration trajectories.

Combining flags

These compose freely. Some useful combinations:

# One suite only
npm run evals --workspace=packages/evals -- --suite memory

# Multi-sample to detect flakiness, no baseline
npm run evals:ci --workspace=packages/evals -- --samples 5

# Tight baseline tolerance
npm run evals:ci --workspace=packages/evals -- --baseline --baseline-noise-floor 1.0

# CI mode but only library tests (fast PR gate without API costs)
npm run evals --workspace=packages/evals -- --deterministic-only

Environment variables

Variable	Required	Default	Purpose
OPENAI_API_KEY	CI semantic track	—	GPT-4o judge
ANTHROPIC_API_KEY	With --provider anthropic	—	Anthropic judge
OLLAMA_BASE_URL	Local	http://localhost:11434	Ollama endpoint
OLLAMA_MODEL	Local	llama3:8b-instruct-q4_K_M	Local judge model
EVAL_DRIFT_CEILING	No	5.0	Drift % gate threshold

CLI flags override env vars where both apply. Judge concurrency is not an env var: it’s a provider option (maxConcurrency), defaulting to 2 for Ollama and 8 for OpenAI and Anthropic.

Exit codes

Code	Meaning
0	Clean run
1	Drift gate failed OR a suite failed to load
2	Baseline regression detected but drift gate passed

For CI scripts, the canonical pattern is to hard-fail on 1 and warn-only (or open an issue) on 2.

Output

By default the runner writes to stdout. CI mode additionally emits GitHub Actions annotations (::error, ::warning) so failures appear inline on PRs.

Not currently emitted:

• Structured report.json artifact
• GitHub step summary
• JUnit XML for test-result aggregators

Troubleshooting

“Ollama: connection refused”: start the server with ollama serve. Confirm the model is pulled with ollama list.

“Cost warning: estimated $X exceeds threshold”: the CI mode estimates total token usage. The default warning fires at $5. Raise it via the provider option in code, or accept it and continue, since the warning is non-blocking.

“Baseline schema version mismatch”: you’re loading an older snapshot than the current code understands. Delete golden/baselines/main-latest.json and re-run with --baseline to bootstrap a fresh one.

“No prior baseline”: expected on the first run with --baseline. The current run becomes the baseline.

Other runners in the package

The regression gate is the main entry point, but two sibling runners ship alongside it:

# Efficacy matrix: measures extraction/compression quality against labeled corpora
npm run evals:efficacy --workspace=packages/evals

# Compression bench: context engine vs reference compressors (or --smoke for a quick pass)
npm run bench --workspace=packages/evals

These measure absolute quality rather than drift, and neither participates in the gate’s exit codes.

Related

• Eval Harness: what’s running under the hood
• Recording Goldens: refresh the trajectories the harness uses
• Drift & Baselines: what the numbers mean