# Evals: what to run and what each result proves

**Authority:** normative developer guide to the eval purposes implemented in the Precise Harness.

**Maturity:** the commands below are runnable from the repository and were run on 2026-09-01. They prove local code and fixture properties; they are not deployment, customer-value, or promotion evidence.

In code, every Harness eval is an `Eval` with a `name`, an optional `proves` statement, and a `run` function returning an `EvalResult`. There is no eval-type enum. The types below are the distinct evidence purposes present in current registrations and the lint-enforced evidence check that runs beside them in CI. Tests prove software behavior; evals prove domain properties and refusals. `recordEvals` can retain results when a caller explicitly uses it, but running `eval` alone does not establish a scheduler or durable production history.

## Which eval when

| Developer intent | Eval type | Command |
|-|-|-|
| Check Portfolio Select code and its public boundary | Success | `cd harness && npm run test:portfolio-select` |
| Run the properties registered by Portfolio Select | Success / floor | `cd harness && npm run precise -- init portfolio-select eval` |
| Refuse training data from a later period | Temporal-leakage floor | `cd harness && npm run precise -- research markov-transition-map eval` |
| Prove a named unavailable surface stays unavailable | Refusal-only surface | `cd harness && npx tsx --test test/media-seat-registry.test.ts` |
| Check a port against its pinned source behavior | Standing differential | `cd harness && npm run test:capture-tape` |
| Refuse broken fixture lineage | Provenance integrity | `cd harness && npm run test:capture-tape` |
| Settle a frozen challenger against an incumbent | Shadow race / holdout | `cd harness && npm run precise -- init portfolio-select eval` |
| Verify a research plan without granting execution | Planning-only research | `cd harness && npm run precise -- research decision-program eval` |
| Prove and resume one local Product build | Product-build session | `cd harness && npm run session:campaign-improvement` |
| Compare predictions on delayed historical outcomes | Historical backtest | `cd harness && npx tsx --test test/campaign-book-backtest.test.ts` |
| Close, score, and replay a learning episode | Calibration / Decision Learning | `cd harness && npm run precise -- decision-learning eval` |
| Catch banned claims and floor bypasses in source | Lint-enforced check | `cd harness && npx tsx --test scripts/floor-lint.test.mjs` |
| Check the frozen experiment protocol | Ablation / experiment | `cd harness && npm run precise -- research eval` |

## Success evals

```bash
cd harness
npm run test:portfolio-select
npm run precise -- init portfolio-select eval
```

The package lane exercises Portfolio Select's pure selection, kernel admission, deterministic allocation, parity fixtures, lazy registration, three refusals, and shadow settlement. The Harness command runs its four registered evals through the public lazy-init road.

**Proves:** the built library and registered local properties work together on deterministic fixtures.

**Does not prove:** a Product calls Portfolio Select, the parameters fit Precise customer evidence, or any deployment is ready.

## Floor-firing evals

```bash
cd harness
npm run precise -- init portfolio-select eval
```

A floor is a named refusal that prevents a result from exceeding its evidence. Portfolio Select registers three floor evals: `select-quality-is-net-return-per-dollar` refuses proxy quality such as win rate; `select-refuses-unmeasured-quality` refuses a candidate with neither a settle nor a visibly shrunk settled family prior; `select-diversity-soft-never-zeroes-dominant` refuses a kernel or allocation that erases a strictly quality-dominant candidate without an explicit zero cap or override.

**Proves:** each adversarial fixture raises its exact `FloorViolation`, so the floor is exercised rather than merely listed.

**Does not prove:** the fixtures enumerate every bad input or that an absent Product caller enforces the floor in a customer workflow.

### Temporal-leakage floor evals

```bash
cd harness
npm run precise -- research markov-transition-map eval
```

`markov-transition-temporal-floor-actually-refuses-leakage` changes a validation-period transition to `train` and requires the map freezer to refuse it because it falls after `training_end`.

**Proves:** the current adversarial fixture cannot move later-period evidence into training.

**Does not prove:** an untested domain reader supplies correct timestamps or that every possible leakage route is covered.

### Refusal-only surface evals

```bash
cd harness
npx tsx --test test/media-seat-registry.test.ts
```

`refusal:named-future-seat-stays-unconnected` attempts to connect a visible, named-future media seat and passes only when `media-seat-named-future` refuses before any connection action.

**Proves:** the local seat registry keeps the unavailable surface visible and refuses its forbidden action through the named code.

**Does not prove:** a vendor connection exists, credentials are stored, or another Product surface preserves the refusal.

## Standing differential evals

```bash
cd harness
npm run test:capture-tape
```

`capture-tape:desk-differential` implements the divergence law: the ported math must reproduce four venue-neutral rows pinned to `desk@e09d1fc772005fab7739caad989d8b004ff9a5c2`. Rounded rows require canonical byte equality; raw accumulators allow only `1e-12` tolerance. Capture Tape is not in the optional CLI registry, so its current runnable eval road is this package lane; `capture-tape-harness.test.ts` calls all seven registered evals directly.

**Proves:** the generalized port still agrees with the pinned parity oracle under the declared comparison rule.

**Does not prove:** live capture, a Product caller, or that the source mechanism is correct for a new domain.

## Provenance-integrity evals

```bash
cd harness
npm run test:capture-tape
```

The Capture Tape trio is `capture-tape:provenance-malformed-refuses` (`capture-provenance-invalid`), `capture-tape:fixture-tamper-refuses` (`replay-mismatch`), and `capture-tape:estate-provenance-mismatch-refuses` (`capture-fixture-provenance-mismatch`). The same lane also refuses raw identifiers as fixture hygiene.

**Proves:** malformed lineage, changed sealed fixture bytes, and disagreement with the estate pin each fail through a named refusal.

**Does not prove:** external signer provenance, live source custody, or tenant storage.

## Shadow-race and holdout evals

```bash
cd harness
npm run precise -- init portfolio-select eval
```

`select-shadow-plan-vs-incumbent-holdout` freezes distinct selector and incumbent plan refs, requires disjoint support, settles held-out net return per dollar, and books a one-window loss as a provisional economic kill. A settled real race after the ad-tech parameter refit is Portfolio Select's open condition; the capability currently reports `settled_shadow_races: 0`.

**Proves:** the deterministic settlement rule preserves the holdout boundary and does not turn a thin result into a final verdict.

**Does not prove:** Portfolio Select beats an incumbent, the parity parameters transfer to ad tech, or the open condition has settled.

## Planning-only research evals

```bash
cd harness
npm run precise -- research decision-program eval
```

`cold-coverage-planner-emits-only-a-verified-planning-receipt` freezes a decision-evidence map, solves the small cold-start coverage fixture to proved optimality, verifies its receipt, and requires both execution and activation authority to remain `none`.

**Proves:** the reference planner reconciles the deterministic fixture into a verified, content-addressed plan without widening authority.

**Does not prove:** admitted host work ran, the plan improved a decision, or the small exact solver is suitable for production scale.

## Product-build session evals

```bash
cd harness
npm run precise -- product-build eval
npm run session:campaign-improvement
```

`unfamiliar-product-reaches-proven-without-a-platform-login` proves the generic Product-build record contract with retained useful and refusal evidence. The Campaign Improvement session command then uses that contract to create or resume one local developer receipt and reports whether any platform project is attached.

**Proves:** a local Product build can reach `proven`, retain its evidence pair, and resume without minting platform identity.

**Does not prove:** external promotion, a platform attachment, deployment, customer use, or host authority.

## Historical backtest evals

```bash
cd harness
npx tsx --test test/campaign-book-backtest.test.ts
```

`campaign-book-backtest-produces-a-useful-paired-historical-result` runs the identity-only Campaign Book harness over delayed outcomes and requires enough campaigns and paired folds, while preserving its historical-observational evidence limit and granting no activation authority.

**Proves:** the retained deterministic fixture produces a valid paired historical result and the lane's time, attribution, input, and runner refusals still pass.

**Does not prove:** causal lift, prospective value, arbitrary workspace backtesting, hosted execution, or Portfolio Select performance.

## Calibration and Decision Learning evals

```bash
cd harness
npm run precise -- decision-learning eval
```

The Decision Learning lane closes and replays v1 and v2 episodes, derives a Forecast Score from frozen prediction and verified outcome, checks the split-conformal Mondrian band in its deterministic fixture, retains descriptive contribution and an authorized Product update, maps one verified episode into the canonical evaluation envelope, and fires target-authority, judgment-reference, and tenant-scope refusals. No Product caller currently sends Portfolio Select outputs into Decision Learning.

**Proves:** the registered local episode, scoring, calibration-fixture, mapping, replay, and refusal properties pass together.

**Does not prove:** continuous customer calibration, hosted custody, model improvement, or any Portfolio Select learning loop.

## Lint-enforced evidence checks

```bash
cd harness
npx tsx --test scripts/floor-lint.test.mjs
```

This is a CI test beside the runtime `Eval` registrations, not an `Eval` object. It proves the static checker flags a banned phrase used as a claim and a judgment-producing harness that never calls or imports a floor, while allowing explicit negation and documented delegation.

**Proves:** the lint rules fire on their adversarial source fixtures and stay quiet on the allowed fixtures.

**Does not prove:** semantic truth of all copy, complete control-flow analysis, or that a passing source file exercised a runtime refusal.

## Ablation and evidence-bound experiment evals

```bash
cd harness
npm run precise -- research eval
```

Use a component ablation only to compare implementations against that component's invariant fixtures. When a model, prompt, strategy, policy, data treatment, or allocation rule could change results, freeze an evidence-bound experiment instead: predeclare the question, alternatives, controls, fixed population, untouched confirmation, costs, courts, and promotion rule; retain `winner: null` when the evidence does not clear it. Follow [Running Precise experiments and ablations](AGENT-ABLATION-RUNBOOK.md) for that full road rather than treating a passing eval as promotion evidence.

**Proves:** the current research protocol's registered freeze, verification, no-authority, and refusal properties pass.

**Does not prove:** a particular challenger wins, a real evidence set ran, or anyone has activation authority.

## Verbatim tails from the commands above

`npm run test:portfolio-select`:

```text
ℹ tests 15
ℹ pass 15
ℹ fail 0
```

`npm run precise -- init portfolio-select eval`:

```json
"pass": 4,
"fail": 0,
"failed": [],
"failures": []
```

`npm run test:capture-tape`:

```text
ℹ tests 11
ℹ pass 11
ℹ fail 0
```

`npm run precise -- decision-learning eval`:

```json
"pass": 14,
"fail": 0,
"failed": [],
"failures": []
```

`npm run precise -- research eval`:

```json
"pass": 23,
"fail": 0,
"failed": [],
"failures": []
```

`npm run precise -- research markov-transition-map eval`:

```json
"pass": 3,
"fail": 0,
"failed": [],
"failures": []
```

`npm run precise -- research decision-program eval`:

```json
"pass": 13,
"fail": 0,
"failed": [],
"failures": []
```

`npm run precise -- product-build eval`:

```json
"pass": 12,
"fail": 0,
"failed": [],
"failures": []
```

`npm run session:campaign-improvement`:

```json
"status": "proven",
"receipt_disposition": "created",
"project_attachment_state": "absent",
"connected_evaluation": null
```

`npx tsx --test test/campaign-book-backtest.test.ts`:

```text
ℹ tests 33
ℹ pass 33
ℹ fail 0
```

`npx tsx --test test/media-seat-registry.test.ts`:

```text
ℹ tests 2
ℹ pass 2
ℹ fail 0
```

`npx tsx --test scripts/floor-lint.test.mjs`:

```text
ℹ tests 12
ℹ pass 12
ℹ fail 0
```