CompanionCourt — Leakage & Certification Policy
Version: v1.0 · Status: launch version. What this is: one of three governance policies of a public docket for pressure-testing AI companions. It draws the boundary between what is public and what stays hidden, says what happens when the public side leaks into training data, and states plainly what we do not sell: v1 offers no certification — no seal, no compliance equivalence. A docket that hands out stamps before it has earned trust would be caving to its own pressure test.
v1.0 changelog (launch): finalized from v0.1-draft; the four
[DRAFT-CALL]s resolved by maintainer ruling — §2 leakage detection signals, §2 hidden-tier disclosure detail, §4 certification-expiry basis (certification still not offered in v1), §5 private-audit existence disclosure.
1. The boundary: public devset vs hidden bar exam
| Tier | Visibility | Role |
|---|---|---|
| Public corpus / devset (16–20 cases) | Ships with the repo; anyone can run it | The runner's input; builds the shared language |
| Public docket (10–12 of those, full case files) | Published with rulings and dissent | The precedent edition of the devset — same cases, not a different set |
| Hidden bar exam (incl. the 14-case held-out certification corpus + rotating variant bank) | Never enters the public repo | Anti-gaming reserve; the private evaluation asset for any future certification or re-examination |
The boundary is structural, not aspirational: nothing from the hidden tier is quoted, excerpted, paraphrased into public rulings, or used to generate public artifacts. Public precedents build the language; the private taste registry and hidden corpus stay private.
2. Leakage: assumption, detection, response
Assumption. Public cases will eventually be trained against. This is priced in — it is why the corpus is versioned, why every published score names its corpus version, and why fix directions in rulings are doctrinal, not recipe-level (the docket teaches judgment; it is written to be a poor answer key).
Detection (ruling). Four signals, and any one triggers a leakage review: (1) verbatim or near-verbatim case text surfacing in public model outputs; (2) a sudden, case-specific score jump on one case across vendors; (3) an explicit disclosure of corpus inclusion in a training set; (4) divergence between a respondent's public-devset scores and its hidden-variant scores on the same pressure shapes — public inflation without hidden movement is the signature of teaching to the test. Triggering a review is not itself a finding — it opens the inquiry that decides retirement.
Response — rotation and retirement (mechanics in the Case Lifecycle Policy):
- A confirmed-leaked case is retired from the active corpus and replaced from the variant bank (same pressure shape and ground truth, new surface) in the next corpus version.
- Scores earned on the leaked version remain published, labeled with their corpus version — leakage changes the future corpus, never the past record.
- A leak of hidden-bar-exam material is a different order of event: the affected hidden cases are burned and replaced, and the incident is disclosed in the Docket Update. Disclosure detail (ruling): we acknowledge the leak event and the affected tier's size — and never the case content. Disclosing the breach must not itself expose what the hidden tier holds.
3. Certification: not offered in v1
- No seal. v1 issues diagnostic reports, not certificates. No respondent may describe a CompanionCourt result as a certification, accreditation, or pass.
- No compliance equivalence. Verdicts are SB 243-adjacent operational evidence at most — behavioral evidence relevant to safety-protocol effectiveness — and are not a legal-compliance equivalent. Any compliance-mapping language requires counsel review before publication.
- The public storyline is companion integrity under pressure; certification is not the pitch, and the first day sells nothing.
4. The future certification path — separately gated, honestly priced
Certification language unlocks on its own line — never bundled with jurisdiction language or predictive-validity claims — and only when all three hold:
- a functioning hidden bar exam (held-out corpus + rotating variants) that a certification could actually be examined against;
- expiry — any future certification is time-boxed and re-examined; there is no perpetual seal, because models drift and corpora rotate. Ruling: expiry is tied to the corpus version, not the calendar — a certification would name the corpus it was examined against, and a new corpus version is itself a re-examination trigger. (This only fixes the term for if and when a certification is ever offered; v1 still offers none);
- paid-pilot evidence — real willingness-to-pay discovered through pilots, with the standing caveat that a vendor's paid or invited participation never counts toward the independence signals that unlock institutional language.
5. Public report vs private audit
- A public report is a diagnostic report on the public corpus: published in full, transcript-linked, on the record, eligible (once evidence gates pass) for cross-model comparison.
- A private audit is a paid engagement run on hidden-tier material for a party that wants to find its failure modes before the public does ("the failure modes regulators and T&S teams will ask about"). Its results are private, confer no seal and no public claim — a party may not cite a private audit as a CompanionCourt result — and, like every paid interaction, it cannot influence case selection, is disclosed under the conflict-of-interest rules, and does not count toward any independence signal. Existence disclosure (ruling): the existence of private-audit engagements is disclosed as an aggregate count per year, with no client names — enough to show the conflict surface honestly, never enough to name who paid.