Independent evaluation · 2026-09-07
V18 is final. The correctness goal is not met.
The execution setup is repaired, but full fixture predicate coverage is not established. The recorded V18 results miss the primary correctness targets: 35/100 first turns error; scorer precision is 57/111 (51.4%); 18/100 first turns complete correctly. The candidate is frozen. No V19 was admitted. All attempts, refusals, losses, and review findings remain visible.
Absolute results
Answer precision
57/111 (51.4%)Correct first-turn answers
18/100 (18.0%)The completion and refusal trade
These are repeated measurements on the opened development set. They do not certify holdout quality. From V15 to V17, correct first turns fell from 10 to 7 out of 28. Refusals rose from 20 to 25 out of 70. Neither candidate met the 40% completion target. V18 was frozen without another development quality run. No V19 was admitted.
| Candidate | Correct first turns | Refusals | 40% first-turn target |
|---|---|---|---|
| V15 | 10/28 (35.7%) | 20/70: 18 unverified + 2 justified | Failed |
| V16 | 6/28 (21.4%) | 23/70: 21 unverified + 2 justified | Failed |
| V17 | 7/28 (25.0%) | 25/70: 23 unverified + 2 justified | Failed |
| V18 | No repeated development quality run | Frozen before the sealed holdout | Measured separately on 100 held first turns |
Every outcome stays visible
Refusals, blocked turns, and unverified comparisons earn no completed-answer credit.
| Frozen outcome | Original / 250 | V18 / 250 |
|---|---|---|
| comparison_unverified | 18 | 21 |
| correct_completed | 36 | 57 |
| refusal_unverified | 4 | 24 |
| silent_wrong | 31 | 54 |
| task_error | 161 | 93 |
| unnecessary_clarification | 0 | 1 |
Evaluation validity
| Gate | Status | Evidence and limits |
|---|---|---|
| E0 cleanup | Verified | Seven inherited processes and two disposable containers were stopped. Receipts identify the owned resources. |
| E1 fixture fidelity | Partially verified; late coverage gap | 31 relations and 566 catalog columns; 420 reader-visible fields. All 38 registered sensitivity controls pass. But the staff-exclusion predicate removes no rows: pd_is_ct_employee has zero true rows. Full E1 predicate fidelity is not established. No fixture-caused execution error was found. |
| E2 definition retrieval | Verified in both arms | Both arms use definition-lexical-concepts-sha256-v1 and the same seven indexed definitions. Real Qdrant retrieval and authorized compilation run. MissingAPIKeyError count is zero. No Cohere key is required. |
| E3 scorer classes | Class rules verified; causal chain limit | The three parent-aware clarification classes are implemented and old V10 scores remain unchanged. The final scorer labels blocked_by_parent only when the child clarifies. V18 has 44 explicitly dependent turns after failed/refused parents; none gives the required small recovery. A zero blocked label count does not prove healthy chains. |
| E4 independent holdout | Opened once after final freeze | 250 distinct texts, 50 families, and 50 four-turn conversations plus 50 standalone questions. There are zero cross-family duplicate texts and zero exact held-question matches in the frozen application. Semantic validity has the limits described below. |
| E5 baseline viability | Failed; user authorized continuation | 60 of 100 supported first turns errored. All 60 causes are product defects. Zero fixture or environment causes. This replaces the 44-of-46 interim cause count with the completed cause table. |
| E6 deterministic latency | Measured; target failed | All 180 timings retained. Original p95 1549.449 ms; V18 1732.008 ms. Regression 11.782%. Fixed-query conformity is 36/90 versus 66/90. The same declared adapter is verified in both arms. |
| Provider availability | Separate from configuration validity | Observed failed external-generation calls: original 0, V18 20. All calls remain counted. An SDK exception alone does not prove a remote outage. |
Baseline first-turn error causes
| Cause | Count | Attribution |
|---|---|---|
| Planner returned no usable source SQL | 36 | Product |
| Generated schema reference was wrong or absent | 10 | Product |
| Required merge key was missing | 7 | Product |
| Generated SQL lacked schema qualification | 5 | Product |
| Generation returned no usable query | 1 | Product |
| Generated function used the wrong type | 1 | Product |
| Fixture omissions | 0 | Fixture |
| Environment or provider failure | 0 | Environment |
| Total supported first-turn errors | 60 | 100 supported first turns |
Final verification
| Check | Result | Evidence and limits |
|---|---|---|
| Final full Python suite | 29,927 pass; 21 fail; 50 skip | 8 workers, --dist loadfile. Every failed node matches the retained pre-existing set. Application and test bytes are unchanged. The full suite is not green. |
| Static source checks | Passed | Strict mypy and Ruff pass over src/. The manifest binds all 1,692 application files. |
| Curated critical invariants | 568/568 passed | Security, cohort, grain, unit, calendar and definition controls. Counts overlap with the full suite. |
| Curated narrative replay | 12/12 passed | Unsupported prose is injected. Runtime output uses checked fact selections. Legacy metadata-free summary callers retain their old behavior. |
| Chain and migration recovery | 57/57 chains; 9/9 migrations passed | Three- and five-turn chains include a failure at each position. Migration tests inject pre-ledger and post-commit crashes. |
| Native browser to database path | 6/6 cases passed | Only external generation is intercepted. The first navigation failed before server readiness and submitted zero jobs; both attempts are retained. The checked chain is 45 clients, GBP 2454.50, and count 45. |
| Saved reads and optional work | 30 reads start zero work | Checked rows are readable after 225.498 ms and 334.198 ms during at least 20-second optional waits. These are two observations, not p95. |
| Other UI checks | 244 Node; 77 mocked Chromium passes retained | All 397 relevant UI inputs match their exact V16 bytes. These are reused checks, separate from the fresh native V18 run. |
| Independent final review | Three reviews complete; material defects remain | All three requested gpt-6-astra/xhigh reviewers became available after the holdout completed. They independently challenged arithmetic, query meaning, recovery and latency. Their final receipts are included. This is separate-agent review, not an external audit. |
| Lead semantic counterexamples | 2 scored-correct first turns disproved on separate inputs | A NULL role is treated as User. A note person replaces a deal person. Read-only VALUES queries prove both differences. No score, fixture, or application is changed. |
All acceptance targets
| Target | Status | Paired result and limits |
|---|---|---|
| Zero curated critical violations | passed | V18: 568 checks, zero failures or skips. Finite curated scope. |
| Zero unclassified errors in both arms | failed | Original 21 → V18 0. Original numeric labels remain unchanged after cause review. |
| Zero fixture-caused errors in both arms | passed | Original 0 → V18 0 fixture-caused execution errors. Required fields are available. Full E1 predicate sensitivity fails because the employee-exclusion predicate changes no answer. |
| Zero environment-caused errors in both arms | failed | Configuration errors: 0 → 0. Failed external-generation calls: 0 → 20. See the full cause limits. |
| First-turn execution errors at most 15% | failed | 60/100 → 35/100. Limit 15/100. |
| Precision at least 80%; lower paired 95% bound above original | failed | 36/67 (53.7%) → 57/111 (51.4%). Paired precision difference: −2.4 percentage points. Its 95% interval is −23.2 to +18.5 points. Zero undefined bootstrap draws. Target 80%, with the lower difference bound above zero. |
| Silent-wrong reduction at least 50%; lower 95% bound at least 30% | failed | 31/250 → 54/250. Relative reduction: −74.2% (an increase). The 95% interval is −235.3% to +2.3%. Zero undefined bootstrap draws. Secondary target. |
| Correct first turns at least 40% | failed | 14/100 → 18/100. Target 40/100. |
| Unnecessary clarification at most 5% after successful parents | passed | 0/62 → 0/106. Candidate rate 0.0%. One additional unnecessary clarification is on a first turn. None of 44 dependent turns after failed/refused parents offers the required recovery. |
| Full conversation success at least 30% | failed | 1/50 → 4/50. Target 15/50. |
| Zero correct-to-error-or-wrong regressions | failed | 7 correct-to-error-or-wrong transitions. V18 retains 24 of 36 original completions. Other losses remain visible. |
| Jobs at deadline at most 1% | passed | 0/250 → 0/250. Every submitted job remains in the denominator. |
| Zero unsupported numeric claims on curated narrative suite | passed | V18: 12 curated narrative cases, zero unsupported numeric output failures. This does not prove the generated query meaning. |
| All required result surfaces agree on facts and scope | unverified | Fresh native API, table, headline, snapshot CSV, authorized full-details share and reload agree. Native Compose and refreshed CSV after source changes remain unverified. |
| Deterministic p95 regression at most 10% | failed | Original 1549.449 ms → V18 1732.008 ms; +11.782%. All 180 timings retained. Fixed-query conformity 36/90 → 66/90. Adapter verification passes. |
31 findings. Every status retained.
F01Conversation ownershipfixed
- Current evidence
- Conversation ownership and auxiliary authorization pass in the unchanged full-suite gate. Fresh native browser calls use the declared synthetic vault.
- Root cause
- Endpoints did not consistently apply requesting-user and vault scoping before result access or dispatch. Auxiliary source authorization is a separate boundary.
- Change
- Retain owner/vault checks before dispatch and bind customization to the actual saved source.
- Regression checks
- ["test_insights_auxiliary_authorization.py", "test_insights_customization_source_identity.py", "test_bic007_bi_conversation_api.py:119", "test_bic007_bi_conversation_api.py:258"]
- Final result
- Fixed in the curated ownership and authorization controls. No claim of exhaustive security proof.
- Remaining limits
- Finite authorization tests pass. Production identity-provider integration was not deployed. Final limit: Fixed in the curated ownership and authorization controls. No claim of exhaustive security proof.
F02SQL authorizationfixed
- Current evidence
- All 568 curated critical controls pass, including SQL allowlist, source identity, masking, and authorization-before-dispatch tests.
- Root cause
- Relation and column extraction did not resolve qualified physical sources and their policies through lexical SQL scopes.
- Change
- Retain qualified AST validation and policy checks; do not bypass them for templates, repairs or follow-ups.
- Regression checks
- ["test_insights_query_correctness.py", "test_insights_auxiliary_authorization.py"]
- Final result
- Fixed in curated SQL authorization controls. Denials occur before dispatch.
- Remaining limits
- The repaired read-only fixture is verified. No customer permission configuration was queried. Final limit: Fixed in curated SQL authorization controls. Denials occur before dispatch.
F03Merge meaningpartially fixed
- Current evidence
- V18 query and merge gates pass. Native three-turn scoped aggregation is correct. The held run is exposing wrong source routing and incorrect count/listing shapes.
- Root cause
- Fallback dropped accepted membership obligations, while aggregation lacked proved physical row identity and join cardinality. See R01 and R02.
- Change
- Apply R01 and R02 through fallback, rescue and aggregation.
- Regression checks
- ["test_churn_tier_routing_and_merge_rescue.py", "test_prd004_federated_merge.py"]
- Final result
- Partially fixed. The held run still has wrong source membership and scalar requests that become per-client listings.
- Remaining limits
- Supported route proofs remain finite. A successful merge does not prove the whole question. Final limit: Partially fixed. The held run still has wrong source membership and scalar requests that become per-client listings.
F04Displayed valuespartially fixed
- Current evidence
- V18 native table, headline, snapshot CSV, share and reload agree on 45 clients, GBP 2454.50, and count 45. Explicit projection and ordering pass native PostgreSQL controls.
- Root cause
- Aggregation and serializers lost native unit lineage. Some views inferred measures from aliases before honoring authoritative types. See R05.
- Change
- Apply R05 and retain the 499/500/501/600-row, null, negative, zero/one, CSV and share checks.
- Regression checks
- ["test_insights_count_metadata.py", "js/bi-answer-values.test.mjs", "test_insights_v17_explicit_output.py", "test_insights_v17_output_recovery_native.py"]
- Final result
- Partially fixed. The six native cases agree on checked values, but 54/250 held answers are wrong and 21 comparisons are unverified.
- Remaining limits
- The fixed latency suite still returns a wrong settled-revenue field alias on three repeats. General model output correctness remains open. Final limit: Partially fixed. The six native cases agree on checked values, but 54/250 held answers are wrong and 21 comparisons are unverified.
F05Semantic proofpartially fixed
- Current evidence
- The 568 critical controls pass. Two held first-turn answers that score correct fail read-only synthetic semantic counterexamples: NULL role is treated as User; note person is substituted for deal person. Final held evidence: Independent native VALUES checks show NULL role treated as User (0 becomes 1), and note person used for deal person (2 becomes 1). Frozen labels stay unchanged.
- Root cause
- The verifier lost positional lineage and folded case-sensitive physical identifiers. Remaining cause: Generated SQL can coincide with one fixture value while using a different predicate or entity key.
- Change
- Trace positional outputs. Normalize only PostgreSQL unquoted names. Resolve aliases before checking physical identity. Unsupported quoted physical names return unverified.
- Regression checks
- ["test_insights_query_correctness.py", "test_insights_postgres_values.py", "test_insights_quoted_semantic_identity.py", "test_insights_v17_explicit_output.py"]
- Final result
- Partially fixed; general semantic proof is disproved. Two scorer-correct first turns fail independent SQL counterexamples. Eligibility, earliest-date and identity defects remain.
- Remaining limits
- Frozen numeric scores remain unchanged. Finite fixture equality is not complete semantic proof. Final limit: Partially fixed; general semantic proof is disproved. Two scorer-correct first turns fail independent SQL counterexamples. Eligibility, earliest-date and identity defects remain.
F06Job and parent protocolpartially fixed
- Current evidence
- All six native jobs finish. Thirty final reads start no new work. Refusal records agree. The 57 three/five-turn chain controls pass, with failures at each position.
- Root cause
- Full-thread context reloads, conflated completion/retrieval states and a separate migration ledger commit allowed inconsistent parent or retry state. See R04, R06 and R09.
- Change
- Apply R04, R06 and R09. Preserve atomic request keys and terminal-write protection.
- Regression checks
- ["test_bic015_async_turn_jobs.py", "test_insights_v8_chain_recovery.py"]
- Final result
- Partially fixed. All 250 jobs finish. Forty-four explicit dependent turns after a failed or refused parent bypass recovery.
- Remaining limits
- Full live conversation success remains a separate required measurement. The native check covers one successful three-turn chain. Final limit: Partially fixed. All 250 jobs finish. Forty-four explicit dependent turns after a failed or refused parent bypass recovery.
F07Early useful rowsfixed
- Current evidence
- Checked rows are readable 225.498 ms and 334.198 ms after native readiness during optional delays of at least 20 seconds.
- Root cause
- Required-ready persistence followed optional naming and prose. A caller waiting on a Future did not stop underlying transport work. See R07.
- Change
- Apply R07. Persist ready rows before optional work and expose separate follow-up readiness.
- Regression checks
- ["test_bic015_async_turn_jobs.py", "test_insights_shared_presentation_budget_review.py"]
- Final result
- Fixed within the measured native controls. Checked rows remain readable in 225.498 ms and 334.198 ms during actual optional delays of at least 20 seconds.
- Remaining limits
- These are two registered native observations. They are not p95 or pixel-paint timing. E6 fails its separate p95 target. Final limit: Fixed within the measured native controls. Checked rows remain readable in 225.498 ms and 334.198 ms during actual optional delays of at least 20 seconds.
F08Small status readsfixed
- Current evidence
- Thirty fresh native final reads create zero queries, provider calls, jobs, or usage rows. Saved refusals and the neutral ready footer agree.
- Root cause
- Status authorization and loading could hydrate result history before returning a small response. Metadata-only scoping and selected-turn retrieval now separate these reads.
- Change
- Retain the small status projection and selected-turn retrieval.
- Regression checks
- ["test_bic015_async_turn_jobs.py", "test_insights_answer_record.py"]
- Final result
- Fixed in final native controls. Thirty final reads start zero SQL, provider work or jobs.
- Remaining limits
- No production large-history payload benchmark was run. Final limit: Fixed in final native controls. Thirty final reads start zero SQL, provider work or jobs.
F09Freshness and provenancepartially fixed
- Current evidence
- The native stored snapshot retains checked GBP, NULL and source metadata through export, share and reload. All 397 UI verification inputs match the retained V16 bytes.
- Root cause
- Run time, source age and snapshot identity could be conflated.
- Change
- Keep source and snapshot metadata separate. Preserve unknown freshness and actual connection identity.
- Regression checks
- ["test_insights_result_source_provenance.py", "js/bi-federation-source.test.mjs", "js/bi-rowcap-gate.test.mjs"]
- Final result
- Partially fixed. Stored scope and provenance agree in native checks. Generated source predicates and entity identity remain fallible.
- Remaining limits
- Source watermarks, refreshed CSV after source changes, and atomic cross-source snapshots are not verified. Final limit: Partially fixed. Stored scope and provenance agree in native checks. Generated source predicates and entity identity remain fallible.
F10Empty-result recoverypartially fixed
- Current evidence
- Empty-result and failed saved-read recovery pass the retained mocked Chromium controls, bound to exact unchanged UI bytes. Fresh native blocked-parent recovery offers rerun or a named group.
- Root cause
- Empty views hid SQL inspection and recovery.
- Change
- Retain SQL access and clear next actions on empty results.
- Regression checks
- ["test_insights_v8_chain_recovery.py", "e2e/insights-review-fixes.spec.js"]
- Final result
- Partially fixed. Curated empty-result recovery passes. The held run still gives 24 unverified refusals and wrong membership answers.
- Remaining limits
- The six fresh native cases do not include empty successful results or an injected result-read outage. Final limit: Partially fixed. Curated empty-result recovery passes. The held run still gives 24 unverified refusals and wrong membership answers.
F11Accessibility and mobilepartially fixed
- Current evidence
- The 244 Node and 77 mocked Chromium passes are source-bound to unchanged V18 UI inputs. Fresh native desktop screenshots show the total, explicit refusal, and recovery.
- Root cause
- Some controls, focus paths and chart alternatives failed.
- Change
- Preserve focus and expose keyboard, table and mobile recovery paths.
- Regression checks
- ["e2e/insights-review-fixes.spec.js"]
- Final result
- Partially fixed. Retained mocked UI checks are bound to 397 unchanged inputs. Screen readers and physical touch remain unverified.
- Remaining limits
- No screen reader, physical touch, or human usability study was run. Final limit: Partially fixed. Retained mocked UI checks are bound to 397 unchanged inputs. Screen readers and physical touch remain unverified.
F12Bounded resourcespartially fixed
- Current evidence
- Shared generation and retry budgets pass final controls. All six native jobs finish. E6 preserves all 180 attempts and confirms the deterministic adapter.
- Root cause
- Connection and model work had incomplete timeout or admission evidence.
- Change
- Retain process-wide admission and bound local HTTP, DNS, SSH and PG work.
- Regression checks
- ["test_insights_v8_provider_attempt_deadline.py", "test_insights_shared_presentation_budget_review.py", "test_insights_pg_deadline.py"]
- Final result
- Partially fixed. Zero of 250 candidate jobs reaches the deadline. Twenty provider-boundary failures remain. Deterministic p95 regresses 11.782%.
- Remaining limits
- Remote inference termination and multiprocess capacity are not proved. E6 p95 rises 11.782%, above the 10% limit. Final limit: Partially fixed. Zero of 250 candidate jobs reaches the deadline. Twenty provider-boundary failures remain. Deterministic p95 regresses 11.782%.
F13Entry-point agreementpartially fixed
- Current evidence
- Fresh native API, table, headline, snapshot CSV, authorized full-details share and reload agree on checked values and scope. Two refusal outcomes agree across saved records and DOM.
- Root cause
- Some callers and serializers dropped actual source identity, unit_column or unit_unverified metadata. Shared execution services did not preserve those contracts by themselves.
- Change
- Propagate checked metadata through runtime summary callers, Compose, exports and shares.
- Regression checks
- ["test_full_result_export_enrichments.py", "test_bi_share_snapshot.py", "tests/js/bi-terminal-rendering.test.mjs"]
- Final result
- Partially fixed. Fresh native API, table, headline, snapshot CSV, authorized full-details share and reload agree. Compose and refreshed CSV after source changes remain unverified.
- Remaining limits
- Native Compose and refreshed CSV after source mutation remain unverified. Do not claim complete coverage of every required surface. Final limit: Partially fixed. Fresh native API, table, headline, snapshot CSV, authorized full-details share and reload agree. Compose and refreshed CSV after source changes remain unverified.
F14Independent answer evaluationpartially fixed
- Current evidence
- The repaired E5 baseline is complete. Final V18 is frozen and the sealed 250-attempt run is admitted. E6 is complete and failed. All final targets will retain fixed denominators. Final held evidence: Three final reviewers reproduce the metrics. Pipedrive employee values are 0 true, 52 false and 12 null; the ratified exclusion removes no rows.
- Root cause
- Passing structural tests did not establish correct values or model quality. Remaining cause: The fixture sensitivity audit covered 38 registered controls but missed a ratified staff predicate. Numeric equality and text separation do not prove complete semantic fidelity.
- Change
- Add real PostgreSQL oracles, frozen corpus splits, counterexample fixtures, live model runs and paired scoring.
- Regression checks
- ["test_insights_goal2_eval.py", "test_insights_goal2_exports.py", "test_insights_postgres_values.py"]
- Final result
- Partially fixed; independent quality certification is disproved. Counts and intervals reproduce. The fixture does not exercise every ratified predicate. All primary quality targets fail.
- Remaining limits
- All three requested reviewers completed final challenges. Their reviews are bounded and are not an external audit. Full fixture predicate sensitivity is incomplete. All primary quality targets fail. The holdout will never be used for another candidate.
N01Apply every ratified predicatefixed
- Current evidence
- Eight definition-schema controls pass for missing, restricted, renamed, and incomplete predicates. Both final arms use real definition retrieval and authorized compilation with the same deterministic adapter.
- Root cause
- Missing, restricted, renamed, and partially parameterized definition predicates are silently dropped by the compiler. The source parser also splits a three-part name incorrectly.
- Change
- Compile the exact ratified version against the authorized schema. Require all definition fields. Pass actual user and connection identities through each generation leg.
- Regression checks
- ["tests/test_insights_n01_definition_schema.py"]
- Final result
- Fixed within eight missing, restricted and renamed-column controls. A named definition cannot silently lose an unavailable column. Full fixture predicate sensitivity remains incomplete.
- Remaining limits
- The proof covers the registered ratified definitions and finite schema controls. It does not prove every generated predicate. Final limit: Fixed within eight missing, restricted and renamed-column controls. A named definition cannot silently lose an unavailable column. Full fixture predicate sensitivity remains incomplete.
N02Classify terminal failurespartially fixed
- Current evidence
- Final terminal contract controls pass. Native write and grain refusals preserve class and outcome. The legacy generic outward error sentence is replaced by a symmetric presentation adapter in both evaluation arms. Final held evidence: The result reviewer checks 117 answer records and 114 present confidence records: zero outcome parity mismatches, all 117 classified, zero generic crashes.
- Root cause
- The legacy error model has no required class. Router catches and wrappers lose typed causes. Remaining cause: Several terminal paths retain a broad category without the precise generated-SQL cause. The original implementation still lacks runtime classes.
- Change
- A shared nine-class terminal contract now reaches API errors and persisted turns. A new append-only migration stores the class and recovery actions.
- Regression checks
- ["tests/test_insights_terminal_outcomes.py", "tests/test_insights_http_terminal_contract.py", "tests/test_insights_v9_contract_review.py", "tests/test_insights_v8_terminal_persistence.py"]
- Final result
- Partially fixed. Candidate terminal errors are classified and no outward generic crash text remains. Original has 21 unclassified errors; some candidate causes remain too general.
- Remaining limits
- Original 21 runtime-unclassified error scores remain. A category-level message can still omit the detailed SQL or planning cause. The both-arm target cannot pass retroactively. Final limit: Partially fixed. Candidate terminal errors are classified and no outward generic crash text remains. Original has 21 unclassified errors; some candidate causes remain too general.
N03Render safe refusals as answerspartially fixed
- Current evidence
- V18 moves money guards into the existing bounded semantic correction path and preserves typed refusals after exhaustion. Native write and grain refusals agree across saved and rendered records.
- Root cause
- A correct guard refusal is stored or rendered with the legacy execution-error path.
- Change
- Typed guard outcomes now retain refusal status and suggested rephrases. The browser uses the refusal outcome even on old error-status records.
- Regression checks
- ["tests/test_insights_terminal_outcomes.py", "tests/test_insights_v18_money_guard_recovery.py"]
- Final result
- Partially fixed. Native permission and grain refusals carry a class, reason and suggested question. All 117 candidate terminal records have matching outcome projections. The 24 held refusals do not earn completion credit.
- Remaining limits
- A safe refusal does not earn completion credit for a supported question. Live unresolved refusals remain in the final denominators. Final limit: Partially fixed. Native permission and grain refusals carry a class, reason and suggested question. All 117 candidate terminal records have matching outcome projections. The 24 held refusals do not earn completion credit.
N04Execute supported first turnspartially fixed
- Current evidence
- E5 has 60 product-caused first-turn errors out of 100. Final source guards and execution controls pass, but the held run still exposes source-routing and generated SQL failures. Final held evidence: Completed E5 cause audit attributes all 60 original first-turn errors to product causes. V18 first-turn errors are 18 SQL, nine schema and eight semantic.
- Root cause
- 60/100 supported first turns failed:36 unusable leg plans,10 unavailable generated schema references,7 missing merge keys,5 missing source qualification,1 empty generation,1 wrong function input type. All60 are product causes. Remaining cause: Generated SQL, source selection and result-shape planning remain unreliable on unfamiliar first-turn questions.
- Change
- Product fixes target unusable plan SQL, fabricated schema names, missing join keys, source qualification, and wrong function input types.
- Regression checks
- ["tests/test_insights_cold_schema_context.py", "tests/test_insights_v18_money_guard_recovery.py"]
- Final result
- Partially fixed. First-turn execution errors fall from 60/100 to 35/100. Correct first turns rise from 14/100 to 18/100. Both absolute targets fail.
- Remaining limits
- V15 development first-turn correct was 10/28; V17 was 7/28. Both were below 40%. Final held metrics remain separate. Final limit: Partially fixed. First-turn execution errors fall from 60/100 to 35/100. Correct first turns rise from 14/100 to 18/100. Both absolute targets fail.
N05Prevent correct-answer regressionspartially fixed
- Current evidence
- The three retained v10 regression families have scoped repairs. V17 preserved the sole correct original development case. Final V18 paired transition and completion-retention gates are required. Final held evidence: The complete retention table lists all 12 lost original completions. No refusal, clarification or blocked turn is counted as a retained answer.
- Root cause
- Write intent after a scope clause escapes the refusal gate. Scalar gold-leg planner support is not wired in the router. Other per-case causes remain under diagnosis. Remaining cause: New routing, shape and guard decisions can reject or misinterpret questions the original answered correctly.
- Change
- Scope-prefixed write checks and Gold final-answer routing are implemented. The paired regression gate retains every original outcome and unavailable export action.
- Regression checks
- ["tests/test_insights_v8_intent_shape_diagnostics.py", "test_insights_v17_rank_thresholds.py"]
- Final result
- Partially fixed; the paired regression gate fails. Seven original correct answers become three errors and four wrong answers. Only 24/36 original completions are retained; four refusals and one clarification account for the other five losses.
- Remaining limits
- All losses to refusal, blocking and clarification must be shown beside the narrow correct-to-error-or-wrong gate. Final limit: Partially fixed; the paired regression gate fails. Seven original correct answers become three errors and four wrong answers. Only 24/36 original completions are retained; four refusals and one clarification account for the other five losses.
N06Repair newly executable wrong answerspartially fixed
- Current evidence
- The retained 24 new-wrong cases have counterexamples and shared source/membership repairs. V18 fixes NULL-name predicate bypass and requested membership projection. The held run still contains wrong source and output shapes. Final held evidence: Independent review reproduces dropped marketing eligibility, earliest-to-any verification date and scalar counts returned as per-client listings.
- Root cause
- Source membership, union and intersection routes, missing values, and tie semantics share query-meaning failures. Remaining cause: Semantic guards do not fully bind explicit question predicates, temporal operators and output operations to the executed SQL.
- Change
- Counterexamples are retained. V10 implements proved identity bridging and source membership fixes, with exact-value PostgreSQL controls in progress.
- Regression checks
- ["test_insights_v17_membership_predicates.py", "tests/test_insights_v18_membership_output.py", "tests/test_insights_v8_ct_input_regressions.py"]
- Final result
- Partially fixed. Silent wrong answers rise from 31/250 to 54/250. Wrong filters, earliest-date logic, source membership and output shape remain.
- Remaining limits
- Two numerically correct held first turns also have proved semantic counterexamples. No score or application change follows this diagnosis. Final limit: Partially fixed. Silent wrong answers rise from 31/250 to 54/250. Wrong filters, earliest-date logic, source membership and output shape remain.
N07Bound attempts and provider retriespartially fixed
- Current evidence
- The final controls bound total generation and one provider retry per turn. E6 confirms all deterministic external calls. Native jobs finish and the held provider/job ledger is retained. Final held evidence: The independent reviewer recomputes all 180 E6 timings. Candidate p95 is 1732.008 ms versus 1549.449 ms. No failed or refused timing is removed.
- Root cause
- Each of the five timeout jobs spent about 179 seconds in its first failed provider call. The request lacks an absolute deadline within the job budget. Remaining cause: Bounded attempts stop parent deadlines but do not remove slow generation or optional cancellation. SDK errors do not by themselves establish a remote outage.
- Change
- All provider calls share a six-attempt turn budget and one provider retry. The HTTP transport now enforces an absolute request deadline. Propagate the typed CT future failure when Gold returns no usable SQL.
- Regression checks
- ["test_insights_v8_provider_attempt_deadline.py", "test_insights_shared_presentation_budget_review.py", "test_insights_v16_provider_drain.py"]
- Final result
- Partially fixed. All 250 candidate jobs finish without a deadline. SQL generation makes 453 boundary calls, with 11 SDK failures; 118 auxiliary calls have nine local cancellation failures. E6 p95 fails.
- Remaining limits
- A failed SDK call may reflect provider availability, a timeout, or cancellation. Remote cause is not proved by exception type alone. E6 p95 fails. Final limit: Partially fixed. All 250 candidate jobs finish without a deadline. SQL generation makes 453 boundary calls, with 11 SDK failures; 118 auxiliary calls have nine local cancellation failures. E6 p95 fails.
N08Preserve and recover conversation scopepartially fixed
- Current evidence
- All 57 three/five-turn recovery controls pass, including a failure at each position. The fresh native three-turn chain carries exactly 45 clients. Final held evidence: The result reviewer calls both frozen scope detectors for all 44 child questions. Both detectors return false for every checked first/third-result or ID reference.
- Root cause
- The anchor search skips a blocked selected parent and can reuse an older group. Sibling how-many follow-ups also conflict with ratified listing continuity. Remaining cause: The reference recognizers miss explicit variants such as first checked result and checked IDs. These turns bypass blocked-parent recovery and checked membership carry.
- Change
- A non-successful selected parent blocks older-group carry. Recovery offers a rerun or a named group. Successful saved counts retain exact checked membership. Mask only the proved aggregate NULL-policy group token before source recognition, preserving offsets and genuine account-group terms.
- Regression checks
- ["test_bi_followup_router.py", "test_insights_v16_aggregate_group_scope.py", "test_insights_v17_output_recovery_native.py", "test_insights_v8_chain_recovery.py", "test_insights_v8_context_contract.py", "tests/test_insights_v18_membership_output.py"]
- Final result
- Partially fixed. Fifty-seven curated chain controls pass, but held full-conversation success is only 4/50 versus 1/50. All 44 explicit continuations after failed/refused parents bypass recovery: 31 errors, nine refusals and four wrong answers.
- Remaining limits
- Full held conversation success is required. Wrong parent answers can propagate a checked but semantically wrong group. Final limit: Partially fixed. Fifty-seven curated chain controls pass, but held full-conversation success is only 4/50 versus 1/50. All 44 explicit continuations after failed/refused parents bypass recovery: 31 errors, nine refusals and four wrong answers.
R01Keep the complete selected grouppartially fixed
- Current evidence
- V18 native intersection and union controls preserve requested aliases, ordering before capping, and complete internal member keys. Native listing, scoped total and count keep the same 45 clients.
- Root cause
- Fallback rerouted an isolated follow-up and could discard independent source membership. Empty-cohort rescue used the populated leg without sufficient ownership proof. An eligible CT fallback also bypassed final-answer shape validation when promoted from a merge input.
- Change
- Carry accepted route and exact membership. Retain independent Gold predicates unless the narrow CT recency rescue proves they can be superseded. Require a complete user_id cohort before an explicit snapshot money source change. Recheck actual executed SQL against the final answer shape before either eligible CT-only promotion.
- Regression checks
- ["test_answer_contract_role_promotion.py", "test_churn_tier_routing_and_merge_rescue.py", "test_federated_semantic_refusal.py", "test_insights_federated_followup_contract.py", "test_insights_v16_rank_words.py", "test_insights_v17_rank_thresholds.py", "test_prd008_federated_planner.py", "tests/test_insights_v18_membership_output.py"]
- Final result
- Partially fixed. Native controls preserve 45 checked clients. Held dependent turns can miss the checked-group reference and use the wrong source or group.
- Remaining limits
- The proof covers supported membership routes. Held failures show that arbitrary generated source predicates are not solved. Final limit: Partially fixed. Native controls preserve 45 checked clients. Held dependent turns can miss the checked-group reference and use the wrong source or group.
R02Prove row identity before summingpartially fixed
- Current evidence
- V18 recognizes a proved complete DISTINCT key tuple. Partial tuples and set-returning projections still refuse. Native deal sums pass with distinct members and equal-valued entities.
- Root cause
- Aggregation used repeated numeric values without establishing physical source identity or join cardinality; equal values do not identify equal entities.
- Change
- Derive bounded source-key, join-cardinality and aggregation contracts from physical lineage. Reject repeated/missing physical keys and unproved deal SUM joins. Keep separate equal-valued entities and apply only declared additive or denominator/member-aware reductions.
- Regression checks
- ["test_insights_federated_followup_contract.py", "test_insights_postgres_values.py", "test_insights_query_correctness.py", "test_insights_v17_distinct_snapshot_sum.py", "test_prd004_federated_merge.py", "tests/test_insights_v18_distinct_membership_grain.py"]
- Final result
- Partially fixed. Curated grain controls pass. A scorer-correct count uses note person identity where deal person identity is required.
- Remaining limits
- This is finite uniqueness and lineage proof. It does not validate every SQL aggregate or join. Final limit: Partially fixed. Curated grain controls pass. A scorer-correct count uses note person identity where deal person identity is required.
R03Follow physical column lineagepartially fixed
- Current evidence
- Quoted identity, physical lineage, DISTINCT snapshot sums, and V18 complete-tuple controls pass on the final source.
- Root cause
- Predicate matching lost positional-output lineage and conflated PostgreSQL case-sensitive quoted physical columns/tables with lowercase identifiers.
- Change
- Normalize only unquoted PostgreSQL identifiers. Resolve CTE/derived aliases and positional outputs to physical columns before comparing required root/schema/table/column identity. Return unverified for unsupported quoted physical shapes.
- Regression checks
- ["test_insights_quoted_semantic_identity.py", "test_insights_postgres_values.py", "test_insights_query_correctness.py", "test_insights_v17_distinct_snapshot_sum.py"]
- Final result
- Partially fixed. Curated physical-lineage controls pass. Held query generation still drops filters and chooses incorrect entity keys.
- Remaining limits
- Semantic-obligation metadata is not universal execution authorization or whole-question proof. Final limit: Partially fixed. Curated physical-lineage controls pass. Held query generation still drops filters and chooses incorrect entity keys.
R04Use the captured parentpartially fixed
- Current evidence
- The 57 final chain controls use the captured parent. Fresh native listing, scoped total and count retain checked membership.
- Root cause
- Context helpers could reload later thread turns instead of captured ancestry; explicit-null roots and omitted parents shared a fallback. Exact saved SQL reruns could also inherit a newer parent or changed request question.
- Change
- Resolve only valid captured-parent ancestry. Preserve explicit roots. For exact saved SQL or merge replay, pin the saved question/recognizer/resolution and saved source choices, and derive no new follow-up/cohort context from a later parent. Keep ordinary follow-ups and confirmations on their captured parent.
- Regression checks
- ["test_sql_anchor_turn_polluted_thread.py", "test_insights_v8_chain_recovery.py"]
- Final result
- Partially fixed. Captured-parent tests pass, but all 44 explicit continuations after failed/refused parents miss the minimal recovery gate.
- Remaining limits
- Arbitrary branch language and full live conversation correctness remain separate. Final limit: Partially fixed. Captured-parent tests pass, but all 44 explicit continuations after failed/refused parents miss the minimal recovery gate.
R05Keep units and non-measure typespartially fixed
- Current evidence
- Final checked-fact narrative and authoritative value metadata controls pass. Native GBP values and NULLs agree across the observed surfaces.
- Root cause
- Unit lineage was lost in aggregation and rendering. Currency proof compared names without complete source/physical lineage. Some views inferred measure types from aliases before honoring metadata.
- Change
- Follow each amount through lexical CTE and derived-table lineage. Require the physical currency or rate from the same source instance. Preserve native unit bindings and authoritative types in every view, including sorted rows. Reject unresolved reduction; retain valid converted CTEs.
- Regression checks
- ["test_insights_count_metadata.py", "js/bi-answer-values.test.mjs", "test_insights_result_source_provenance.py"]
- Final result
- Partially fixed. Native GBP scope and values agree. Mixed-currency browser output and all generated measure choices remain unverified.
- Remaining limits
- Mixed-currency native browser behavior remains unverified. The latency alias mismatch is retained. Final limit: Partially fixed. Native GBP scope and values agree. Mixed-currency browser output and all generated measure choices remain unverified.
R06Recover result reads without another querypartially fixed
- Current evidence
- Thirty native saved-answer reads start zero new execution. Retained source-bound Chromium controls cover an unavailable saved-result read.
- Root cause
- The UI treated a done job as a retrieved answer. Optional full-thread refresh could replace accepted local reading and ancestry state.
- Change
- Retry the stored result read. Show a recoverable result-ready state. Preserve the selected view, disclosures, focus and current thread state.
- Regression checks
- ["test_bic015_async_turn_jobs.py", "e2e/insights-review-fixes.spec.js"]
- Final result
- Partially fixed. Final native reads start no new execution. Broader surface coverage remains incomplete.
- Remaining limits
- The fresh native six-case path does not inject a read outage. Final limit: Partially fixed. Final native reads start no new execution. Broader surface coverage remains incomplete.
R07Bound optional work and actual local resourcespartially fixed
- Current evidence
- Native checked rows remain readable during both 20-second optional delays. The final full suite binds provider deadlines, shared budgets and local cancellation tests.
- Root cause
- Required-ready publication followed optional naming. Inactivity/Future timeouts did not stop underlying provider resources. A cancel acknowledgment could block.
- Change
- Publish checked rows before optional presentation. Kill and reap local provider transports. Close canceled PG connections and retain server deadlines.
- Regression checks
- ["test_insights_shared_presentation_budget_review.py", "test_insights_v8_provider_attempt_deadline.py", "test_insights_pg_deadline.py"]
- Final result
- Partially fixed. Optional 20-second work does not block the two native row observations. All 180 E6 timings show an 11.782% p95 regression.
- Remaining limits
- Two DOM readiness observations are not p95. Remote resource use and cross-process capacity remain unverified. Final limit: Partially fixed. Optional 20-second work does not block the two native row observations. All 180 E6 timings show an 11.782% p95 regression.
R08Keep the requested calendarfixed
- Current evidence
- Typed calendar guards, date parameters, calendar operands and real PostgreSQL calendar controls pass on frozen V18 within the 568 critical checks.
- Root cause
- The earlier rule assumed every CT timestamp was naive UTC, did not consistently preserve an explicit requested zone through postpasses, and could leave incompatible timestamp operands/anchors. SQL text alone cannot establish a physical timestamp storage type.
- Change
- Use the same authorized current CT connection schema types and requested question in generation/repair and final validation. Naive UTC columns become an instant then the selected wall clock; timestamptz becomes that wall clock directly. Normalize compatible bounds and preserve absolute instant ordering. Refuse unsupported/missing typed lineage and meaning-changing approved SQL rather than rewriting an exact approved statement.
- Regression checks
- ["test_insights_typed_calendar_guard.py", "test_insights_query_correctness.py", "test_insights_postgres_values.py"]
- Final result
- Fixed within curated calendar controls. These finite controls do not establish all live date interpretations.
- Remaining limits
- Arbitrary SQL calendar expressions remain outside the finite proof. Final limit: Fixed within curated calendar controls. These finite controls do not establish all live date interpretations.
R09Commit schema and ledger togetherfixed
- Current evidence
- All nine migration recovery controls pass on V18. Schema and ledger remain atomic across injected failure and restart. Native migrated state persists terminal classes and suggestions.
- Root cause
- A transaction-bearing migration could commit schema changes before the runner inserted its ledger row, so restart repeated non-idempotent DDL.
- Change
- For supported explicitly transactional scripts, insert the migration ledger immediately before their COMMIT in the same transaction; preserve trigger bodies and PRAGMA boundaries. Reject unsupported transaction layouts. Leave ordinary scripts on their existing runner path.
- Regression checks
- ["test_insights_migration_recovery.py", "test_insights_result_source_provenance.py"]
- Final result
- Fixed in nine final migration recovery controls, including injected pre-ledger and post-commit crashes.
- Remaining limits
- No universal claim covers every production database history. Final limit: Fixed in nine final migration recovery controls, including injected pre-ledger and post-commit crashes.
What Insights users will notice
- Errors have a named class. The native write and grain checks show an explicit refusal, its reason, and a suggested read-only question.
- Recognized follow-ups can reuse the exact checked client group. Curated blocked-parent forms offer a rerun or ask for the group name. All 44 held dependent turns after a failed/refused parent miss this recovery.
- Missing or unauthorized columns in a named ratified definition cause an explicit schema refusal in the verified controls.
- Saved results remain readable during optional work. Reading a final answer does not start another query.
- First-turn reliability and meaning remain inadequate on this holdout. Some count requests return listings. Some source predicates or entity keys are wrong. Users still need to inspect successful answers.
Method and independence
- The comparison uses original e479e81 and one frozen final candidate. The locked set has 250 attempts, 100 supported first turns and 50 conversations.
- The date anchor is 2026-09-06T12:00:00Z. Process time is UTC. The fixture database and default business calendar use Europe/London. Explicit requested timezones remain part of the question contract. Schema, policies, provider configuration and scorer are frozen.
- Paired uncertainty uses 50 intent-family clusters and 10,000 bootstrap repeats. The seed is 2026090611. Undefined draws stay visible. Conditional intervals cannot certify a target.
- Precision is correct completed divided by correct completed plus silent wrong. Refusals, blocked turns and comparison-unverified answers are shown separately. They earn no completed-answer credit.
- All original-to-candidate case transitions are included. Correct-to-error or wrong regressions and other lost original completions are reported separately.
- Two pre-admission policy annexes restore ratified open-deal archive and booking-deletion predicates to semantic review. All 15 affected expected results are unchanged. The original inputs, scores and denominators are preserved.
- The old V10 holdout is opened development evidence. It supports diagnosis only. No quality claim uses that invalid run.
- The original viability gate failed. The user accepted the repaired environment and made absolute correctness targets primary. Relative silent-wrong reduction remains secondary.
- The final V18 application was frozen before the sealed candidate run. The isolated E6 run completed first. The live holdout ran once. No V19 or post-holdout tuning was admitted.
- The two numerical semantic counterexamples were found by the lead after admission and reproduced by the independent query reviewer. They do not amend the scorer or add another scored attempt. The 250-attempt denominator is unchanged.
- V18 was selected at 09:37:17 UTC on 7 September, before the 11:00 deadline. Its pre-freeze scoped gate passed 1,162 checks. The final full suite later retained 21 pre-existing failures and no new failed nodes. The sole live candidate holdout ran from 09:50:28 to 10:28:12 UTC.
Remaining limits
- A late read-only audit found zero true pd_is_ct_employee rows among 64 Pipedrive people. The ratified staff-exclusion predicate changes no answer. Full E1 predicate coverage is incomplete. This run cannot certify independent quality improvement. Numerical gate results are retained observations, not a validity certificate.
- The stated correctness goal is not achieved. Failed targets and partially fixed findings remain open. V18 stays frozen under the user stop instruction.
- All three requested V18 reviewers completed their final challenge after earlier service limits cleared. They found material defects. Their bounded same-model reviews are not an external audit or exhaustive semantic approval.
- The evaluator authored the holdout apart from the implementers. This is not an external human evaluation. There are no duplicate texts across families, but shared SQL operators and follow-up forms recur.
- At least two scored-correct first turns fail separate semantic counterexamples. Reported precision is the frozen scorer measurement, not proof of complete question meaning.
- Some expected SQL and domain defaults need further semantic reconciliation. The complete inputs and separate policy annexes remain visible. No original expected values or numeric labels are changed.
- Refusals and blocked turns reduce completion. V15 to V17 development correctness fell from 10/28 to 7/28 while refusals rose from 20/70 to 25/70. Both completion rates miss 40%.
- The original has 21 runtime-unclassified errors. A later audit names their causes but cannot satisfy the both-arm zero-unclassified target.
- The 44 follow-ups after failed or refused parents produce 31 task errors, nine refusals and four wrong answers. They refer to the first/third checked result or checked IDs, but both scope detectors miss these forms. Zero blocked_by_parent labels is not evidence of recovery. No post-holdout fix is allowed.
- A raw SDK connection or timeout error does not establish remote cause. Every failed provider-boundary call is counted. Optional cancellation and remote outage may be indistinguishable from retained traces.
- The full suite retains 21 pre-existing failures and 50 skips. Live-model results, deterministic tests, mocked UI checks, and fresh native browser observations are separate evidence.
- The native surface check uses GBP and a separate structural fixture. Native Compose, refreshed CSV after source changes, mixed-currency native browser output, screen readers and physical touch remain unverified.
- E6 fails the p95 target. Two native readiness observations do not override it. Live-model latency is advisory because host load and provider responses are not controlled.
- Only review artifacts are published. No application changes are deployed, pushed, merged, or rebased.
Every paired case
All 250 attempts remain in each arm.
| Case | Family | Original | Candidate | Review note |
|---|---|---|---|---|
| independent11:activity_deadline_before_creation:first-count | independent11:activity_deadline_before_creation | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_deadline_before_creation:t1 | independent11:activity_deadline_before_creation | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_deadline_before_creation:t2 | independent11:activity_deadline_before_creation | comparison_unverified | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_deadline_before_creation:t3 | independent11:activity_deadline_before_creation | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_deadline_before_creation:t4 | independent11:activity_deadline_before_creation | comparison_unverified | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_duration_missing_due_time:first-count | independent11:activity_duration_missing_due_time | task_error | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_duration_missing_due_time:t1 | independent11:activity_duration_missing_due_time | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_duration_missing_due_time:t2 | independent11:activity_duration_missing_due_time | comparison_unverified | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_duration_missing_due_time:t3 | independent11:activity_duration_missing_due_time | correct_completed | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:activity_duration_missing_due_time:t4 | independent11:activity_duration_missing_due_time | comparison_unverified | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:all_commission_unknown:first-count | independent11:all_commission_unknown | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:all_commission_unknown:t1 | independent11:all_commission_unknown | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:all_commission_unknown:t2 | independent11:all_commission_unknown | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:all_commission_unknown:t3 | independent11:all_commission_unknown | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:all_commission_unknown:t4 | independent11:all_commission_unknown | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:booked_without_settled_history:first-count | independent11:booked_without_settled_history | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:booked_without_settled_history:t1 | independent11:booked_without_settled_history | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:booked_without_settled_history:t2 | independent11:booked_without_settled_history | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:booked_without_settled_history:t3 | independent11:booked_without_settled_history | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:booked_without_settled_history:t4 | independent11:booked_without_settled_history | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:campaign_and_web_referral:first-count | independent11:campaign_and_web_referral | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:campaign_and_web_referral:t1 | independent11:campaign_and_web_referral | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:campaign_and_web_referral:t2 | independent11:campaign_and_web_referral | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:campaign_and_web_referral:t3 | independent11:campaign_and_web_referral | silent_wrong | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:campaign_and_web_referral:t4 | independent11:campaign_and_web_referral | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:corporate_registration_without_trade:first-count | independent11:corporate_registration_without_trade | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:corporate_registration_without_trade:t1 | independent11:corporate_registration_without_trade | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:corporate_registration_without_trade:t2 | independent11:corporate_registration_without_trade | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:corporate_registration_without_trade:t3 | independent11:corporate_registration_without_trade | silent_wrong | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:corporate_registration_without_trade:t4 | independent11:corporate_registration_without_trade | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:customer_referrals_with_reward_event:first-count | independent11:customer_referrals_with_reward_event | correct_completed | unnecessary_clarification | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:customer_referrals_with_reward_event:t1 | independent11:customer_referrals_with_reward_event | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:customer_referrals_with_reward_event:t2 | independent11:customer_referrals_with_reward_event | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:customer_referrals_with_reward_event:t3 | independent11:customer_referrals_with_reward_event | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:customer_referrals_with_reward_event:t4 | independent11:customer_referrals_with_reward_event | correct_completed | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deadline_actual_date_disagreement:first-count | independent11:deadline_actual_date_disagreement | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deadline_actual_date_disagreement:t1 | independent11:deadline_actual_date_disagreement | silent_wrong | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deadline_actual_date_disagreement:t2 | independent11:deadline_actual_date_disagreement | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deadline_actual_date_disagreement:t3 | independent11:deadline_actual_date_disagreement | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deadline_actual_date_disagreement:t4 | independent11:deadline_actual_date_disagreement | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_assignment_override:first-count | independent11:deal_assignment_override | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_assignment_override:t1 | independent11:deal_assignment_override | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_assignment_override:t2 | independent11:deal_assignment_override | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_assignment_override:t3 | independent11:deal_assignment_override | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_assignment_override:t4 | independent11:deal_assignment_override | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_person_organization_conflict:first-count | independent11:deal_person_organization_conflict | silent_wrong | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_person_organization_conflict:t1 | independent11:deal_person_organization_conflict | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_person_organization_conflict:t2 | independent11:deal_person_organization_conflict | silent_wrong | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_person_organization_conflict:t3 | independent11:deal_person_organization_conflict | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_person_organization_conflict:t4 | independent11:deal_person_organization_conflict | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_stage_timestamp_order:first-count | independent11:deal_stage_timestamp_order | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_stage_timestamp_order:t1 | independent11:deal_stage_timestamp_order | correct_completed | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_stage_timestamp_order:t2 | independent11:deal_stage_timestamp_order | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_stage_timestamp_order:t3 | independent11:deal_stage_timestamp_order | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deal_stage_timestamp_order:t4 | independent11:deal_stage_timestamp_order | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deleted_booking_live_account:first-count | independent11:deleted_booking_live_account | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deleted_booking_live_account:t1 | independent11:deleted_booking_live_account | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deleted_booking_live_account:t2 | independent11:deleted_booking_live_account | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deleted_booking_live_account:t3 | independent11:deleted_booking_live_account | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:deleted_booking_live_account:t4 | independent11:deleted_booking_live_account | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:different_sales_ownership:first-count | independent11:different_sales_ownership | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:different_sales_ownership:t1 | independent11:different_sales_ownership | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:different_sales_ownership:t2 | independent11:different_sales_ownership | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:different_sales_ownership:t3 | independent11:different_sales_ownership | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:different_sales_ownership:t4 | independent11:different_sales_ownership | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:first_booked_before_first_broker:first-count | independent11:first_booked_before_first_broker | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:first_booked_before_first_broker:t1 | independent11:first_booked_before_first_broker | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:first_booked_before_first_broker:t2 | independent11:first_booked_before_first_broker | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:first_booked_before_first_broker:t3 | independent11:first_booked_before_first_broker | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:first_booked_before_first_broker:t4 | independent11:first_booked_before_first_broker | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:funnel_incomplete_contact:first-count | independent11:funnel_incomplete_contact | silent_wrong | correct_completed | The candidate treats a NULL role as an ordinary User. The question and reference require role User. This input changes the count from zero to one. |
| independent11:funnel_incomplete_contact:t1 | independent11:funnel_incomplete_contact | silent_wrong | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:funnel_incomplete_contact:t2 | independent11:funnel_incomplete_contact | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:funnel_incomplete_contact:t3 | independent11:funnel_incomplete_contact | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:funnel_incomplete_contact:t4 | independent11:funnel_incomplete_contact | silent_wrong | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:fx_missing_on_settled:first-count | independent11:fx_missing_on_settled | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:fx_missing_on_settled:t1 | independent11:fx_missing_on_settled | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:fx_missing_on_settled:t2 | independent11:fx_missing_on_settled | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:fx_missing_on_settled:t3 | independent11:fx_missing_on_settled | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:fx_missing_on_settled:t4 | independent11:fx_missing_on_settled | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:invoice_before_settlement:first-count | independent11:invoice_before_settlement | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:invoice_before_settlement:t1 | independent11:invoice_before_settlement | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:invoice_before_settlement:t2 | independent11:invoice_before_settlement | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:invoice_before_settlement:t3 | independent11:invoice_before_settlement | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:invoice_before_settlement:t4 | independent11:invoice_before_settlement | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_external_person_qualification:first-count | independent11:known_external_person_qualification | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_external_person_qualification:t1 | independent11:known_external_person_qualification | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_external_person_qualification:t2 | independent11:known_external_person_qualification | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_external_person_qualification:t3 | independent11:known_external_person_qualification | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_external_person_qualification:t4 | independent11:known_external_person_qualification | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_zero_booked_commission:first-count | independent11:known_zero_booked_commission | silent_wrong | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_zero_booked_commission:t1 | independent11:known_zero_booked_commission | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_zero_booked_commission:t2 | independent11:known_zero_booked_commission | correct_completed | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_zero_booked_commission:t3 | independent11:known_zero_booked_commission | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:known_zero_booked_commission:t4 | independent11:known_zero_booked_commission | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:london_signup_month_edge:first-count | independent11:london_signup_month_edge | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:london_signup_month_edge:t1 | independent11:london_signup_month_edge | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:london_signup_month_edge:t2 | independent11:london_signup_month_edge | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:london_signup_month_edge:t3 | independent11:london_signup_month_edge | silent_wrong | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:london_signup_month_edge:t4 | independent11:london_signup_month_edge | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:marketing_registration_gap:first-count | independent11:marketing_registration_gap | refusal_unverified | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:marketing_registration_gap:t1 | independent11:marketing_registration_gap | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:marketing_registration_gap:t2 | independent11:marketing_registration_gap | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:marketing_registration_gap:t3 | independent11:marketing_registration_gap | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:marketing_registration_gap:t4 | independent11:marketing_registration_gap | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:multi_broker_activation:first-count | independent11:multi_broker_activation | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:multi_broker_activation:t1 | independent11:multi_broker_activation | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:multi_broker_activation:t2 | independent11:multi_broker_activation | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:multi_broker_activation:t3 | independent11:multi_broker_activation | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:multi_broker_activation:t4 | independent11:multi_broker_activation | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:negative_realized_commission:first-count | independent11:negative_realized_commission | correct_completed | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:negative_realized_commission:t1 | independent11:negative_realized_commission | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:negative_realized_commission:t2 | independent11:negative_realized_commission | correct_completed | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:negative_realized_commission:t3 | independent11:negative_realized_commission | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:negative_realized_commission:t4 | independent11:negative_realized_commission | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:note_author_deal_owner_difference:first-count | independent11:note_author_deal_owner_difference | task_error | correct_completed | The candidate counts the person attached to the note. The reference counts the person attached to the deal. Independent foreign keys do not make those identities equal. Two deal people can share one note person. |
| independent11:note_author_deal_owner_difference:t1 | independent11:note_author_deal_owner_difference | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:note_author_deal_owner_difference:t2 | independent11:note_author_deal_owner_difference | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:note_author_deal_owner_difference:t3 | independent11:note_author_deal_owner_difference | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:note_author_deal_owner_difference:t4 | independent11:note_author_deal_owner_difference | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:notified_without_verification:first-count | independent11:notified_without_verification | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:notified_without_verification:t1 | independent11:notified_without_verification | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:notified_without_verification:t2 | independent11:notified_without_verification | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:notified_without_verification:t3 | independent11:notified_without_verification | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:notified_without_verification:t4 | independent11:notified_without_verification | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:org_contact_without_website:first-count | independent11:org_contact_without_website | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:org_contact_without_website:t1 | independent11:org_contact_without_website | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:org_contact_without_website:t2 | independent11:org_contact_without_website | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:org_contact_without_website:t3 | independent11:org_contact_without_website | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:org_contact_without_website:t4 | independent11:org_contact_without_website | silent_wrong | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_client_type_multiple:first-count | independent11:organization_client_type_multiple | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_client_type_multiple:t1 | independent11:organization_client_type_multiple | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_client_type_multiple:t2 | independent11:organization_client_type_multiple | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_client_type_multiple:t3 | independent11:organization_client_type_multiple | task_error | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_client_type_multiple:t4 | independent11:organization_client_type_multiple | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_employee_data_gap:first-count | independent11:organization_employee_data_gap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_employee_data_gap:t1 | independent11:organization_employee_data_gap | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_employee_data_gap:t2 | independent11:organization_employee_data_gap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_employee_data_gap:t3 | independent11:organization_employee_data_gap | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_employee_data_gap:t4 | independent11:organization_employee_data_gap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_erp_multiple:first-count | independent11:organization_erp_multiple | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_erp_multiple:t1 | independent11:organization_erp_multiple | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_erp_multiple:t2 | independent11:organization_erp_multiple | silent_wrong | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_erp_multiple:t3 | independent11:organization_erp_multiple | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_erp_multiple:t4 | independent11:organization_erp_multiple | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_hedging_multiple:first-count | independent11:organization_hedging_multiple | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_hedging_multiple:t1 | independent11:organization_hedging_multiple | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_hedging_multiple:t2 | independent11:organization_hedging_multiple | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_hedging_multiple:t3 | independent11:organization_hedging_multiple | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_hedging_multiple:t4 | independent11:organization_hedging_multiple | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_introducer_link:first-count | independent11:organization_introducer_link | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_introducer_link:t1 | independent11:organization_introducer_link | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_introducer_link:t2 | independent11:organization_introducer_link | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_introducer_link:t3 | independent11:organization_introducer_link | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_introducer_link:t4 | independent11:organization_introducer_link | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_wallet_share_unknown:first-count | independent11:organization_wallet_share_unknown | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_wallet_share_unknown:t1 | independent11:organization_wallet_share_unknown | correct_completed | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_wallet_share_unknown:t2 | independent11:organization_wallet_share_unknown | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_wallet_share_unknown:t3 | independent11:organization_wallet_share_unknown | correct_completed | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:organization_wallet_share_unknown:t4 | independent11:organization_wallet_share_unknown | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:outdated_still_activated:first-count | independent11:outdated_still_activated | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:outdated_still_activated:t1 | independent11:outdated_still_activated | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:outdated_still_activated:t2 | independent11:outdated_still_activated | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:outdated_still_activated:t3 | independent11:outdated_still_activated | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:outdated_still_activated:t4 | independent11:outdated_still_activated | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:overdue_expected_close:first-count | independent11:overdue_expected_close | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:overdue_expected_close:t1 | independent11:overdue_expected_close | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:overdue_expected_close:t2 | independent11:overdue_expected_close | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:overdue_expected_close:t3 | independent11:overdue_expected_close | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:overdue_expected_close:t4 | independent11:overdue_expected_close | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:partner_paid_and_unpaid_statements:first-count | independent11:partner_paid_and_unpaid_statements | silent_wrong | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:partner_paid_and_unpaid_statements:t1 | independent11:partner_paid_and_unpaid_statements | comparison_unverified | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:partner_paid_and_unpaid_statements:t2 | independent11:partner_paid_and_unpaid_statements | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:partner_paid_and_unpaid_statements:t3 | independent11:partner_paid_and_unpaid_statements | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:partner_paid_and_unpaid_statements:t4 | independent11:partner_paid_and_unpaid_statements | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_communication_preferences:first-count | independent11:person_communication_preferences | silent_wrong | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_communication_preferences:t1 | independent11:person_communication_preferences | refusal_unverified | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_communication_preferences:t2 | independent11:person_communication_preferences | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_communication_preferences:t3 | independent11:person_communication_preferences | refusal_unverified | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_communication_preferences:t4 | independent11:person_communication_preferences | refusal_unverified | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_multi_frequency_declarations:first-count | independent11:person_multi_frequency_declarations | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_multi_frequency_declarations:t1 | independent11:person_multi_frequency_declarations | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_multi_frequency_declarations:t2 | independent11:person_multi_frequency_declarations | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_multi_frequency_declarations:t3 | independent11:person_multi_frequency_declarations | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_multi_frequency_declarations:t4 | independent11:person_multi_frequency_declarations | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_note_pinned_exclusively:first-count | independent11:person_note_pinned_exclusively | task_error | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_note_pinned_exclusively:t1 | independent11:person_note_pinned_exclusively | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_note_pinned_exclusively:t2 | independent11:person_note_pinned_exclusively | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_note_pinned_exclusively:t3 | independent11:person_note_pinned_exclusively | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_note_pinned_exclusively:t4 | independent11:person_note_pinned_exclusively | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_organization_owner_gap:first-count | independent11:person_organization_owner_gap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_organization_owner_gap:t1 | independent11:person_organization_owner_gap | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_organization_owner_gap:t2 | independent11:person_organization_owner_gap | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_organization_owner_gap:t3 | independent11:person_organization_owner_gap | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:person_organization_owner_gap:t4 | independent11:person_organization_owner_gap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:prebillable_eligible_stats:first-count | independent11:prebillable_eligible_stats | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:prebillable_eligible_stats:t1 | independent11:prebillable_eligible_stats | silent_wrong | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:prebillable_eligible_stats:t2 | independent11:prebillable_eligible_stats | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:prebillable_eligible_stats:t3 | independent11:prebillable_eligible_stats | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:prebillable_eligible_stats:t4 | independent11:prebillable_eligible_stats | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:private_declared_currency_overlap:first-count | independent11:private_declared_currency_overlap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:private_declared_currency_overlap:t1 | independent11:private_declared_currency_overlap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:private_declared_currency_overlap:t2 | independent11:private_declared_currency_overlap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:private_declared_currency_overlap:t3 | independent11:private_declared_currency_overlap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:private_declared_currency_overlap:t4 | independent11:private_declared_currency_overlap | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:professional_zero_rate_referrals:first-count | independent11:professional_zero_rate_referrals | silent_wrong | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:professional_zero_rate_referrals:t1 | independent11:professional_zero_rate_referrals | silent_wrong | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:professional_zero_rate_referrals:t2 | independent11:professional_zero_rate_referrals | silent_wrong | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:professional_zero_rate_referrals:t3 | independent11:professional_zero_rate_referrals | silent_wrong | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:professional_zero_rate_referrals:t4 | independent11:professional_zero_rate_referrals | silent_wrong | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:qualified_without_deal_stage:first-count | independent11:qualified_without_deal_stage | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:qualified_without_deal_stage:t1 | independent11:qualified_without_deal_stage | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:qualified_without_deal_stage:t2 | independent11:qualified_without_deal_stage | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:qualified_without_deal_stage:t3 | independent11:qualified_without_deal_stage | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:qualified_without_deal_stage:t4 | independent11:qualified_without_deal_stage | task_error | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_conversion_valid_link:first-count | independent11:quote_conversion_valid_link | silent_wrong | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_conversion_valid_link:t1 | independent11:quote_conversion_valid_link | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_conversion_valid_link:t2 | independent11:quote_conversion_valid_link | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_conversion_valid_link:t3 | independent11:quote_conversion_valid_link | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_conversion_valid_link:t4 | independent11:quote_conversion_valid_link | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_days_before_activation:first-count | independent11:quote_days_before_activation | correct_completed | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_days_before_activation:t1 | independent11:quote_days_before_activation | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_days_before_activation:t2 | independent11:quote_days_before_activation | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_days_before_activation:t3 | independent11:quote_days_before_activation | task_error | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_days_before_activation:t4 | independent11:quote_days_before_activation | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_only_system_activity:first-count | independent11:quote_only_system_activity | correct_completed | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_only_system_activity:t1 | independent11:quote_only_system_activity | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_only_system_activity:t2 | independent11:quote_only_system_activity | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_only_system_activity:t3 | independent11:quote_only_system_activity | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:quote_only_system_activity:t4 | independent11:quote_only_system_activity | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:trade_sell_not_declared:first-count | independent11:trade_sell_not_declared | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:trade_sell_not_declared:t1 | independent11:trade_sell_not_declared | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:trade_sell_not_declared:t2 | independent11:trade_sell_not_declared | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:trade_sell_not_declared:t3 | independent11:trade_sell_not_declared | task_error | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:trade_sell_not_declared:t4 | independent11:trade_sell_not_declared | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:unknown_sti_with_broker:first-count | independent11:unknown_sti_with_broker | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:unknown_sti_with_broker:t1 | independent11:unknown_sti_with_broker | silent_wrong | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:unknown_sti_with_broker:t2 | independent11:unknown_sti_with_broker | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:unknown_sti_with_broker:t3 | independent11:unknown_sti_with_broker | silent_wrong | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:unknown_sti_with_broker:t4 | independent11:unknown_sti_with_broker | task_error | task_error | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_after_registration:first-count | independent11:verified_after_registration | correct_completed | refusal_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_after_registration:t1 | independent11:verified_after_registration | comparison_unverified | silent_wrong | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_after_registration:t2 | independent11:verified_after_registration | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_after_registration:t3 | independent11:verified_after_registration | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_after_registration:t4 | independent11:verified_after_registration | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_private_rejected_registration:first-count | independent11:verified_private_rejected_registration | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_private_rejected_registration:t1 | independent11:verified_private_rejected_registration | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_private_rejected_registration:t2 | independent11:verified_private_rejected_registration | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_private_rejected_registration:t3 | independent11:verified_private_rejected_registration | comparison_unverified | comparison_unverified | All original outcome labels retained. See the raw result and semantic review inventory. |
| independent11:verified_private_rejected_registration:t4 | independent11:verified_private_rejected_registration | correct_completed | correct_completed | All original outcome labels retained. See the raw result and semantic review inventory. |
Evidence and reproduction
- Late fixture validity gap — The ratified staff-exclusion predicate removes no rows. Full E1 fidelity is not established.
- Paired results and bootstrap intervals — All 250 cases per arm; 50 family clusters; 10,000 repeats.
- All 15 acceptance decisions — Exact measurements, statuses, limits, and original baseline viability.
- Every case transition — All original-to-candidate outcomes. No omitted attempts.
- Raw original attempts — All 250 sanitized original records.
- Raw V18 attempts — All 250 sanitized final candidate records.
- Terminal and environment audit — Class, cause, generic-message, definition and provider checks.
- Successful-answer review inventory — All successful candidate records. Numeric equality is not a meaning certificate.
- Lead semantic counterexamples — Two scored-correct answers fail separate read-only SQL counterexamples.
- Original completion retention — All losses, including refusal, clarification and blocked chains.
- Conversation breaks — Full-conversation scoring counts each chain once. These diagnostics can list several non-clarification failures in one chain; all descendants remain in raw results.
- E5 completed cause audit — All 60 first-turn errors and 161 total errors attributed to product causes.
- Fixture versus authorized schema — Added, removed and corrected fields and source-map type conflicts.
- Old V10 and revised scorer labels — Historical numeric scores are unchanged.
- Reproduction commands — Source revisions, method hashes, run commands and separated verification paths.
- The opened final holdout — 250 texts with their original contracts and expected SQL. No further tuning is allowed.
- Frozen input inventory — Original file paths and hashes. Source revisions retain the application and domain definitions.
- Isolated E6 summary — All 180 timings retained; target failed.
- Raw E6 timings and conformity — 30 fixed queries, three repeats per arm.
- Live latency, advisory only — All feature durations, including failures and refusals.
- Final curated and matrix test binding — 568 critical, 12 narrative, 57 chain and 9 migration passes.
- Full-suite failure disposition — 29,927 passes, 21 pre-existing failures, 50 skips.
- Fresh native browser results — Six completed cases, saved-read counters, and actual optional delays.
- Native shared total screenshot — GBP 2,454.50 and the checked 45-client scope.
- Native refusal screenshot — Permission class, cause, and suggested question.
- Independent final review history — The final retry resolved the earlier service-limit gap. All three review receipts are linked.
- Final independent evaluator review — Recomputed counts and intervals match. Full fixture validity remains incomplete.
- Final independent query review — Six material semantic findings, including two false-correct counterexamples.
- Final independent result review — Forty-four missed recovery cases, complete terminal parity checks and failed E6 latency.
- Initial six V8 files — Each initial unverified file is retained or revised and bound to final checks.
- Complete durable matrix — F01–F14, R01–R09, N01–N08. All history and limits remain.
- Complete published file inventory — Hashes, sizes, and sanitization state for every published artifact.
- Final process and container cleanup — All evaluation servers stopped. Three owned containers stopped with their volumes preserved. Other resources were not touched.
