COINFLIP / RESEARCH CONSOLENEBIUS × NVIDIA · 2026

Proof. Not probability.

Statistical Flaky-Test Investigator & Verifier

FINAL VERDICTPASS95% confidence

Replaying real recorded execution (2,000+ certification runs + NVIDIA Nemotron 3 Ultra)CONNECTING…

PROJECT: example-service
TEST IDENTIFIER: example-service::test_worker.py::test_job_finishes
TRACE: trace-hero-2026

STAGE 01 // BASELINE COIN TOSS

The Coin-Toss Dot Matrix

Original Test

BROKEN

Quiet Baseline

7.0%21 / 300 FAILED

● PASS ● FAIL

Heavy CPU load (4x busier)

58.0%174 / 300 FAILED

● PASS ● FAIL

Summary:21/300 quiet (7.0%) · 174/300 stress (58.0%)

Attempt 1 · Naive Quick Fix (Agent Baseline Control)

REJECTED

Quiet Baseline

0.0%0 / 1,000 FAILED

● PASS ● FAIL

Heavy CPU load (4x busier)

6.1%61 / 1,000 FAILED

● PASS ● FAIL

Summary:0/1,000 quiet · 61/1,000 stress (6.1%)

Attempt 2 · Coinflip Certified Fix

CERTIFIED

Quiet Baseline

0.0%0 / 1,000 FAILED

● PASS ● FAIL

Heavy CPU load (4x busier)

0.0%0 / 1,000 FAILED

● PASS ● FAIL

Summary:0/1,000 quiet · 0/1,000 stress (Upper bound < 0.30%)

STAGE 02A // ADAPTIVE BANDIT SEARCH & OUT-OF-SAMPLE CONFIRMATION

Stage 2A — Flake Amplifier (Adaptive Bandit Search & Out-of-Sample Confirmation)

CONFIRMED AMPLIFIED CONDITION: cpu_heavy_load @ L3_EXTREMERuns to First Failure: 2

Budget: n=60 explore + n=40 confirm

Baseline Flake Rate

6.7%

Unstressed test suite

Confirmed Amplified Rate

92.5%

Out-of-sample confirmed

Amplification Factor

13.9×

Baseline 6.7% → Confirmed Amplified 92.5% (13.9x Amplification)

95% Wilson Score CI

[80.1%, 97.4%]

Separated confirmation batch

Nemotron Static-AST Hypothesis Signals:nvidia/Nemotron-3_5-Lightningnemotron_static_ast

imports:threadingcalls:time.sleep

Successive Halving Leaderboard

Adaptive multi-arm bandit probe rounds (R1 Probe n=8 → Top-2 R2 Probe n=14 → Confirmation Batch n=40)

PRIOR RANK	CONDITION & INTENSITY	NEMOTRON RATIONALE	ROUND 1 PROBE (k/n)	ROUND 2 PROBE (k/n)	STATUS
#1	cpu_heavy_loadL3_EXTREME	Heavy CPU load starves worker thread scheduling, causing completion to lag fixed-sleep windows.	9 / 10(90%)	19 / 20(95%)	WINNER -> CONFIRMED
#2	cpu_half_coresL2_MODERATE	Halving CPU cores increases thread latency and context switching overhead.	6 / 10(60%)	9 / 20(45%)	ELIMINATED ROUND 2
#3	clock_shiftL1_MILD	Clock shift tests timer monotonic assumptions but thread race depends on compute cycles.	2 / 10(20%)	Eliminated R1	ELIMINATED ROUND 1
#4	shuffle_orderL1_MILD	Order shuffling detects cross-test pollution; worker test runs in isolation.	1 / 10(10%)	Eliminated R1	ELIMINATED ROUND 1

WINNER'S CURSE GUARD //

The reported amplified failure rate comes from a fresh Confirmation Batch (n=40) separate from the Exploration Budget (n=60) to eliminate selection bias (Winner's Curse), and Stage 4 certifies patches at this exact amplified condition.

STAGE 02 // PERTURBATION LEDGER

Stage 2 · Experiment Ledger

Whiskers = 95% Wilson confidence interval · red = triggering condition

PERTURBATION	FAILED / RUNS	FAILURE RATE	95% WILSON INTERVAL	HOLM ADJ P-VALUE	OBSERVATION
Normal run (baseline)baseline	21 / 300	7.0%	[0.0462, 0.1046]	—	CONTROL
Shuffle the test ordershuffle_order	21 / 300	7.0%	[0.0462, 0.1046]	—	CONTROL
Use a different random seedseed_change	18 / 300	6.0%	[0.0383, 0.0928]	—	CONTROL
Shift the clockclock_shift	21 / 300	7.0%	[0.0462, 0.1046]	—	CONTROL
Cut the networknetwork_cut	21 / 300	7.0%	[0.0462, 0.1046]	—	CONTROL
Halve the CPU corescpu_half_cores	93 / 300	31.0%	[0.2603, 0.3645]	—	CPU SENSITIVE
Heavy CPU load (4x busier)cpu_heavy_load	174 / 300	58.0%	[0.5235, 0.6345]	—	TRIGGERING PERTURBATION DETECTED

Causal Inference Ledger

Fisher's Exact Test & Relative Risk · versus baseline

Normal run (baseline)RR 1.00×

7.0% · 95% CI 4.62%–10.46%

p = 1.0000 · holm_adjusted_p_value: 1.0000

REFERENCE BASELINE

Shuffle the test orderRR 1.00×

7.0% · 95% CI 4.62%–10.46%

p = 1.0000 · holm_adjusted_p_value: 1.0000

NOISE / NOT SIGNIFICANT

Use a different random seedRR 0.86×

6.0% · 95% CI 3.83%–9.28%

p = 0.7409 · holm_adjusted_p_value: 1.0000

NOISE / NOT SIGNIFICANT

Shift the clockRR 1.00×

7.0% · 95% CI 4.62%–10.46%

p = 1.0000 · holm_adjusted_p_value: 1.0000

NOISE / NOT SIGNIFICANT

Cut the networkRR 1.00×

7.0% · 95% CI 4.62%–10.46%

p = 1.0000 · holm_adjusted_p_value: 1.0000

NOISE / NOT SIGNIFICANT

Halve the CPU coresRR 4.43×

31.0% · 95% CI 26.03%–36.45%

p = < 0.0001 · holm_adjusted_p_value: < 0.0001

SIGNIFICANT CAUSAL ASSOCIATION (Holm adj p < 0.01 & RR ≥ 1.8)

Heavy CPU load (4x busier)RR 8.29×

58.0% · 95% CI 52.35%–63.45%

p = < 0.0001 · holm_adjusted_p_value: < 0.0001

SIGNIFICANT CAUSAL ASSOCIATION (Holm adj p < 0.01 & RR ≥ 1.8)TRIGGER

Association alone does not establish causality; perturbation isolation and diagnosis provide context.

Root cause · nvidia/Nemotron-3-Ultra-550b-a55b

Order, seed, clock, and network make no difference. Only CPU pressure spikes failures (31% at half cores, 58% under heavy load). The test races the background worker with a fixed sleep instead of waiting for the completion signal.

Log summary · nvidia/Nemotron-3_5-Lightning

All 369 failure logs share one signature: worker.status was still 'RUNNING' when asserted after a fixed 1.0s sleep. No network, ordering, or seed-related errors found.

AssertionError: assert worker.status == 'DONE' (was 'RUNNING' after 1.0s sleep)

DIAGNOSTICS // SCHEDULER MICROSCOPE

Sub-Millisecond Thread Race Profiler

NEMOTRON 3 ULTRA / DIAGNOSTIC SUMMARYHIGH-PRECISION SCHEDULER TELEMETRY · VERIFIED TRACE

0ms10ms20ms30ms

Quiet Machine

+7.66ms · SAFE MARGIN → PASS

WORKER

12.34ms

Main assert 20.00ms · worker completion 12.34ms

Busy CPU · cpu_heavy_load

-6.40ms · RACE VIOLATION → FAIL

WORKER

26.40ms

Main assert 20.00ms · worker completion 26.40ms

STAGE 03 & 04 // STRESS REJECTION GATE

A fix is only as good as its evidence.

Why One Green Run Proves Nothing

Rule of Three: 0 failures in N runs ⇒ true rate < 3/N (95%)

REJECTED UNDER STRESS

QUIET MACHINE

0.0%0/1,000

TRIGGER CONDITION

6.1%61/1,000

Attempt 1 · Naive Quick Fix (Agent Baseline Control)

REFERENCE NAIVE AGENT FIXTURE

REJECTED

PATCH DIFF / unified

1--- a/test_worker.py
2+++ b/test_worker.py
3@@ -12,2 +12,2 @@
4-    time.sleep(1.0)
5+    time.sleep(2.0)
6     assert worker.status == 'DONE'

AST Static Anti-Cheat Gatekeeper

REFERENCE NAIVE AGENT FIXTURE

AST STATIC ANTI-CHEAT GATEKEEPER · COMPILER-GRADE PYTHON AST AUDIT

Assertions preserved — verification criterion
Zero @skip / xfail / try-pass suppressions — verification criterion
Numeric sleep inflation in supplied patch: 1.0s → 2.0s

Wald’s SPRT · Sequential Testing

REJECTED_EARLY_STOP · LLR = +14.82 > Upper Rejection Bound B = +2.94

Early-Reject at Run #50 → 950 Sandbox Runs Saved (95.0% Compute Saved)

Sequential Probability Ratio Test (SPRT) log-likelihood ratio crossed upper decision boundary B = ln((1-β)/α) = +2.94, terminating stress certification early and certifying empirical rejection with 95.0% compute savings.

Naive fix detected: passed 1000/1000 runs on a quiet machine (0.0%), but failed 61/1000 runs (6.1%) under CPU stress.

CERTIFIED (< 0.30% BOUND)

QUIET MACHINE

0.0%0/1,000

TRIGGER CONDITION

0.0%0/1,000

Attempt 2 · Coinflip Deterministic Fix: wait for worker completion signal

LIVE NEMOTRON PATCH

CERTIFIED

PATCH DIFF / unified

1--- a/test_worker.py
2+++ b/test_worker.py
3@@ -12,2 +12,2 @@
4-    time.sleep(1.0)
5+    assert worker.done_event.wait(timeout=5.0), 'Worker timed out'
6     assert worker.status == 'DONE'

AST Static Anti-Cheat Gatekeeper

LIVE NEMOTRON PATCH

AST STATIC ANTI-CHEAT GATEKEEPER · COMPILER-GRADE PYTHON AST AUDIT

Assertions preserved — verification criterion
Zero @skip / xfail / try-pass suppressions — verification criterion
Synchronization primitive in supplied patch: Event.wait

Wald’s SPRT · Sequential Testing

WALD SPRT ACCEPTED H0 · FULL SAMPLE CERTIFIED · 0 FAILURES IN 1,000 STRESS RUNS

0 / 1,000 normal · 0 / 1,000 stress. Upper bound 0.30% < 0.50% target

Rule of Three: True failure rate < 0.30% at 95% statistical confidence

0/1,000 normal · 0/1,000 stress. Upper bound 0.30% < 0.50% target.

STAGE 05 // STATISTICAL CERTIFICATE

OFFICIAL

Coinflip Certificate

FINAL VERDICT

PASS

Project & Test: example-service
example-service::test_worker.py::test_job_finishes
Trace ID & Timestamp: trace-hero-2026
2026-10-17T08:30:00Z
Root Cause Found: Triggered by cpu_heavy_load
Before the Fix: 21 / 300 normal · 174 / 300 stress
Rejected Quick Fix: 0 / 1,000 normal · 61 / 1,000 stress
After the Fix: 0 / 1,000 normal · 0 / 1,000 stress
95% Confidence Upper Bound: < 0.30% (Rule of Three) vs 0.50% target
Models Used: nvidia/Nemotron-3_5-Lightning
nvidia/Nemotron-3-Ultra-550b-a55b
Execution Backend: nebius_sandbox
Attestation: SHA-256 Merkle & HMAC Attested

RULE OF THREE / 95% CONFIDENCE

Upper Bound = 3 / N = 0.30% · N = 1,000

SHA-256 Merkle & HMAC Attested

Ledger Merkle Root (SHA-256): Calculating…
Patch Diff SHA-256: Calculating…
Rule of Three Upper Bound (95% CI): Upper Bound = 3 / N = 0.30%