Check power: how often each check catches a known defect¶
A check that says PASS carries no stated power until it has been run against defects of known size. The
fault-injection suite (pinneapple_analysis.validation.fault_injection, issue #697) does that: it takes correct cases
from a verified reference solver, injects controlled faults into the model output or into the dataset, and records
every check's response. The result is a detection matrix (check × fault × magnitude) with the false-alarm rate of
each check on clean cases and the minimal magnitude each check detects.
from pinneapple_analysis.validation.fault_injection import run_fault_injection
rep = run_fault_injection("darcy_2d", n_cases=50) # or "burgers_1d"
print(rep.to_markdown()) # detection matrix, false alarms, minimal detectable magnitude, ROC AUC
assert rep.passed, rep.failures() # every fault caught at its documented magnitude, <= 1 % false alarms per check
rep.check_power() # the "check power" section of a validation report (dict)
The current matrix, regenerated by python scripts/fault_injection_matrix.py --out docs/guides, is in
the detection matrix. CI runs the same script at a small size (--small) through
tests/test_fault_injection.py.
Problems¶
Both wrap the verified solvers of pinneapple_simulation.numerical_solvers.operator_benchmarks
(tests/test_operator_benchmarks.py: Cole-Hopf for Burgers, a manufactured solution with second-order convergence
for Darcy).
| problem | equation | output checked | clean variation (inside the training envelope) |
|---|---|---|---|
darcy_2d |
-div(a grad u) = 1, u = 0 on the unit square, a a thresholded Gaussian random field (Li et al. 2021) |
u on 32 × 32 cells |
coefficient contrast, feature size, fraction of high-permeability cells, reference solved 1-3× finer and averaged |
burgers_1d |
u_t + (u²/2)_x = ν u_xx, periodic |
u(t, x) at 21 times × 128 points |
amplitude, feature size, ν (0.8-1.25×), reference solved 1-3× finer |
Every case is regenerated from its seed and its variation, so a case found in a report is a few numbers.
Checks¶
Each check is a pinneapple_physics.physics_audit.monitor.ValidityCheck returning a continuous value (larger = more
suspicious) and rejecting above its tolerance. They reuse the library: the finite-difference provider and relative
residual of pinneapple_check.numerics, the solver's own conservative Darcy stencil
(operator_benchmarks.darcy_operator) and Burgers integrator, the spectral shells of
validation.field_metrics, the Darcy consistency and the duplicate and leakage detectors of the dataset audit
(pinneapple_data.audit), and the bounds and training-envelope checks of the physics monitor.
| problem | check | what it measures |
|---|---|---|
| both | residual (finite differences) | PDE residual with second-order differences, relative to the terms that balance |
| Darcy | residual (conservative, solver stencil) | |A⁻¹(A u - f)| / |A⁻¹ f| with the two-point-flux operator of the solver |
| Darcy | boundary condition (u = 0) | u extrapolated to the faces |
| Darcy | bounds | 0 ≤ u ≤ 0.0737 max f / min a (minimum principle, comparison with the torsion function of the square) |
| Darcy | conservation | net finite-volume outflow against the integral of the source |
| Burgers | residual (one-step re-integration) | each snapshot advanced to the next with the verified integrator |
| Burgers | initial condition, periodicity, bounds (max principle), mass conservation, energy decay | |
| both | spectral content | energy above a quarter (Darcy) or the 2/3-rule cutoff (Burgers) of the grid wavenumbers |
| both | training envelope | physics_audit.Envelope of the input features (amplitude, spectral centroid, range) |
default_checks(problem) is the set a user runs today (finite-difference residual, boundary or initial condition,
bounds, training envelope); check_library(problem) has all of them. Dataset checks (dataset_checks) add
near-duplicate inputs, train/test leakage and the worst sample of every case check.
Faults¶
| fault | level | what it does |
|---|---|---|
bias, scale |
model | output + m max|u|, output × (1 + m) |
noise |
model | white noise of m std(u) (high frequency) |
smoothing |
model | Gaussian smoothing over m cells (spectral bias) |
wrong_boundary |
model | Darcy: u = m max|u| on x = 0; Burgers: a ramp that breaks periodicity |
source_sign_flip |
model | Darcy: f → -f on x < m; Burgers: advection sign flipped for the last fraction m of time |
out_of_range |
model | inputs beyond the training range; the model saturates (Darcy) or scales linearly (Burgers) |
stale_model |
model | trained on an older configuration (Darcy: low permeability × (1 + m); Burgers: ν × (1 + m)) |
coefficient_drift |
data | every sample solved with a coefficient off by (1 + m), modelled on NVIDIA/physicsnemo#2036 |
duplicates, low_diversity |
data | training samples replaced by copies or near-copies (NVIDIA/physicsnemo#2035) |
leakage |
data | test samples copied from training |
unit_mixup |
data | outputs in mm instead of m |
swapped_channels |
data | Darcy: input and output swapped; Burgers: snapshots in reverse time order |
Each fault has a ladder of magnitudes and a documented magnitude (per problem where they differ) at which at least one check must catch it in 90 % of the trials; the acceptance of #697 also requires at most 1 % false alarms per check on the clean cases.
Reading the matrix¶
- min. detectable: the smallest magnitude a check detects in at least 90 % of the trials; "—" means not at any magnitude of the ladder. Some checks are blind to some faults by construction (the training envelope never sees a smoothed output); the matrix says which.
- rate / AUC: detection rate at the check's tolerance, and the ROC AUC of the check's value between clean and faulty cases (0.5 = blind, 1 = perfect separation, below 0.5 = the fault makes the case look cleaner, e.g. smoothing lowers the spectral content).
- true error: median relative L2 error of the faulty output against the correct field; it tells which detections matter.
The tolerances here are hand-set: the matrix measures them, it does not tune them. The validation arena attacks them, calibrates them with a stated false-alarm rate and hardens the set of checks.