Check power: the detection matrix¶
Generated by python scripts/fault_injection_matrix.py (full size, seed 0); do not edit by hand. The method is described in Check power.
Check power: darcy_2d¶
200 clean cases (seed 0), 40 datasets of [16, 8] samples; detected = flagged in at least 90% of the trials.
Model faults¶
Checks (model level, darcy_2d): C1 = residual (finite differences) (tol 0.6); C2 = residual (conservative, solver stencil) (tol 0.08); C3 = boundary condition (u = 0) (tol 0.1); C4 = bounds (0 <= u <= max torsion / a_min) (tol 0); C5 = conservation (outflow = source) (tol 0.05); C6 = spectral content (k > n/4) (tol 0.001); C7 = training envelope (tol 0.5)
| fault | documented | caught at documented by | min. detectable (any check) | min. C1 | min. C2 | min. C3 | min. C4 | min. C5 | min. C6 | min. C7 |
|---|---|---|---|---|---|---|---|---|---|---|
| bias | 0.05 (fraction of max | u | ) | C2, C5 | 0.005 | — | 0.05 | 0.1 | — | 0.005 |
| scale | 0.2 (relative) | C2, C5 | 0.1 | — | 0.1 | — | — | 0.1 | — | — |
| noise | 0.1 (fraction of std(u)) | C1, C3, C4, C6 | 0.1 | 0.1 | 0.3 | 0.1 | 0.1 | — | 0.1 | — |
| smoothing | 2 (cells (Gaussian sigma)) | C3, C5 | 0.5 | — | 4 | 2 | — | 0.5 | — | — |
| wrong_boundary | 0.1 (fraction of max | u | ) | C2, C3, C5 | 0.01 | — | 0.1 | 0.1 | — | 0.01 |
| source_sign_flip | 0.2 (fraction) | C2, C4, C5 | 0.05 | 0.5 | 0.2 | — | 0.1 | 0.05 | — | — |
| out_of_range | 1 (relative excess) | C2, C5, C7 | 0.3 | 4 | 0.3 | — | — | 0.3 | — | 0.3 |
| stale_model | 0.5 (relative) | C2, C5 | 0.5 | — | 0.5 | — | — | 0.5 | — | — |
False-alarm rate on 200 clean cases: C1 0.0%; C2 0.0%; C3 0.0%; C4 0.0%; C5 1.0%; C6 0.0%; C7 0.0%; any check 1.0%.
| fault | magnitude | true error (median) | any | C1 rate / AUC | C2 rate / AUC | C3 rate / AUC | C4 rate / AUC | C5 rate / AUC | C6 rate / AUC | C7 rate / AUC |
|---|---|---|---|---|---|---|---|---|---|---|
| bias | 0.005 | 0.00949 | 99% | 0% / 0.50 | 0% / 0.67 | 0% / 0.56 | 0% / 0.50 | 99% / 1.00 | 0% / 0.48 | 0% / 0.50 |
| bias | 0.01 | 0.019 | 100% | 0% / 0.50 | 0% / 0.83 | 0% / 0.65 | 0% / 0.50 | 100% / 1.00 | 0% / 0.46 | 0% / 0.50 |
| bias | 0.02 | 0.0379 | 100% | 0% / 0.50 | 1% / 0.98 | 2% / 0.80 | 0% / 0.50 | 100% / 1.00 | 0% / 0.43 | 0% / 0.50 |
| bias | 0.05 | 0.0949 | 100% | 0% / 0.50 | 99% / 1.00 | 14% / 0.97 | 0% / 0.50 | 100% / 1.00 | 0% / 0.34 | 0% / 0.50 |
| bias | 0.1 | 0.19 | 100% | 0% / 0.50 | 100% / 1.00 | 99% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.22 | 0% / 0.50 |
| bias | 0.2 | 0.379 | 100% | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.07 | 0% / 0.50 |
| scale | 0.01 | 0.01 | 1% | 0% / 0.51 | 0% / 0.70 | 0% / 0.50 | 0% / 0.50 | 1% / 0.57 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.02 | 0.02 | 2% | 0% / 0.51 | 0% / 0.85 | 0% / 0.50 | 0% / 0.50 | 2% / 0.65 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.05 | 0.05 | 40% | 0% / 0.53 | 12% / 1.00 | 0% / 0.50 | 0% / 0.50 | 35% / 0.96 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.1 | 0.1 | 100% | 0% / 0.56 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 0.99 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.2 | 0.2 | 100% | 0% / 0.62 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.5 | 0.5 | 100% | 0% / 0.79 | 100% / 1.00 | 0% / 0.50 | 1% / 0.51 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 |
| noise | 0.001 | 0.00054 | 1% | 0% / 0.50 | 0% / 0.55 | 0% / 0.50 | 0% / 0.50 | 1% / 0.55 | 0% / 0.50 | 0% / 0.50 |
| noise | 0.003 | 0.00163 | 1% | 0% / 0.52 | 0% / 0.55 | 0% / 0.51 | 0% / 0.50 | 1% / 0.56 | 0% / 0.51 | 0% / 0.50 |
| noise | 0.01 | 0.00541 | 10% | 0% / 0.64 | 0% / 0.56 | 0% / 0.54 | 10% / 0.55 | 1% / 0.61 | 0% / 0.60 | 0% / 0.50 |
| noise | 0.03 | 0.0162 | 68% | 2% / 0.99 | 0% / 0.72 | 0% / 0.76 | 66% / 0.83 | 10% / 0.75 | 0% / 0.98 | 0% / 0.50 |
| noise | 0.1 | 0.0537 | 100% | 100% / 1.00 | 0% / 1.00 | 95% / 1.00 | 100% / 1.00 | 52% / 0.92 | 100% / 1.00 | 0% / 0.50 |
| noise | 0.3 | 0.164 | 100% | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 86% / 0.97 | 100% / 1.00 | 0% / 0.50 |
| smoothing | 0.25 | 2.7e-05 | 1% | 0% / 0.50 | 0% / 0.54 | 0% / 0.50 | 0% / 0.50 | 1% / 0.55 | 0% / 0.50 | 0% / 0.50 |
| smoothing | 0.5 | 0.00854 | 100% | 0% / 0.73 | 0% / 0.55 | 4% / 0.77 | 0% / 0.50 | 100% / 1.00 | 0% / 0.05 | 0% / 0.50 |
| smoothing | 1 | 0.0293 | 100% | 4% / 1.00 | 0% / 0.86 | 64% / 0.99 | 0% / 0.50 | 100% / 1.00 | 0% / 0.00 | 0% / 0.50 |
| smoothing | 2 | 0.0709 | 100% | 24% / 1.00 | 0% / 1.00 | 99% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.00 | 0% / 0.50 |
| smoothing | 4 | 0.169 | 100% | 20% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.01 | 0% / 0.50 |
| wrong_boundary | 0.01 | 0.011 | 99% | 0% / 0.50 | 0% / 0.69 | 0% / 0.57 | 0% / 0.50 | 99% / 1.00 | 0% / 0.49 | 0% / 0.50 |
| wrong_boundary | 0.02 | 0.0219 | 100% | 0% / 0.50 | 0% / 0.87 | 1% / 0.69 | 0% / 0.50 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 |
| wrong_boundary | 0.05 | 0.0548 | 100% | 0% / 0.50 | 12% / 1.00 | 8% / 0.94 | 0% / 0.50 | 100% / 1.00 | 0% / 0.62 | 0% / 0.50 |
| wrong_boundary | 0.1 | 0.11 | 100% | 0% / 0.51 | 100% / 1.00 | 96% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.93 | 0% / 0.50 |
| wrong_boundary | 0.2 | 0.219 | 100% | 0% / 0.52 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 1% / 1.00 | 0% / 0.50 |
| wrong_boundary | 0.5 | 0.548 | 100% | 0% / 0.59 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 |
| source_sign_flip | 0.05 | 0.0435 | 100% | 0% / 0.50 | 0% / 0.88 | 0% / 0.38 | 65% / 0.82 | 100% / 1.00 | 0% / 0.30 | 0% / 0.50 |
| source_sign_flip | 0.1 | 0.0791 | 100% | 0% / 0.65 | 32% / 1.00 | 0% / 0.37 | 100% / 1.00 | 100% / 1.00 | 0% / 0.23 | 0% / 0.50 |
| source_sign_flip | 0.2 | 0.243 | 100% | 0% / 0.95 | 100% / 1.00 | 0% / 0.37 | 100% / 1.00 | 100% / 1.00 | 0% / 0.23 | 0% / 0.50 |
| source_sign_flip | 0.5 | 1.07 | 100% | 100% / 1.00 | 100% / 1.00 | 1% / 0.69 | 100% / 1.00 | 100% / 1.00 | 0% / 0.68 | 0% / 0.50 |
| source_sign_flip | 1 | 1.99 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.42 | 100% / 1.00 | 100% / 1.00 | 0% / 0.44 | 0% / 0.50 |
| out_of_range | 0.1 | 0.0608 | 66% | 0% / 0.54 | 6% / 1.00 | 0% / 0.50 | 0% / 0.50 | 63% / 0.97 | 0% / 0.50 | 18% / 1.00 |
| out_of_range | 0.3 | 0.153 | 100% | 0% / 0.68 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 |
| out_of_range | 1 | 0.423 | 100% | 3% / 0.99 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 |
| out_of_range | 2 | 0.714 | 100% | 52% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 |
| out_of_range | 4 | 1.11 | 100% | 97% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 |
| stale_model | 0.05 | 0.0339 | 0% | 0% / 0.39 | 0% / 0.72 | 0% / 0.39 | 0% / 0.50 | 0% / 0.63 | 0% / 0.42 | 0% / 0.50 |
| stale_model | 0.1 | 0.0546 | 8% | 0% / 0.40 | 0% / 0.97 | 0% / 0.37 | 0% / 0.50 | 8% / 0.85 | 0% / 0.39 | 0% / 0.50 |
| stale_model | 0.2 | 0.0924 | 74% | 0% / 0.42 | 56% / 1.00 | 0% / 0.33 | 0% / 0.50 | 52% / 0.95 | 0% / 0.36 | 0% / 0.50 |
| stale_model | 0.5 | 0.183 | 100% | 0% / 0.54 | 100% / 1.00 | 0% / 0.23 | 0% / 0.50 | 90% / 0.98 | 0% / 0.27 | 0% / 0.50 |
| stale_model | 1 | 0.286 | 100% | 0% / 0.73 | 100% / 1.00 | 0% / 0.14 | 0% / 0.50 | 96% / 0.99 | 0% / 0.18 | 0% / 0.50 |
| stale_model | 2 | 0.412 | 100% | 0% / 0.91 | 100% / 1.00 | 0% / 0.08 | 0% / 0.50 | 98% / 1.00 | 0% / 0.13 | 0% / 0.50 |
Data faults¶
Checks (data level, darcy_2d): C1 = duplicated or non-diverse samples (tol 0); C2 = train/test leakage (tol 0); C3 = every sample: residual (finite differences) (tol 0.6); C4 = every sample: residual (conservative, solver stencil) (tol 0.08); C5 = every sample: boundary condition (u = 0) (tol 0.1); C6 = every sample: bounds (0 <= u <= max torsion / a_min) (tol 0); C7 = every sample: conservation (outflow = source) (tol 0.05); C8 = every sample: spectral content (k > n/4) (tol 0.001)
| fault | documented | caught at documented by | min. detectable (any check) | min. C1 | min. C2 | min. C3 | min. C4 | min. C5 | min. C6 | min. C7 | min. C8 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| coefficient_drift | 0.1 (relative) | C7 | 0.1 | — | — | — | 0.2 | — | — | 0.1 | — |
| duplicates | 0.1 (fraction of training samples) | C1 | 0.05 | 0.05 | — | — | — | — | — | — | — |
| low_diversity | 0.1 (fraction of training samples) | C1 | 0.05 | 0.05 | — | — | — | — | — | — | — |
| leakage | 0.1 (fraction of test samples) | C1, C2 | 0.05 | 0.05 | 0.05 | — | — | — | — | — | — |
| unit_mixup | 0.05 (fraction of samples) | C3, C4, C6, C7 | 0.05 | — | — | 0.05 | 0.05 | — | 0.05 | 0.05 | — |
| swapped_channels | 0.05 (fraction of samples) | C3, C4, C5, C8 | 0.05 | 0.1 | 0.25 | 0.05 | 0.05 | 0.05 | — | — | 0.05 |
False-alarm rate on 40 clean datasets: C1 0.0%; C2 0.0%; C3 0.0%; C4 0.0%; C5 0.0%; C6 0.0%; C7 0.0%; C8 0.0%; any check 0.0%.
| fault | magnitude | true error (median) | any | C1 rate / AUC | C2 rate / AUC | C3 rate / AUC | C4 rate / AUC | C5 rate / AUC | C6 rate / AUC | C7 rate / AUC | C8 rate / AUC |
|---|---|---|---|---|---|---|---|---|---|---|---|
| coefficient_drift | 0.01 | 0.0371 | 0% | 0% / 0.50 | 0% / 0.50 | 0% / 0.33 | 0% / 0.00 | 0% / 0.29 | 0% / 0.50 | 0% / 0.00 | 0% / 0.16 |
| coefficient_drift | 0.02 | 0.0423 | 0% | 0% / 0.50 | 0% / 0.50 | 0% / 0.33 | 0% / 0.00 | 0% / 0.29 | 0% / 0.50 | 0% / 0.00 | 0% / 0.16 |
| coefficient_drift | 0.05 | 0.0591 | 0% | 0% / 0.50 | 0% / 0.50 | 0% / 0.33 | 0% / 0.41 | 0% / 0.29 | 0% / 0.50 | 0% / 0.61 | 0% / 0.31 |
| coefficient_drift | 0.1 | 0.086 | 100% | 0% / 0.50 | 0% / 0.50 | 0% / 0.33 | 0% / 1.00 | 0% / 0.29 | 0% / 0.50 | 100% / 1.00 | 0% / 0.31 |
| coefficient_drift | 0.2 | 0.136 | 100% | 0% / 0.50 | 0% / 0.50 | 0% / 0.33 | 100% / 1.00 | 0% / 0.53 | 0% / 0.50 | 100% / 1.00 | 0% / 0.31 |
| coefficient_drift | 0.5 | 0.263 | 100% | 0% / 0.50 | 0% / 0.50 | 0% / 0.30 | 100% / 1.00 | 0% / 0.53 | 0% / 0.50 | 100% / 1.00 | 0% / 0.59 |
| coefficient_drift | 1 | 0.411 | 100% | 0% / 0.50 | 0% / 0.50 | 0% / 0.33 | 100% / 1.00 | 0% / 0.55 | 0% / 0.50 | 100% / 1.00 | 0% / 0.85 |
| duplicates | 0.05 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.49 | 0% / 0.47 | 0% / 0.49 | 0% / 0.50 | 0% / 0.49 | 0% / 0.48 |
| duplicates | 0.1 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.48 | 0% / 0.48 | 0% / 0.49 | 0% / 0.50 | 0% / 0.49 | 0% / 0.48 |
| duplicates | 0.25 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.44 | 0% / 0.44 | 0% / 0.43 | 0% / 0.50 | 0% / 0.45 | 0% / 0.45 |
| duplicates | 0.5 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.38 | 0% / 0.45 | 0% / 0.43 | 0% / 0.50 | 0% / 0.40 | 0% / 0.41 |
| low_diversity | 0.05 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.48 | 0% / 0.48 | 0% / 0.48 | 0% / 0.50 | 0% / 0.47 | 0% / 0.50 |
| low_diversity | 0.1 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.49 | 0% / 0.49 | 0% / 0.50 | 0% / 0.49 | 0% / 0.44 |
| low_diversity | 0.25 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.49 | 0% / 0.46 | 0% / 0.46 | 0% / 0.50 | 0% / 0.46 | 0% / 0.47 |
| low_diversity | 0.5 | 0 | 95% | 95% / 0.97 | 0% / 0.50 | 0% / 0.44 | 0% / 0.41 | 0% / 0.36 | 0% / 0.50 | 0% / 0.38 | 0% / 0.39 |
| leakage | 0.05 | 0 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.49 | 0% / 0.50 | 0% / 0.48 | 0% / 0.46 |
| leakage | 0.1 | 0 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.49 | 0% / 0.49 | 0% / 0.50 | 0% / 0.46 | 0% / 0.50 |
| leakage | 0.25 | 0 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.48 | 0% / 0.45 | 0% / 0.46 | 0% / 0.50 | 0% / 0.49 | 0% / 0.48 |
| leakage | 0.5 | 0 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.43 | 0% / 0.43 | 0% / 0.42 | 0% / 0.50 | 0% / 0.43 | 0% / 0.44 |
| unit_mixup | 0.05 | 999 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 0% / 0.49 | 100% / 1.00 | 100% / 1.00 | 0% / 0.49 |
| unit_mixup | 0.1 | 999 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 0% / 0.49 | 100% / 1.00 | 100% / 1.00 | 0% / 0.47 |
| unit_mixup | 0.25 | 999 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 0% / 0.53 | 100% / 1.00 | 100% / 1.00 | 0% / 0.48 |
| unit_mixup | 1 | 999 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 0% / 0.56 | 100% / 1.00 | 100% / 1.00 | 0% / 0.33 |
| swapped_channels | 0.05 | 1.22e+03 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 |
| swapped_channels | 0.1 | 1.46e+03 | 100% | 100% / 1.00 | 45% / 0.73 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 |
| swapped_channels | 0.25 | 1.66e+03 | 100% | 100% / 1.00 | 98% / 0.99 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 |
| swapped_channels | 1 | 2.01e+03 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 |
Acceptance: every fault caught at its documented magnitude, false alarms <= 1 % per check.
Check power: burgers_1d¶
200 clean cases (seed 0), 40 datasets of [16, 8] samples; detected = flagged in at least 90% of the trials.
Model faults¶
Checks (model level, burgers_1d): C1 = residual (finite differences) (tol 0.3); C2 = residual (one-step re-integration) (tol 0.02); C3 = initial condition (tol 0.02); C4 = periodicity (tol 4); C5 = bounds (max |u| <= max |u0|) (tol 0); C6 = mass conservation (tol 0.01); C7 = energy decay (tol 0.001); C8 = spectral content (k > n/3) (tol 0.0001); C9 = training envelope (tol 0.5)
| fault | documented | caught at documented by | min. detectable (any check) | min. C1 | min. C2 | min. C3 | min. C4 | min. C5 | min. C6 | min. C7 | min. C8 | min. C9 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bias | 0.05 (fraction of max | u | ) | C3, C6 | 0.01 | — | 0.1 | 0.02 | — | — | 0.01 | — |
| scale | 0.2 (relative) | C2, C3, C5 | 0.01 | — | 0.2 | 0.05 | — | 0.01 | — | — | — | — |
| noise | 0.1 (fraction of std(u)) | C1, C2, C3, C5, C6, C8 | 0.03 | 0.03 | 0.03 | 0.03 | — | 0.1 | 0.1 | 0.3 | 0.03 | — |
| smoothing | 4 (cells (Gaussian sigma)) | C2, C3 | 2 | — | 4 | 4 | — | — | — | — | — | — |
| wrong_boundary | 0.1 (fraction of max | u | ) | C4 | 0.05 | — | 0.2 | — | 0.05 | — | — | — |
| source_sign_flip | 0.2 (fraction) | C2 | 0.05 | 1 | 0.05 | — | — | — | — | — | — | — |
| out_of_range | 1 (relative excess) | C1, C2 | 0.3 | 1 | 0.3 | — | — | — | — | — | — | 2 |
| stale_model | 0.5 (relative) | C2 | 0.5 | — | 0.5 | — | — | — | — | — | — | — |
False-alarm rate on 200 clean cases: C1 0.0%; C2 0.0%; C3 0.0%; C4 0.0%; C5 0.0%; C6 0.0%; C7 0.0%; C8 0.0%; C9 0.0%; any check 0.0%.
| fault | magnitude | true error (median) | any | C1 rate / AUC | C2 rate / AUC | C3 rate / AUC | C4 rate / AUC | C5 rate / AUC | C6 rate / AUC | C7 rate / AUC | C8 rate / AUC | C9 rate / AUC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bias | 0.005 | 0.0124 | 82% | 0% / 0.51 | 0% / 0.94 | 0% / 1.00 | 0% / 0.50 | 56% / 0.78 | 57% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| bias | 0.01 | 0.0248 | 100% | 0% / 0.52 | 1% / 0.98 | 0% / 1.00 | 0% / 0.50 | 57% / 0.79 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| bias | 0.02 | 0.0496 | 100% | 0% / 0.56 | 6% / 0.99 | 100% / 1.00 | 0% / 0.50 | 57% / 0.79 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| bias | 0.05 | 0.124 | 100% | 0% / 0.75 | 80% / 1.00 | 100% / 1.00 | 0% / 0.50 | 62% / 0.81 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| bias | 0.1 | 0.248 | 100% | 0% / 0.94 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 67% / 0.84 | 100% / 1.00 | 0% / 0.50 | 0% / 0.49 | 0% / 0.50 |
| bias | 0.2 | 0.496 | 100% | 10% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 80% / 0.90 | 100% / 1.00 | 0% / 0.50 | 0% / 0.49 | 0% / 0.50 |
| scale | 0.01 | 0.01 | 100% | 0% / 0.52 | 0% / 0.89 | 0% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.51 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.02 | 0.02 | 100% | 0% / 0.54 | 0% / 0.96 | 64% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.51 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.05 | 0.05 | 100% | 0% / 0.60 | 2% / 0.99 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.51 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.1 | 0.1 | 100% | 0% / 0.72 | 38% / 1.00 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.51 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.2 | 0.2 | 100% | 0% / 0.89 | 94% / 1.00 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.52 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| scale | 0.5 | 0.5 | 100% | 2% / 0.99 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.52 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| noise | 0.001 | 0.001 | 44% | 0% / 0.58 | 0% / 0.82 | 0% / 1.00 | 0% / 0.86 | 44% / 0.72 | 0% / 1.00 | 0% / 0.50 | 0% / 0.84 | 0% / 0.50 |
| noise | 0.003 | 0.003 | 55% | 0% / 0.83 | 0% / 0.93 | 0% / 1.00 | 0% / 0.91 | 55% / 0.77 | 0% / 1.00 | 0% / 0.50 | 0% / 0.96 | 0% / 0.50 |
| noise | 0.01 | 0.00998 | 70% | 16% / 0.99 | 0% / 0.99 | 0% / 1.00 | 0% / 0.93 | 67% / 0.84 | 0% / 1.00 | 0% / 0.50 | 1% / 0.99 | 0% / 0.50 |
| noise | 0.03 | 0.03 | 100% | 100% / 1.00 | 96% / 1.00 | 98% / 1.00 | 0% / 0.93 | 78% / 0.89 | 0% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.50 |
| noise | 0.1 | 0.1 | 100% | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.93 | 97% / 0.98 | 98% / 1.00 | 21% / 0.64 | 100% / 1.00 | 0% / 0.50 |
| noise | 0.3 | 0.3 | 100% | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.93 | 100% / 1.00 | 100% / 1.00 | 99% / 1.00 | 100% / 1.00 | 0% / 0.50 |
| smoothing | 0.25 | 1.07e-05 | 0% | 0% / 0.50 | 0% / 0.55 | 0% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| smoothing | 0.5 | 0.00342 | 0% | 0% / 0.54 | 0% / 0.94 | 0% / 1.00 | 0% / 0.49 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.46 | 0% / 0.50 |
| smoothing | 1 | 0.0149 | 25% | 0% / 0.71 | 23% / 0.99 | 2% / 1.00 | 0% / 0.49 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.31 | 0% / 0.50 |
| smoothing | 2 | 0.0473 | 98% | 0% / 0.91 | 86% / 1.00 | 84% / 1.00 | 0% / 0.48 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.21 | 0% / 0.50 |
| smoothing | 4 | 0.122 | 100% | 10% / 0.98 | 100% / 1.00 | 100% / 1.00 | 0% / 0.43 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.21 | 0% / 0.50 |
| wrong_boundary | 0.01 | 0.00418 | 84% | 0% / 0.50 | 0% / 0.89 | 0% / 0.50 | 84% / 0.97 | 0% / 0.50 | 0% / 0.99 | 0% / 0.50 | 0% / 0.80 | 0% / 0.50 |
| wrong_boundary | 0.02 | 0.00836 | 86% | 0% / 0.50 | 0% / 0.95 | 0% / 0.50 | 86% / 0.98 | 0% / 0.50 | 0% / 1.00 | 0% / 0.50 | 0% / 0.88 | 0% / 0.50 |
| wrong_boundary | 0.05 | 0.0209 | 92% | 0% / 0.51 | 0% / 0.99 | 0% / 0.50 | 92% / 1.00 | 0% / 0.50 | 0% / 1.00 | 0% / 0.50 | 0% / 0.97 | 0% / 0.50 |
| wrong_boundary | 0.1 | 0.0418 | 97% | 0% / 0.54 | 9% / 1.00 | 0% / 0.50 | 96% / 1.00 | 0% / 0.50 | 0% / 1.00 | 0% / 0.50 | 1% / 0.99 | 0% / 0.50 |
| wrong_boundary | 0.2 | 0.0836 | 100% | 0% / 0.63 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 0% / 0.50 | 0% / 1.00 | 0% / 0.50 | 21% / 0.99 | 0% / 0.50 |
| wrong_boundary | 0.5 | 0.209 | 100% | 0% / 0.88 | 100% / 1.00 | 0% / 0.50 | 100% / 1.00 | 2% / 0.51 | 0% / 1.00 | 12% / 0.58 | 100% / 1.00 | 0% / 0.50 |
| source_sign_flip | 0.05 | 0.0181 | 98% | 0% / 0.86 | 98% / 1.00 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| source_sign_flip | 0.1 | 0.0365 | 98% | 0% / 0.84 | 98% / 1.00 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| source_sign_flip | 0.2 | 0.0812 | 100% | 0% / 0.92 | 100% / 1.00 | 0% / 0.50 | 0% / 0.53 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| source_sign_flip | 0.5 | 0.302 | 100% | 47% / 0.99 | 100% / 1.00 | 0% / 0.50 | 0% / 0.55 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.48 | 0% / 0.50 |
| source_sign_flip | 1 | 0.777 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.54 | 0% / 0.50 | 0% / 0.16 | 0% / 0.50 | 0% / 0.15 | 0% / 0.50 |
| out_of_range | 0.1 | 0.0609 | 38% | 0% / 0.72 | 38% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.60 |
| out_of_range | 0.3 | 0.174 | 100% | 0% / 0.95 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 23% / 0.75 |
| out_of_range | 1 | 0.475 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 86% / 1.00 |
| out_of_range | 2 | 0.768 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 |
| out_of_range | 4 | 1.16 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 |
| stale_model | 0.05 | 0.00742 | 0% | 0% / 0.57 | 0% / 0.97 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.15 | 0% / 0.50 | 0% / 0.12 | 0% / 0.50 |
| stale_model | 0.1 | 0.0146 | 0% | 0% / 0.65 | 0% / 0.99 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.16 | 0% / 0.50 | 0% / 0.09 | 0% / 0.50 |
| stale_model | 0.2 | 0.0286 | 4% | 0% / 0.78 | 4% / 1.00 | 0% / 0.50 | 0% / 0.51 | 0% / 0.50 | 0% / 0.16 | 0% / 0.50 | 0% / 0.05 | 0% / 0.50 |
| stale_model | 0.5 | 0.0669 | 94% | 0% / 0.94 | 94% / 1.00 | 0% / 0.50 | 0% / 0.49 | 0% / 0.50 | 0% / 0.15 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 |
| stale_model | 1 | 0.122 | 100% | 1% / 0.98 | 100% / 1.00 | 0% / 0.50 | 0% / 0.47 | 0% / 0.50 | 0% / 0.15 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 |
| stale_model | 2 | 0.209 | 100% | 36% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.45 | 0% / 0.50 | 0% / 0.14 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 |
Data faults¶
Checks (data level, burgers_1d): C1 = duplicated or non-diverse samples (tol 0); C2 = train/test leakage (tol 0); C3 = every sample: residual (finite differences) (tol 0.3); C4 = every sample: residual (one-step re-integration) (tol 0.02); C5 = every sample: initial condition (tol 0.02); C6 = every sample: periodicity (tol 4); C7 = every sample: bounds (max |u| <= max |u0|) (tol 0); C8 = every sample: mass conservation (tol 0.01); C9 = every sample: energy decay (tol 0.001); C10 = every sample: spectral content (k > n/3) (tol 0.0001)
| fault | documented | caught at documented by | min. detectable (any check) | min. C1 | min. C2 | min. C3 | min. C4 | min. C5 | min. C6 | min. C7 | min. C8 | min. C9 | min. C10 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| coefficient_drift | 0.5 (relative) | C4 | 0.5 | — | — | — | 0.5 | — | — | — | — | — | — |
| duplicates | 0.1 (fraction of training samples) | C1 | 0.05 | 0.05 | — | — | — | — | — | — | — | — | — |
| low_diversity | 0.1 (fraction of training samples) | C1 | 0.05 | 0.05 | — | — | — | — | — | 0.25 | — | — | — |
| leakage | 0.1 (fraction of test samples) | C1, C2 | 0.05 | 0.05 | 0.05 | — | — | — | — | — | — | — | — |
| unit_mixup | 0.05 (fraction of samples) | C3, C4, C5, C7 | 0.05 | — | — | 0.05 | 0.05 | 0.05 | — | 0.05 | 0.25 | — | — |
| swapped_channels | 0.05 (fraction of samples) | C3, C4, C5, C9 | 0.05 | — | — | 0.05 | 0.05 | 0.05 | — | — | — | 0.05 | — |
False-alarm rate on 40 clean datasets: C1 0.0%; C2 0.0%; C3 0.0%; C4 0.0%; C5 0.0%; C6 0.0%; C7 0.0%; C8 0.0%; C9 0.0%; C10 0.0%; any check 0.0%.
| fault | magnitude | true error (median) | any | C1 rate / AUC | C2 rate / AUC | C3 rate / AUC | C4 rate / AUC | C5 rate / AUC | C6 rate / AUC | C7 rate / AUC | C8 rate / AUC | C9 rate / AUC | C10 rate / AUC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| coefficient_drift | 0.01 | 0.00221 | 0% | 0% / 0.50 | 0% / 0.50 | 0% / 0.65 | 0% / 0.00 | 0% / 0.50 | 0% / 0.37 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 | 0% / 0.00 |
| coefficient_drift | 0.02 | 0.00438 | 0% | 0% / 0.50 | 0% / 0.50 | 0% / 0.65 | 0% / 0.01 | 0% / 0.50 | 0% / 0.37 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 | 0% / 0.00 |
| coefficient_drift | 0.05 | 0.0109 | 0% | 0% / 0.50 | 0% / 0.50 | 0% / 0.67 | 0% / 1.00 | 0% / 0.50 | 0% / 0.25 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 | 0% / 0.00 |
| coefficient_drift | 0.1 | 0.0214 | 0% | 0% / 0.50 | 0% / 0.50 | 0% / 0.85 | 0% / 1.00 | 0% / 0.50 | 0% / 0.25 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 | 0% / 0.00 |
| coefficient_drift | 0.2 | 0.0419 | 0% | 0% / 0.50 | 0% / 0.50 | 0% / 0.98 | 0% / 1.00 | 0% / 0.50 | 0% / 0.14 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 | 0% / 0.00 |
| coefficient_drift | 0.5 | 0.0978 | 100% | 0% / 0.50 | 0% / 0.50 | 0% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.49 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 | 0% / 0.00 |
| coefficient_drift | 1 | 0.177 | 100% | 0% / 0.50 | 0% / 0.50 | 0% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.27 | 0% / 0.50 | 0% / 0.00 | 0% / 0.50 | 0% / 0.00 |
| duplicates | 0.05 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.48 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.47 | 0% / 0.50 | 0% / 0.48 |
| duplicates | 0.1 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.45 | 0% / 0.48 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.45 | 0% / 0.50 | 0% / 0.48 |
| duplicates | 0.25 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.46 | 0% / 0.44 | 0% / 0.50 | 0% / 0.46 | 0% / 0.50 | 0% / 0.47 | 0% / 0.50 | 0% / 0.44 |
| duplicates | 0.5 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.37 | 0% / 0.40 | 0% / 0.50 | 0% / 0.40 | 0% / 0.50 | 0% / 0.32 | 0% / 0.50 | 0% / 0.41 |
| low_diversity | 0.05 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.47 | 0% / 1.00 | 0% / 0.50 | 40% / 0.70 | 0% / 0.51 | 0% / 0.50 | 0% / 0.47 |
| low_diversity | 0.1 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.49 | 0% / 0.46 | 0% / 1.00 | 0% / 0.49 | 78% / 0.89 | 0% / 0.47 | 0% / 0.50 | 0% / 0.46 |
| low_diversity | 0.25 | 0 | 100% | 100% / 1.00 | 0% / 0.50 | 0% / 0.46 | 0% / 0.44 | 0% / 1.00 | 0% / 0.48 | 98% / 0.99 | 0% / 0.41 | 0% / 0.50 | 0% / 0.43 |
| low_diversity | 0.5 | 0 | 100% | 98% / 0.99 | 0% / 0.50 | 0% / 0.40 | 0% / 0.36 | 0% / 1.00 | 0% / 0.41 | 100% / 1.00 | 0% / 0.35 | 0% / 0.50 | 0% / 0.37 |
| leakage | 0.05 | 0 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.47 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 |
| leakage | 0.1 | 0 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.49 | 0% / 0.50 | 0% / 0.50 | 0% / 0.48 | 0% / 0.50 | 0% / 0.49 | 0% / 0.50 | 0% / 0.49 |
| leakage | 0.25 | 0 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.48 | 0% / 0.48 | 0% / 0.50 | 0% / 0.46 | 0% / 0.50 | 0% / 0.48 | 0% / 0.50 | 0% / 0.47 |
| leakage | 0.5 | 0 | 100% | 100% / 1.00 | 100% / 1.00 | 0% / 0.46 | 0% / 0.45 | 0% / 0.50 | 0% / 0.44 | 0% / 0.50 | 0% / 0.48 | 0% / 0.50 | 0% / 0.44 |
| unit_mixup | 0.05 | 999 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.51 | 100% / 1.00 | 35% / 0.78 | 0% / 0.50 | 0% / 0.49 |
| unit_mixup | 0.1 | 999 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.51 | 100% / 1.00 | 52% / 0.96 | 0% / 0.50 | 0% / 0.48 |
| unit_mixup | 0.25 | 999 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.56 | 100% / 1.00 | 98% / 1.00 | 0% / 0.50 | 0% / 0.46 |
| unit_mixup | 1 | 999 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.65 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.31 |
| swapped_channels | 0.05 | 0.675 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.49 |
| swapped_channels | 0.1 | 0.845 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.48 |
| swapped_channels | 0.25 | 0.977 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.46 |
| swapped_channels | 1 | 1.02 | 100% | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 100% / 1.00 | 100% / 1.00 | 0% / 0.50 | 0% / 0.50 | 0% / 0.50 | 100% / 1.00 | 0% / 0.27 |
Acceptance: every fault caught at its documented magnitude, false alarms <= 1 % per check.