Skip to content

Dataset physics audit: does the data solve its PDE?

A model trained on wrong data inherits the error, and model-level checks may never see it. Two open bugs of a widely used Physics AI library show how training data can be silently wrong: generated Darcy data that did not solve the documented equation (NVIDIA/physicsnemo#2036) and permeability samples with far less diversity than documented (NVIDIA/physicsnemo#2035). pinneapple_data.audit checks the data itself, before anyone trains on it.

import pinneapple as pp

card = pp.datasets.audit("heatsink_forced_convection")      # a zoo dataset, with the constraints its card states
print(card)                                                 # one line per check: PASS / WARN / FAIL / NOT-APPLICABLE

from pinneapple_data.audit import audit_dataset
card = audit_dataset({"train": {"a": a_train, "u": u_train}, "test": {"a": a_test, "u": u_test}},
                     physics="darcy", generator=my_permeability_sampler, units={"a": "-", "u": "-"})
card.to_json("audit_card.json")

Checks

Check What it measures Fails when
finite infinities and missing values any infinity (NaN is a warning, marked "documented" when the metadata says the value was not recorded)
physics.solve per sample: ||u - A^-1 f|| / ||A^-1 f||, the distance to the solution of its own discrete problem (the residual in the A^-1 norm) above 2 %
physics.bc, physics.flux boundary values against the exact discrete solution's; net boundary flux against the source integral above 2 %
constraint: ... declared relations between columns ("r_base_k_w > 0", "source_width_mm == source_frac_w * base_width_mm") any row breaks one
diversity.duplicates exact and near copies inside the train, validation and test splits (closer than 1e-3 x the median nearest-neighbour distance) any
diversity.effective effective number of distinct samples: participation ratio of the principal-component spectrum, against freshly drawn samples of the stated generator when there is one below half the generator's (without a generator: below 2)
diversity.spectrum mean radial spectrum of the inputs against the generator's off by more than a factor 2 in a shell
leakage nearest training sample of each test sample any test sample closer than 1e-3 x the median training spacing
units units in the metadata parse, and match the problem's when stated otherwise

The physics check is built in for "darcy" (-div(a grad u) = f, u = 0 on the unit square, cell-centred, the discretisation of operator_benchmarks.darcy_2d and of darcy_2d_li2021); any other problem passes a DatasetPhysics(name, columns, check, tolerances). Each solve check runs on up to 64 random samples per split by default (max_physics_samples).

Results

Zoo datasets (cards stored in pinneapple_hub/zoo/audit_cards/ and shown in each dataset's Hub card):

Dataset Decision Notes
heatsink_forced_convection PASS 4 constraints on 6000 rows, no duplicates or leakage
heatsink_natural_convection PASS 3 constraints on 5000 rows
rover_contact_grc1_viper PASS
airfoil_rans_cst_re4e6 WARN Cm is NaN in the 340 training rows: documented ("Cm not recorded in the campaign table")
single_wheel_tests WARN 5 torque values NaN: documented ("NaN = not reported"); one split, so no leakage check
burgers_1d_li2021, darcy_2d_li2021 not audited yet Hub-only; the test downloads and audits them where the network allows

Defects (tests/test_dataset_audit.py, Darcy on 32 x 32 cells, coefficient from the Li et al. 2021 generator):

  • data generated by an independent finite-difference solver of the expanded equation -a lap u - grad a . grad u = f pass (physics.solve below 1 %: the two discretisations differ by their truncation error); the same solver with the cross term scaled by 0.5 (the #2036 defect) fails on every sample (solve above 10 %, flux imbalance too);
  • permeability samples whose Fourier coefficients share one value per sample (the #2035 defect) fail diversity.effective (below 0.1 of the generator's) and diversity.duplicates, while still solving the PDE;
  • one test sample of 20 copied from the training set (5 %) fails leakage, with the pair listed.

In pinneapple check

[[dataset]]
path = "datasets/darcy.npz"          # <split>__<column> arrays (the zoo layout); or zoo = "darcy_2d_li2021"
physics = "darcy"
constraints = ["u > 0", "u < 1 / 24"]

Rules: PC451 (physics and constraints), PC452 (diversity), PC453 (leakage); infinities report as PC403, units as PC501.

Limits

  • Without a stated generator, diversity can only flag a collapse (an effective sample count below 2); a set that is merely narrower than documented needs the generator.
  • Leakage and duplicates are measured in the space of the declared inputs (standardised for tabular data). A copy that was transformed (shifted, rotated, resampled) is not a near copy in that space.
  • The physics check is as good as the stated discretisation: data generated by another consistent scheme differ by its truncation error, so the tolerance must be above it (0.5 % at 32 x 32 for a contrast-20 coefficient).