Learned Simulators: Early Warnings

The result

We trained five copies of a small network on 512 flows and tested them on 704 new ones: 64 like the training flows, and 32 at each of five growing shifts along four axes (faster flow, new shapes, obstacles off center, larger obstacles). Every warning signal, its threshold and the way they’re combined were fixed on a separate development set before any test flow was looked at, and the test was written down in advance.

The warnings work. Outside the training range, a combination of all the signals ranks frames by whether the network is about to fail with an AUROC of 0.92–0.96 on every axis, and it was on before every failure, typically 19–30 frames ahead (half to most of a shedding cycle). Checking the network’s frames for the divergence a real fluid can’t have is nearly as good and, with the spread between the five networks, the best at telling which runs at the same shift are going wrong. But far out, a simple score of how unusual the inputs are warns as early as any physics check: it knows the case is new from the first frame.

Within the training range, the warnings mostly tell the time. The network drifts off the real flow there too, at a median of frame 87, and a signal that just counts frames ranks those moments almost as well (0.98) as the best checks (0.99). Compared only at the same frame, the best checks still score about 0.89, so they see something the clock doesn’t.

Warning doesn’t make the network worth using. A policy that hands back to the solver when the combination alarms, runs it for 40 frames and resumes, still failed the decision check (frequency, lift swing and drag of the vortex street) in 44–66% of runs, and cost more than running the accurate solver alone: the combination needs all five networks every frame. On this site’s own GPU code the solver is faster still, about 7 times faster than one network step. Training the network with noise on its inputs delays failure about threefold but makes it harder to see coming: almost every signal warns worse.

One network design, one flow, CPU timings for the study. A stronger network or faster hardware for it would change the cost side; the warning results are about this class of emulator.

Loading the results…

Conclusion

A neural network trained to stand in for a flow solver does fail when a case strays from its training, and its own output warns of it. Combining simple checks, such as whether the flow obeys the laws a fluid must, how much several copies of the network disagree, and how unusual the input is, flagged failures outside the training range well and typically 19 to 30 frames ahead, and far out, a plain measure of how unfamiliar the case is warned as early as any of them, from the first frame. Inside the training range the warnings mostly track time: the network drifts as it runs, and the clock alone predicts that nearly as well. But warning was not enough to make the network worth using. Handing back to the solver whenever a warning came on still got the quantities an engineer needs wrong in 44–66% of runs, and cost more than running the accurate solver alone. For this kind of learned simulator, the warnings are a useful safety check, not a way to make it pay; the open question is whether larger networks, or hardware that suits them far better, would change the cost side.

The science

The solver is a lattice Boltzmann method: the fluid as populations of particles moving and colliding on a grid, which recovers the Navier–Stokes equations for slow, weakly compressible flow. Behind a cylinder, above a Reynolds number of about 47, the wake sheds vortices alternately from each side, the von Kármán vortex street, and the obstacle feels a lift that swings at the shedding frequency. The solver passed the standard cylinder benchmark to within 3%.

The network is a U-Net, a convolutional network common in learned simulation, that steps a 2× coarser copy of the flow 25 solver steps at a time. It sees the flow, the obstacle and the Reynolds number, and was trained on circles 12–16 cells across, near the middle of the channel, at Reynolds numbers 80–120. Learned simulators are known to fail outside their training data, and studies keep finding that comparisons against weak baselines overstate them. The open question here is whether the network’s own output carries a usable warning, how early, and whether acting on it pays.

The signals: divergence, mass flux and the momentum residual check the network’s frames against the equations a fluid obeys; wall slip checks that nothing flows inside the obstacle; spectral pile-up looks for energy collecting at the finest scales; ensemble spread measures how much five networks trained the same way disagree; input novelty measures how unlike the training cases the input is. None of them sees the solver.