Reading the results.

Most of the simulations involve chance: where cells start, which mutation happens, which way a bot turns. A seed is the number that fixes all of those random choices, so a run with the same seed and settings gives the same result every time. Running many seeds shows how much of a result is the method and how much is luck.

A control is the comparison that tells you what would happen without the thing being tested. In Biobot Tierra the control is a dish where nothing is inherited; in the Chip Archipelago it is a body optimized directly on the final chip, without stepping stones. A result is only as meaningful as its control: “speed rose by 49” means little until you know how much it rose in the control.

Mean ± standard error

Results are usually written as a mean plus or minus its standard error, for example +0.12 ± 0.04. The standard error is how much the mean would be expected to move if the whole experiment were rerun with new seeds. It is the spread of the individual results divided by the square root of their number, so it shrinks as more runs are added.

A rough rule: if the mean is less than about two standard errors from zero (or from the control), the result could easily be chance. A difference of +0.05 ± 0.03 is suggestive at best; +0.12 ± 0.04 is three standard errors and harder to dismiss, but see fair tests below.

Counts and sign tests

“Ahead on 10 of 12 seeds” is a count. If two methods were equally good, each seed would be a coin flip, and a sign test asks how often a fair coin gives a split at least that lopsided. 10 of 12 happens about 4% of the time by chance; 6 of 12 is exactly what a coin gives.

Many comparisons

Test enough conditions and one will look impressive by luck. If nine conditions are compared with a control, finding one that sits two standard errors away is not surprising. Biobot Tierra’s result section says this about its own best result.

The strongest evidence comes from tests set up so they can fail. Two practices recur on the site:

• Fresh seeds. While a method is being developed, it gets adjusted after looking at results. Those results are then no longer an independent test. Running new seeds afterwards, with nothing changed, checks whether the result holds. In the Chip Archipelago it didn’t: a clear advantage on the first 12 seeds vanished on 12 fresh ones.
• Pre-registration. Writing down the runs, the measure and the rule for success before running anything, then reporting the outcome whatever it is. The Archipelago’s full-profile pilot was run this way.
• Held-out tests. Scores measured on cases the method never saw while it was being fitted: the decoders in Growing Anthrobots are scored on bodies they weren’t trained on, and the walls in Worlds at the Boundary are confirmed on fresh runs and new layouts.

Cellular automaton

A grid of cells, each updated from its neighbors by the same local rule. See the Rule Observatory.

Fitness / score

The number a search tries to raise: distance travelled, how robust a body is, how far along a channel it gets.

Evolutionary search

Try variants of the current designs, keep the better ones, repeat. No gradients needed, so it works on any simulator.

Quality-diversity

A search that keeps the best design for each kind of behavior instead of one overall best (MAP-Elites is one).

Heritability

How much offspring resemble their parent in a trait: the slope of offspring value against parent value. 1 is an exact copy, 0 is nothing passed on.

Genetic drift

Change in a population's traits by chance, strongest when the population is small.

R²

The share of variation a prediction explains, scored on held-out data: 1 is perfect, 0 is no better than guessing the average.

Silhouette score

How well separated clusters are, from −1 to 1: near 1 is distinct groups, near 0 is no real grouping.

Low Reynolds number

The regime of very small swimmers in water, where drag dominates and nothing coasts: stop pushing and you stop.

WebGPU / JAX

WebGPU runs the simulations on your graphics card in the browser; JAX runs the same models in Python for the research runs.

For a worked example of most of these, read the Chip Archipelago results.