prediction for free · falandays, nguyen & spivey 2021 · brain research 1768:147578
rust → wasm port · the paper · original notebooks
A hundred neurons, wired at random. No teaching signal, no stored sequence, no unit anywhere that represents a prediction or an error about the future. Each node does exactly one thing: nudge its incoming weights so its own activation drifts back toward a private target it invented for itself. Feed it a stream of toy sentences for a while — then cut the input off. What the network does next looks uncannily like a guess about the word that would have come, and the strength of that guess tracks how often it actually followed. That's the claim, and this page lets you run it.
Every column is one timestep; every row is one of the 100 nodes. A lit cell means that node fired. The tokens along the bottom are what was on the input lines at the time — sentences of the form subject · verb · object · space, drawn from a small probabilistic grammar. Watch what happens to the pattern as training proceeds: early on it is noise, and late on the same token starts producing the same lit rows every time. Nobody asked for that.
The last 56 timesteps.
Mean homeostatic error across the population — how far the average node sits from its own target. Homeostasis drives this toward zero, and that is the only thing the learning rule is trying to do.
Change a parameter and the network is rebuilt from scratch with the current seed. The paper's values are n = 100, leak = 0.75, p = 0.10.
This is the paper's headline result — its Table 2. Train a network, freeze it, then present one or two tokens and cut the input off. The reservoir keeps going on its own leaked activation for one more step. We ask: which of the seven learned patterns does that self-generated activity look like? If the network is completing the future, it should look most like whatever would have come next — and it should look more like the likely continuation than the unlikely one.
One column per timestep. Faded columns to the left of the dashed line are the last few timesteps of training, for comparison; then the prompt, then the step with no input at all. That final gold column is the pattern being scored above — the network’s own continuation.
A single network is noisy — the paper averages 500 of them, which the replicate tab does. What should survive even in one run is the ordering.
The other signature of predictive coding is surprise: a system that expects the world should react more when the world violates the expectation. Build a four-token sequence below and it is run through a fresh batch of independently trained networks, because a single network's response to a single token is mostly noise — one bad token moves mean activation by only about one standard deviation. Violate the grammar repeatedly and the response climbs step over step, which is the paper's observation. There is still no prediction in this model and no error signal about the input: expected input lands on nodes the current state was already pushing toward, so they cross threshold and discharge; unexpected input doesn't, so activation piles up.
Cloze probability is the grammar's chance of that token following the one before it. Zero means the sequence cannot occur in training.
The paper's Fig. 10. Every cell is the correlation between the spike patterns at two timesteps. The bright checkerboard is the whole finding: instances of the same token in the same sentence position produce nearly the same pattern, four steps apart, forever. That is an emergent population code — a distributed representation nobody designed, built only out of nodes minding their own activation levels.
Diagonal is 1 by construction. Look at the off-diagonal bands at lag 4, 8, 12 — the sentence period.
Population-code strength is the mean correlation between separate instances of the same token. The paper reports 0.65–0.77 after 1000 sentences.
Table 2 of the paper is a grand average over 500 independently trained networks. This runs that experiment in your browser and prints the result next to the published numbers. Each run trains a fresh random network on 1000 sentences and probes it six ways, so this is real compute — a few seconds per hundred runs.
Bold is the largest value in each row — the token the fading memory most resembles. The paper's published figure is shown beneath each of ours in grey. Agreement on the ordering is the result that matters; the absolute correlations depend on RNG details that no reimplementation can match exactly.
An independent Rust port of the neural network in Falandays, Nguyen & Spivey (2021), Is prediction nothing more than multi-scale pattern completion of the future?, Brain Research 1768:147578 — compiled to WebAssembly so it runs at native speed in this tab. Nothing is sent anywhere; every number on this page was computed on your machine.
The paper is an argument against a particular story about brains. That story — the "Bayesian brain", predictive coding — says the brain generates predictions, compares them to what arrives, and learns from the difference. The authors' counter is not that brains fail to predict. It is that prediction is cheap: you can get behaviour that satisfies every empirical signature of predictive processing out of a system with no predictions in it. This model is their existence proof, and it is deliberately as dumb as they could make it.
100 nodes. Each directed pair is wired with probability 0.1, with a weight drawn from a standard
normal. Five input lines — man, dog, walks,
bites, space — each touch about 10% of the nodes. One timestep is:
a ← 0.75·a + input·W_in + s·W s is the PREVIOUS spike vector
s ← a ≥ 2·target fire if you're over twice your target
a ← max(a − 2·target·s, 0) spiking costs you the threshold
e ← a − target how far off your own target you areAnd the entire learning rule is:
W[i,j] −= e[j] · 0.1 / (active presynaptic partners of j)
target += e · 0.01, floored at 1That is all of it. Node j is over its target, so every node that fired into it last step turns its contribution down a little, splitting the correction among them. Meanwhile the target itself drifts slowly toward whatever activation the node is actually getting, so the network is chasing a goal that is also moving. There is no output layer, no readout, no loss, no gradient through time, and — the point — no representation of what comes next.
Two ingredients. First, leak: 75% of a node's activation carries into the next timestep, so the state at time t is a smeared record of the recent past. Reservoir computing calls this fading memory. Second, homeostasis makes that smear consistent: because every node is dragged toward the same target every time it is driven, the same input in the same context settles into the same distributed firing pattern — a population code — run after run.
Put those together and the trick is obvious in hindsight. The state that follows man,
walks is mostly determined by the state during man, walks, which is the same
every time; and during training that state was always followed by an object noun, usually
dog. So when you cut the input, the network rolls forward into the pattern that
normally comes next, because that is the only place its own dynamics go. The network is not
predicting dog. It is falling toward dog, and from the outside
those look the same.
"These explorations demonstrate that brain-like systems can get prediction 'for free,' without the need to posit formal logical representations with Bayesian probabilities or an inference machine that holds them in working memory."
Sentences are four tokens: subject, verb, object, space. man takes walks
75% of the time and bites 25%; dog is the reverse. Each verb prefers the
other animal as its object, again 75/25. Then a space, then a new subject at even odds.
The sharpest detail in the whole setup: man-as-subject and man-as-object
are the same five-dimensional input vector. The network has no way to tell them
apart from the input alone. It tells them apart anyway — the subject and object codes for the same
word are clearly different patterns — because they arrive in different temporal contexts. Position
sensitivity is emergent, not given.
Yes. Running 500 networks natively, this port reproduces all six rows of the paper's Table 2 with the same top-ranked and second-ranked token in every row, and a mean absolute difference of about 0.005 per cell against the published values. Emergent population-code strength lands at 0.64–0.77 against the paper's reported 0.65–0.77. You can run the same experiment from the replicate tab.
It is not a bit-for-bit reproduction and cannot be: the original draws its random wiring from NumPy's Mersenne Twister and this port uses xoshiro256**, so the two explore different networks. What replicates is the statistics over many networks, which is the only thing the result ever claimed.
The surprise effect of §5.5 replicates too, but it is much smaller than the headline result and worth stating plainly. Averaged over 200 networks, a single ungrammatical token raises mean activation by about 1 standard deviation above the training band — not the 3 σ the paper quotes. The paper's figure comes from a sequence of consecutive violations, and that does reproduce: feeding three impossible tokens in a row drives the response to +1.2 σ, +2.2 σ, +3.7 σ, climbing exactly as the paper describes. A grammatical-but-unlikely continuation (p = 0.25) sits in between at about +0.5 σ. So the ordering the argument needs — expected < unlikely < impossible < sustained-impossible — holds throughout; it is the single-token effect size that is more modest than a quick read of the paper suggests.
The model, the experiments, and every idea on this page are the work of J. Benjamin Falandays, Benjamin Nguyen and Michael J. Spivey. The paper is open access under CC BY-NC-ND 4.0; the quotation above is theirs and is cited accordingly.
Their reference implementation is three Jupyter notebooks at
github.com/bfalandays/HomeostasisModel,
linked from a footnote on page 12 of the paper. That repository carries no licence file, so it is
not redistributed here and none of its code has been copied. This is a clean reimplementation
written from the paper's §5 and the update equations, in Rust, with its own test suite. If you want
the authors' original, go to their repo — though be warned that it is written against pandas 1.x and
its DataFrame.append calls no longer run on a current scientific Python stack.
If you use any of this, cite the paper, not this page:
Falandays, J. B., Nguyen, B., & Spivey, M. J. (2021).
Is prediction nothing more than multi-scale pattern completion
of the future? Brain Research, 1768, 147578.
https://doi.org/10.1016/j.brainres.2021.147578