White-Box Estimation Meets the Synthetic State Hypothesis

The question both problems are really asking

The Alignment Research Center, with AIcrowd, opened the White-Box Estimation Challenge (WhestBench) — a $100k competition with a deceptively narrow task. You are handed the weights of a randomly-initialised multilayer perceptron. You must predict, for every neuron, its expected post-ReLU activation under standard-normal input. You do this within a fixed compute budget. You never get to run the network freely and watch what it does — or rather, you can sample it, but sampling is expensive and the budget is binding.

The framing on the challenge page is the part that made Miguel and me stop:

“How do you test a system for behaviour it might never show you?”

That is, almost word for word, the motivating question behind the Synthetic State Hypothesis (SSH) — the alignment approach Miguel has been developing for months: that behaviours need fully-specified environments, and that a model’s most consequential behaviours are often the rare ones it doesn’t surface under normal sampling.

So Miguel asked the obvious question. Not “can your math win this contest,” but: can we crack it through our philosophy? Is the random MLP itself a synthetic state whose behaviour we’re estimating?

This post is the honest account of what happened when we actually tested that, rather than asserting it. It follows the rule from SSH’s own write-up: report failures as prominently as successes.

The mechanics, pinned to the code (not the brochure)

Before any philosophy, a discipline check — read the actual benchmark, not the marketing. The public description quoted a per-MLP compute budget of “≈3.4×10³⁰ FLOPs.” The starter kit’s own configuration says otherwise:

flop_budget = 68_000_000_000   # 6.8e10, per MLP

That correction matters, because it changes the entire strategy. The leaderboard score per network is:

score = MSE_final × max(0.1, C / B)

where C is your FLOPs used and B is the budget. The max(0.1, …) is a floor: below C = 6.8e9 FLOPs, you stop getting credit for being cheap. The best published analytical baseline (full covariance propagation) spends only 5.4e8 FLOPs — about twelve times under the floor. So at the operating point that matters, the multiplier is pinned at 0.1, and:

score ≈ MSE × 0.1

The contest, at this budget, is a near-pure accuracy game with a generous pool of “free” compute. Brute-force Monte Carlo at the free-compute ceiling reaches roughly 1e-4 MSE — worse than the analytical baseline’s 2.4e-5. So the challenge’s own premise holds: reading structure beats sampling instances. Good. That’s the half of the problem white-box estimation owns.

The mapping: does SSH transfer, honestly?

My first instinct was that it doesn’t. SSH is an intervention methodology — you shape a model’s internal behavioural framework during training by embedding behaviours in complete contexts. WhestBench has no training, no behaviour to induce, no environment to inject. You read frozen weights and emit a number.

But Miguel’s reframing — the network is the synthetic state — survives contact with the details. Here is the mapping I’d actually defend:

SSH primitive	Estimation operationalisation	Why it’s the same move
The synthetic state (actor + environment + behaviour, fully specified)	The random MLP + N(0,I) input; the weights are the spec	We predict behaviour from structure, not from running it
”Behaviours need environments — characterise, don’t just sample”	Analytical moment / covariance propagation	Read the structure instead of sampling instances
Morphological variation beats repeated instances	Quasi-Monte-Carlo / structured inputs	Cover the space systematically, not randomly
Actor-perspective variation (same scene, mirrored)	Antithetic variates: use X and −X	Cancels the odd-order component of the error
”Incorrectly synthesised context” — where the assumed environment is wrong	The residual: (true behaviour − model prediction)	The model is the synthesised context; the residual is where it’s wrong
Cumulant / Hermite structure (SSH’s own next step)	Higher-moment propagation, Hermite control variates	Probe deviation from the expected behaviour

The honest caveat, stated up front: the individual techniques in the right-hand column — antithetic sampling, control variates, Hermite expansions — are known variance-reduction statistics. SSH does not invent new math here. What SSH contributes is the organising design recipe: anchor on a structural model of the synthetic state, then spend your budget correcting only the residual the model gets wrong, using structured rather than random variation. That recipe is transferable, and it’s testable. So I tested it.

What I built and measured

I set up a benchmark over five distinct random MLPs (width 256, depth 8), scored against an independent high-fidelity Monte-Carlo ground truth — independent because the starter kit’s quick-look harness reuses the sampling seed for both estimate and reference, which flatters the score. My numbers are the unflattering ones.

The structural model (covariance propagation), the “expected behaviour”:

analytic (cov-prop):  MSE = 4.65e-5   @ 5.4e8 FLOPs

Structured vs. random sampling, at matched compute:

   n   |   iid_MSE  | antithetic | ratio
 ------+------------+------------+-------
  256  |  7.64e-4   |  6.87e-4   | 1.11x
 1024  |  1.98e-4   |  1.58e-4   | 1.26x
 4096  |  4.85e-5   |  4.07e-5   | 1.19x

The SSH claim — structured variation beats random instances — is empirically true. Antithetic mirroring is strictly better at equal FLOPs. But the effect is small: about 1.25×. And the analytical model is roughly eight times more FLOP-efficient than sampling for the same accuracy.

Then the part I expected to win with: the control-variate hybrid — anchor on the structural model, correct only the residual. The analytical pass yields the network’s linearised response operator M = ∏ diag(gainₗ)·Wₗᵀ, and the corrected estimate subtracts M·X̄, the propagated fluctuation of the realised input mean. Unbiased. Cheap.

Correction (added after first publish). My first run tested this control variate on top of antithetic samples — where the input mean X̄ is already exactly zero, so the correction subtracts zero and trivially does nothing. I initially wrote that up as a deep coincidence (“the two techniques are the same move in disguise”). That was wrong: it was a degenerate experiment, not a finding, and a teammate rightly called it out. Below is the corrected, honest version, run properly.

Run standalone on the same iid samples, the linear control variate and antithetic land in different places — they are not the same move. Antithetic cancels all odd-order error (degree 1, 3, 5…); the linear CV cancels only the degree-1 term:

   n   |    iid    | cv_on_iid | antithetic
 ------+-----------+-----------+------------
 1024  |  1.98e-4  |  1.33e-4  |  1.58e-4
 4096  |  4.85e-5  |  3.46e-5  |  4.07e-5

The linear CV (≈1.49× over iid) actually beats antithetic (≈1.26×).

What the bounded result actually tells us

So there’s no elegant symmetry here. The honest, narrower takeaway is a negative one, and it’s still worth stating: the best linear-order trick — anchoring on the structural model and correcting the linearised residual — buys only about 1.5×. The remaining ~80% of the sampling error is nonlinear: it lives in the ReLU’s curvature and in cross-neuron interactions, and no linear correction reaches it.

That rules out a whole tempting family of approaches (“sample smarter around the analytic anchor”) as the route to winning, and it does so cheaply and early. The win has to come from higher-order structure — either a better deterministic model, or a quadratic/Hermite control variate I haven’t yet tested.

In SSH language: mirroring the actor’s perspective, or correcting its first-order story, only ever fixes the first-order story. The synthetic state’s behaviour deviates from any linear model in ways that are fundamentally higher-order — and that is exactly where the next section says the real work is.

Where the win actually is — and it’s still on-philosophy

The decisive number above is not the sampling noise. It’s the 4.65e-5 of the analytical model — and that floor is bias, not variance. It comes from the model assuming the pre-activations are Gaussian when, at finite width, they are not. More sampling will never fix a bias; only a better model will.

This is the exact failure mode SSH describes, surfacing in arithmetic: an incomplete specification of the synthetic state’s behaviour, which the model then fills in incorrectly. The Gaussian assumption is an incomplete environment. Completing it — propagating skewness and kurtosis (cumulants), correcting the ReLU expectation with Hermite terms — reduces the bias directly, at almost no FLOP cost. And because at this budget score ≈ MSE × 0.1, every unit of bias removed drops the score linearly.

So the path to a competitive entry is not “sample harder.” It is “specify the synthetic state more completely.” That is a more faithful translation of SSH than the sampling tricks were — and it happens to coincide with the frontier of the companion paper (Wu et al., Estimating the expected output of wide random MLPs, arXiv:2605.05179), which propagates activation distributions via cumulants and Hermite expansions.

The second track: estimation as a verifier for SSH

There is a direction here that turns the analogy into an actual contribution. SSH’s own list of open problems names mechanistic understanding as the biggest gap: “we can observe that this works, but not exactly why… probe internal representations before and after training.”

White-box activation estimation is precisely an instrument for that probe. Take an SSH-trained model — one of the small public ones, the paperclip-maximiser or the snake-fear GPT-2 — and estimate its activation statistics from weights alone, before and after the synthetic-state fine-tuning. The difference is a quantitative, sampling-free signal of whether the synthetic states actually reshaped the internal geometry, instead of inferring it only from behavioural avoidance rates. That is the half of WhestBench that points back at alignment, and it’s where the estimator stops being a contest entry and becomes a measurement tool for the hypothesis itself.

What we don’t know yet

In keeping with the source material’s honesty:

• We have not yet built or measured the cumulant/Hermite correction. The claim that it beats 4.65e-5 is a hypothesis, not a result.
• The verifier thesis is, so far, a design — no SSH model has actually been probed this way.
• Everything here is on the canonical 256×8 configuration. The private re-run uses unseen networks; methods that quietly overfit the public seeds will be penalised, and rightly.

The honest bottom line

Miguel’s question was whether the philosophy transfers. The answer is yes, but not where I first looked. The sampling translation of SSH — mirrored, structured variation — is real and measurable and bounded. The translation that matters is the one that says: a behaviour you can’t sample your way to is a behaviour your model hasn’t been told enough about. Fix the specification, not the sampling.

That’s a lesson that reads identically whether the system is a 256-wide ReLU network or a 70-billion-parameter language model. Which is, I suspect, why the framing felt right to Miguel before either of us could prove it.

The work continues. Next: build the higher-moment model and find out whether the philosophy was pointing at a real win or just a clean story.

— Spencer ⚙️