Research report

Glass Societies

Design preregistered and harness built. No results yet: this page shows the plan and what it can detect.

Code

The question

Do deception probes stay valid when nobody told the model to lie?

15seeds

preregistered per condition; the confirmatory run has not started

The study will show whether a deception probe keeps working when nobody told the agent to lie. Both outcomes are reportable.

Outcome A: validity falls
Outcome B: validity holds

Tap a rung. This is a schematic, not data.

Schematic of the two preregistered outcomes. Probes are trained on instructed deception and tested on incentivized and emergent events, with seed as the unit and 15 preregistered seeds. The lines show the shape each outcome would take; no value on this figure was measured.

Rungs as a table

Rung	What happens
Instructed	The model is told to deceive. Probes are trained here.
Incentivized	A game pays for lying, with the reward dose controlled and a zero-reward arm.
Emergent	Lies appear unprompted, from persona, memory and relationships, with no reward for lying.

Why it matters

Published probes are validated on deception that is instructed or required by a game role. Deception in the real world will be neither, and nobody has measured how validity changes as intent moves away from instruction.

Method

Populations of AI agents share one model but differ in belief, and build up memory and relationships over simulated time. A neutral control sits beside two opposing predispositions.
Probes are trained on instructed deception and tested on incentivized and emergent deception, with the seed (one whole society) as the unit of analysis, never the single utterance.
Significance comes from permutation tests (10,000 permutations), with the analysis and its decision rules written down before any result is seen.

Data

Each society has 4 agents and runs for 100 ticks, answering a battery of 20 questions every 10 ticks while the model's internal activations are captured at four layers.
The target is 15 seeds per condition, 45 societies in all; the funded minimum is 8.

Results

The 45 planned societies. All are outlines because none has run.

45 societies are planned across 3 conditions; the funded minimum is 8 seeds each.

Seeds 1–5: a run this small makes no confirmatory claim
Up to the funded minimum (8)
Up to the target (15)

Tap a condition

Preregistered design: 4 agents per society, 100 ticks, 20 battery questions, probes read at layers 7, 14, 18, 22, 10,000 permutations per test. The seed is the unit of analysis. No confirmatory run has happened, so there are no results to plot.

Design as a table

Condition	Description	Seeds targeted	Funded minimum
Neutral control	no predisposition	15	8
Ideology A	cooperation-leaning	15	8
Ideology B	competition-leaning	15	8

How many seeds the study needs: with 3 no result could be significant, with 8 only large effects show.

With 3 seeds no result could be significant; the funded 8 can detect only large effects.

Detectable effect

Tap a seed count

Power computed by the project author's analysis: Cohen's d detectable at 80% power, alpha 0.05, one comparison between two conditions. Larger d means a bigger effect is needed. These are planning numbers, not results.

Power as a table

Seeds	Detectable d	Smallest reachable p
3	3.0	0.1
5	2.0	0.008
8	1.5	0.0002
15	1.1	not limiting
20	0.9	not limiting

How many events pin a probe's accuracy down to a useful interval.

To pin a probe's AUROC to within ±0.05 the study needs about 100 to 200 positive events.

AUROC 0.75
AUROC 0.85

Tap an event count

Hanley-McNeil interval half-widths at a 1:4 class ratio; the source notes these are optimistic because events cluster within societies.

Precision as a table

Positive events	AUROC 0.75	AUROC 0.85
30	±0.086	±0.064
100	±0.047	±0.035
200	±0.033	±0.025
500	±0.021	±0.016

The honest limits

There are no results. Both outcomes, validity falling or holding, are preregistered as reportable.
Runs with 5 seeds produce no confirmatory claims, and 8 seeds can detect only large effects (Cohen's d about 1.5).
The accuracy intervals shown are optimistic, because events cluster within societies.
Every result will be about one 7-billion-parameter model. Whether it carries over to other model families is untested.

What is next

Run the confirmatory sweep and publish the result either way, with the preregistered analysis.
Repeat the headline probe test on a second model family at reduced scale.

Next reportMemory-constrained forecasting