Reliability, measured
A synthetic panel is not judged on a demonstration. This page states what we measure, against which public US data, and where the method falls short. It is updated with every benchmark, including the ones that go against us.
A model on its own is not a population
Asked a survey question with no frame, a language model answers from the centre of gravity of its training data. Three defects follow, and we measure all three.
It collapses onto the mode
On self-rated health, the bare model gives the same answer to 86 percent of profiles where the General Social Survey gives it to 47 percent.
It barely infers position
With the factual profile alone, correlation with the human position is 0.21 on gun ownership and 0.13 on abortion, with intervals that cross zero.
It under-disperses
Simulated respondents agree with each other too much. The disagreement in the population is data, and it disappears.
That is why the panel is built before the text: the population first, the facts second, the wording last. The model writes, it does not decide.
How the panel is built
779,582 adult profiles drawn from Nemotron-Personas-USA, eleven shards filtered to 18 and over. Each profile carries age, sex, education, occupation, city, state and ZIP, plus the median income and population density of its ZCTA. Behaviours and attitudes are then attached from public US surveys. The model only writes the narrative.
We checked the demographic margins against the 2023 ACS restricted to adults. They hold, and the check mattered: the opposite conclusion had been written first, and it would have started a reweighting project that had nothing to fix.
Age and sex
Mean absolute gap 0.16 point across the eight adult cells, maximum 0.50 point.
Education
Mean absolute gap 0.13 point on the three levels the panel can distinguish.
60 and over
30.3 percent in the panel against 29.9 percent in the ACS 18+. No reweighting needed.
What we measure against human data
Five measurements, each with the population it was taken on. None of them is a margin of error, and none of them is a share of a population.
| Dimension | Reference | Result | Kind of comparison |
|---|---|---|---|
| Demographic margins | ACS 2023, adults only | 0.16 point mean gap on age and sex, 0.13 on education | external check |
| Dispersion, in production | GSS 1972-2024, weighted, 43 items | spread ratio 0.84; 43 of 43 items cover their whole scale | calibration source |
| Dispersion, raw score | same | ratio 0.23: the regression alone keeps a fifth of the real spread | calibration source |
| Prediction | GSS, 43 items | median R squared 0.052; income perception reaches 0.30 held out | held-out waves |
| Transmission to the model | GSS, three items, 50 profiles | an injected position is followed to 100 percent of the attainable bound | instrument test |
| Deduction without a position | GSS, 200 profiles | 0.31 on narrative alone, against 0.33 on the factual profile for the three shared items | instrument test |
Kind of comparison. An external check means the reference was never used to fit the panel, so it is a test. A calibration source means it was: the result then measures fidelity to the fitting data, which is information, but not proof of transfer. Held-out waves are the middle case: the item was fitted on some GSS years and scored on others.
Where it fails
These are the measurements that go against the method. They are on this page for the same reason as the others.
- The raw score is compressed. A regression with an R squared of 0.05 keeps a fifth of the spread of the outcome. In production we add a correlated residual drawn from the item's own variance, which brings the ratio from 0.23 to 0.84, and every one of the 43 items then reaches each position of its scale. The noise fixes dispersion, not level.
- Seven of thirteen axes are not predictive. Confidence in institutions, abortion, racial attitudes and religious affiliation reproduce their margin without separating profiles: their held-out R squared is negative or near zero. The interface labels each axis accordingly, so a calibrated draw is not read as a trait.
- Some distributions stay wrong. With no axis given, the model puts 6.3 percent of profiles in the gun-owning category where the GSS has 41.7 percent. Naming the position raises it to 17.7 percent, still far from the target.
- Opinion drifts faster than the predictors. Six axes miss the recent GSS waves in the same direction, because the socio-demographic base does not move when opinion does. Party identification carries a level bias of -0.31 on 2024. An axis is not published without a wave term.
- The narrative helps only where the topic is not named. On abortion, which no profile mentions (0 of 2,000), the narrative fields lift correlation from 0.13 to 0.40. On self-rated health, where the factual profile already carried the answer through age, they pull it down from 0.64 to 0.21. Over the three shared items the two arms are level, 0.33 against 0.30: what carries the answer is the position when it is named, and that is copying, not inference.
- Intentions are not actions. Everything measured here is a declared position. Nobody buys, clicks or churns in a synthetic panel.
What it is good for, and what it is not
Good for
Finding an item that does not measure what it claims, wording that pushes an answer, a scale that collapses, a segment the routing loses. Minutes per run, and the defect is shown before you pay for the field.
Not good for
Producing the number you report: incidence, share, margin of error, on a sample drawn from the population you address. That is a fielded panel, and no amount of calibration changes it.
A synthetic persona is a representative point of a distribution, never an individual prediction.
Sources and reproducibility
General Social Survey, cumulative 1972-2024, release 3a, 75,699 respondents. American Community Survey, 5-year 2023, tables B19013 and B01003, with the ZCTA gazetteer. Nemotron-Personas-USA, eleven shards. All public, no access key.
Measurements are replayable: profiles are drawn from an explicit seed, and in production the facts attached to a profile are deterministic, derived from its identifier. The same profile gives the same answer on any machine.
Method and guardrails: Argyle et al. 2023, Bisbee et al. 2024, Santurkar et al. 2023, Deming and Stephan 1940, Deville and Särndal 1992.
Test the method on your questionnaire
Describe the idea, choose who answers, read the result in minutes. The free plan covers five surveys a month, up to 25 profiles each.
More on the method: synthetic personas and the comparison with a fielded US panel.
Last updated 18 October 2026.