How close the answers are
Accuracy is measured on questions real people have answered but the model never saw. Some results are for each person, others for the audience as a whole. Tools built on prompting rarely say how they measured.
What an accuracy figure measures
Two levels of accuracy
A score can check the percentage of an audience choosing each answer, or check each simulated person against the real one. The same answers can score perfectly on the first and badly on the second.
Ten people answer one question on a scale of 1 to 5, and a model simulates each of them. On the left, the bars compare how many real and simulated people chose each answer. On the right, each person’s real answer sits beside their simulated one. An audience-level score would call this model perfect, and an individual-level score would say it got 3 of 10 people right.
The counts match on all five answers
- Person 1 1 1
- Person 2 2 2
- Person 3 2 2
- Person 4 3 2
- Person 5 2 4
- Person 6 4 3
- Person 7 5 3
- Person 8 3 5
- Person 9 1 2
- Person 10 2 1
3 of 10 people get their own answer
What a percentage can mean
One system can be called 68.85% or 85% accurate, depending on the metric. An audience-level figure can be high while most individuals are wrong.
| Metric | What it measures | Level | Example |
|---|---|---|---|
| Exact match | Share of questions where the simulated person picks the real person’s answer | Individual | 68.85% on the General Social Survey Park et al. (Simile), 2024 (v1) |
| Ratio to human retest | Exact match divided by how often people repeat their own answers when asked again | Individual | 85%, from the same 68.85% Park et al. (Simile), 2024 (v1) |
| Ordinal alignment | 1 minus the average distance between the simulated and real answer on a scale, as a share of the scale | Individual | 0.717 to 0.775 Toubia et al., 2025; Kinzinger and Hartmann, 2026; Dahiya (Semilattice), 2026 |
| Correlation across questions | Whether the model ranks questions or effects the way people do | Audience | r = 0.98 for experiment effect sizes Park et al. (Simile), 2024 (v1) |
| Correlation across people | Whether the model tells people apart on one question | Individual | r ≈ 0.20 in new studies Peng et al., 2025 |
| Distribution overlap | How much the simulated and real percentages for each answer overlap, question by question | Audience | 86% Artificial Societies |
| 1 − mean absolute error | 100 minus the average gap, in points, between the simulated and real percentages | Audience | 96%, a best case Electric Twin |
| Mean absolute error | The average gap, in points, between the simulated and real percentages | Audience | 3.53 points Aaru, 2026 |
| Undefined percentage | A single figure published without the formula behind it | Unknown | 88% similarity Evidenza |
What is reported at each level
The ranges published so far.
IndividualIndividual level
Each simulated person’s answer is checked against the real person’s answer. Research papers on twins and trained models since 2024 score this way. So do three of the six providers, and two of them publish a figure.
- Ordinal alignment
- Questions with ordered answers, such as scales
- 0.72 to 0.78
- Out of 1. Random answers score about 0.6. The same people asked again score 0.82.
- Toubia et al., 2025; Peng et al., 2025; Dahiya (Semilattice), 2026
- Exact match
- Any multiple-choice question
- 66% to 69%
- The share of answers exactly right. On the General Social Survey, guessing gets 27%. The same people asked again matched 81%.
- Park et al. (Simile), 2024 (v1); Park et al. (Simile), 2024; Dahiya (Semilattice), 2026
AudienceAudience level
The percentage of simulated people choosing each answer is checked against the real percentage. Three of the six providers report this.
- Mean absolute error
- The percentage choosing each answer
- 3.5 to 4 points
- The average gap between the simulated and real percentages. The range runs from one provider’s average to another’s best case. A model can match the percentages while giving most individuals the wrong answer.
- Aaru, 2026; Electric Twin
UnknownTools built on prompting
Tools built on prompting often publish a single figure without saying how they measured it.
- Method usually not published
- “88% similarity”
- These figures are published without saying how they were calculated, so they can’t be checked or compared.
- Evidenza
What research papers report
Ordinal alignment, the score most papers use
Most research papers that score each simulated person use ordinal alignment. Each answer gets credit for how close it is to the real one on the scale. A score of 1 means every answer matched.
The same people, asked twiceReal people on one panel, asked the same questions at least two weeks apart, matched their own answers this closely. Headline scores in published research run from 0.717 to 0.775. Some papers report their best model, some an average and some a median, as labelled under each. Toubia et al., 2025
Why research uses ordinal alignment
It gives partial credit on scales: answering 6 when the person said 5 scores better than answering 1. It also turns scales of any length into a score from 0 to 1, so papers can compare results.
Used in 5 papers: Toubia et al., 2025; Kolluri et al. (Simile), 2025; Peng et al., 2025; Kinzinger and Hartmann, 2026; Dahiya (Semilattice), 2026
Most of these papers share authors, or took the score from Twin-2K-500. So it is the habit of one line of research, not an agreed standard.
Its problems
- A high floor. Random answers score 0.59 to 0.63, so the useful range is roughly 0.6 to 0.82. Toubia et al., 2025; Peng et al., 2025
- It rewards the middle. A model that answers near the middle of every scale scores well. In one study GPT-4o did this, and beat the model whose answers were spread most like real people’s. Kolluri et al. (Simile), 2025
- Personas add little. Giving every twin the same empty persona scored 0.734, against 0.748 with each person’s full profile. Peng et al., 2025
- Only scales. It can’t score unordered choices, multi-select or open-ended answers. On two-option questions it is the same as exact match. Peng et al., 2025
An alternative. Macro-F1 is the standard score for classification in machine learning. It checks whether a model tells people apart. A model that always gives the most common answer scores low. It counts a near miss on a scale as fully wrong, and none of the survey-simulation benchmarks here use it as their main score. Holtdirk et al., 2025
Surveys of real people have errors too
Simulated answers are scored against real survey answers. Real surveys get things wrong too.
- People change their own answersAsked the same General Social Survey questions again two weeks later, people gave the same answer 81% of the time, and changed nearly one in five. Park et al. (Simile), 2024 (v1)
- Opt-in panels miss benchmarksAgainst 28 official benchmark figures, online opt-in panels were off by 5.8 points on average. Panels that recruit people at random were off by 2.6. Mercer and Lau (Pew), 2023
- Some respondents are not genuineEven after quality checks, 3% to 7% of interviews in opt-in panels came from bogus respondents, against 1% in panels recruited by post. Kennedy et al. (Pew), 2020
- Rare answers get inflatedIn one opt-in poll, 12% of adults under 30 said they were licensed to operate a nuclear submarine. The real share rounds to 0%. Mercer et al. (Pew), 2024
Which critiques apply to which models
Each well-known criticism of simulated people tested only one of the three ways of building them. Products can also combine methods, for example by giving a fine-tuned model each person’s interview. The providers page shows which method each provider uses.
Prompted
A general-purpose model is told who to be, such as “You are a 34-year-old nurse in Leeds.” AI personas and silicon samples are built this way.
Models flatten identity groups into stereotypes
Four language models were each told to answer as one of 16 identities. Their answers were compared with those of 3,200 people who hold those identities.
Other workGiving each simulated person the real person’s interview, instead of their demographics, made accuracy more even across racial and ideological groups. Park et al. (Simile), 2024
Synthetic answers are not reliable for statistical inference
ChatGPT was given each respondent’s demographics and politics from a US election survey. Its average answers matched the survey, but they were less varied. When the answers were analysed, 48% of the regression results differed significantly from the survey’s. The same prompts also gave different answers after a model update.
Other workA model fine-tuned on answers from 210 experiments gave a spread of answers closer to people’s on experiments it hadn’t seen. Kolluri et al. (Simile), 2025
Models’ opinions match some groups and not others
Models’ answers to US opinion polls were compared with those of 60 demographic groups. Telling a model which group to answer as did not close the gaps.
Other workReweighting personas to a human reference is being tested. Hu et al., 2025
Grounded
A general-purpose model reads one real person’s interview or survey answers before it answers as them. The digital twins in Toubia et al. and Peng et al. are built this way.
Twins are only slightly better than a model with no persona
Twins were built from each person’s answers to 500 questions and tested in 19 studies. They scored 0.748. The same model told nothing about the person scored 0.734. Across people, the twins’ answers correlated with the real ones at r ≈ 0.20.
Trained
A model is trained on what real people said and did, usually by fine-tuning a language model. Some trained models are not language models at all.
Not tested by these critiques
None of the critiques above tested a trained model. The published results for trained models come from the teams that built them. Kolluri et al. (Simile), 2025; Dahiya (Semilattice), 2026
LimitTraining needs a large set of real answers, and a model per audience needs a set for each. Kolluri et al. (Simile), 2025; Dahiya (Semilattice), 2026
What researchers are working on
- Variance. Simulated people’s answers vary less than the answers of the real people they model. Being tried:
- Fine-tuning on human responses Kolluri et al. (Simile), 2025
- Reporting the spread of answers and correlation across people alongside accuracy Peng et al., 2025; Kinzinger and Hartmann, 2026
- Training a separate model for each cluster of similar people, though its paper reports the spread is now above the human level Dahiya (Semilattice), 2026
- Bias. Models lean towards agreeing, and describe groups the way outsiders see them. Fanous et al., 2025; Wang et al., 2025 Being tried:
- Reweighting personas to a human reference Hu et al., 2025
- Grounding each agent in the person’s own interview instead of their demographics Park et al. (Simile), 2024
- Training on each question framed both positively and negatively, which reduced the lean towards agreeing without removing it Dahiya (Semilattice), 2026
- Prompt brittleness. Answers shift with the wording of a question and the order of its options. Being tried:
- Testing whether simulated answers shift with rewording by as much as people’s do Artificial Societies, 2026
- Shuffling the answer options in training and in testing Dahiya (Semilattice), 2026
- Generalisation. Accuracy drops in new studies, and on questions unlike the data the model was built from. Being tried:
- Measuring twins in new studies Peng et al., 2025
- Predicting each result’s error, or declining to answer Simile; Listen Labs