Market research with ChatGPT: what a chatbot cannot measure
Asking ChatGPT for market research is now the first reflex, and for writing it is a good one. It is not a measurement: you get an argued opinion, not the answers of a population. The difference is not the model you use, it is the method, and the method decides what you can do with the result.
What a chatbot does well
Writing the questionnaire
Rewording a badly put question, proposing a scale, spotting a leading phrasing: this is where it saves the most time, and no survey tool replaces it.
Documenting the market
A deep-research run gathers public sources and returns a report with its references. For a landscape, a list of players and public figures, it is faster than we are.
Exploring angles
Finding objections nobody thought of, on any subject, without opening an account with a research vendor.
On those three, a chatbot is faster and lighter than a panel. We are not trying to replace it: we are not an assistant, and the question we answer begins where the chatbot's stops.
What you get when you ask a chatbot for market research
You get a written answer. Ask it to play twenty profiles and it will return twenty paragraphs, often convincing ones, which is what makes the confusion frequent and expensive. Those twenty profiles are not drawn from a population. They come from your prompt, the count is the count you asked the model to play, and the table of answers that comes out is written by the model.
The consequence is easy to state and hard to see in use: when you count those answers, you are measuring the model, not a public. A sentence like 60 percent of your customers would accept this price increase then describes no population, and nothing tells you how far it sits from reality.
Two ways to answer the same question
The two columns describe two approaches, not two products: they are the steps a response goes through, and each one decides what you can do with the output.
| The step | Asking a chatbot | Asking a persona panel |
|---|---|---|
| Who answers | the model, playing the profiles you describe in your prompt | profiles drawn from a panel of 779,582 US adults, margins checked against the 2023 ACS, behaviours attached from public surveys outside the model |
| How many respondents | the count you ask the model to play: asking for four thousand opinions does not create four thousand distinct respondents | you choose it too, with a slider and fixed steps, inside your plan (25 personas on the free plan, 5,000 on Enterprise), and the draw can be restricted: age and gender from Pro, every criterion on Enterprise |
| Where the facts come from | the model's training data, collapsed into common sense, with no link to the population you ask about | public US data chosen to match the population under test: 13 dimensions and 43 variables per persona, seven of them calibrated draws rather than predictors |
| What you get back | a written answer, to be re-read | one answer per profile per question, kept |
| What can be recounted | nothing: the table of answers is written, counting it measures the model | everything: the figure is recounted from stored answers |
| Running it again tomorrow | the served model changes with each version, and nothing anchors the answer in fixed data | the model can change too: what does not move is the data anchoring, and that is what counts (same population, same facts, same seed, same distribution) |
| What it is worth for a decision | an argued opinion, useful to frame a question | a distribution, with the gap measured against named human surveys |
| What it does not say | no population estimate at all | not a purchase: a stated intention |
What separates the two columns is not the model's brand, it is the data. Our profiles are built on diversified data chosen to match the population on offer: the ACS margins for structure, public survey items for behaviour, and that correspondence is checkable. A chatbot has neither a population nor a correspondence: it has only what it read, collapsed into common sense, and it publishes no limits, answering with the same confidence where it is reliable and where it is not. The check takes one move: ask the same question twice, and see whether the set of profiles comes back the same.
Why a model on its own gets it wrong three ways
These are not defects of one product: they are what a language model produces when it answers with no frame. We measured them on our own output before correcting them.
It collapses onto the mode
On self-rated health, the bare model gives the same answer to 86 percent of profiles where the General Social Survey gives it to 47 percent.
It barely infers position
With the factual profile alone, correlation with the human position is 0.21 on gun ownership and 0.13 on abortion, with intervals that cross zero.
It under-disperses
Simulated respondents agree with each other too much. The disagreement in the population is information, and it disappears.
Building a population is what corrects these three. The protocols, the benchmarks against the ACS and the General Social Survey, and the places where convergence drops are on the reliability page.
Frequently asked
Can you do market research with ChatGPT?
Yes for the part that gets written and sourced: understanding a market, listing the players, writing the questionnaire, preparing an interview guide. No for the part that gets measured: what you get back is text, and nothing fixes the population it would describe.
Can a chatbot simulate a panel?
It can play the profiles you describe, and it does it convincingly. That is not a panel: the profiles are drawn from no defined population, and nothing anchors them in reality. A panel means a population that exists before the question, respondents whose traits map to real people, and named public data that makes the mapping checkable.
What is the difference between a chatbot and a synthetic persona panel?
The data, and its correspondence to the real population. Our panel holds 779,582 adult profiles drawn from Nemotron-Personas-USA, with demographic margins checked against the 2023 ACS and behaviours attached from named public surveys such as the General Social Survey. Each respondent is a profile, and each profile maps to a real category. A chatbot playing twenty profiles maps nobody to anybody.
Are my questions and results used to train a model?
Yes by default on ChatGPT's side: its own documentation says conversations may be reviewed and used to improve the models, with a setting to opt out. At Mirubiki, the opposite is the rule and it is not a setting you have to find: your questions, your surveys and your results are not used to train our models. Recipients and retention are in the privacy policy, twelve months at most after account deletion.
Test the difference
Ask the same question of a panel calibrated on US public data, and read the distribution rather than the answer. The free plan runs five surveys a month, up to 25 personas each.
Related: how the US panel is built, what we measure against human data, and the comparison with a fielded US panel.
ChatGPT is a trademark of OpenAI, cited here as the most common example of a conversational assistant. This page is neither published nor endorsed by OpenAI. Facts about ChatGPT were noted on 10 November 2026 from its public pages and may have changed since.