AI agents find defects by operating your product, synthetic users simulate opinions, heuristic evaluation applies expert rules, and real users are the ground truth. Every vendor in this market (us included, we know) will happily sell you their quadrant as the whole map. So here is the four-way comparison with the “what it misses” column left in.
Four methods, one honest table
Definitions first, because the marketing blurs them on purpose. An AI agent operates your product: it opens a browser, signs up, fills forms, and records what happened. A synthetic user never touches the product: it is an LLM persona answering interview questions in character. Real users are humans attempting tasks while someone watches, and a heuristic evaluation is an expert (possibly you) grading the interface against published rules.
| Method | What it actually is | What it catches | What it misses | Cost | Time | Use it when |
|---|---|---|---|---|---|---|
| AI agent audit | A browser agent walks your live flows and checks each step against named rules (Nielsen’s heuristics, WCAG) | Broken flows, silent form failures, dead ends, rule violations with replayable evidence and screenshots | Motivation, desirability, trust, whether your copy persuades. Anything that requires an opinion | From $49 one-time (ClapBack); Loop11 agents from $179/mo | Under an hour | Pre-launch defect sweep, then before every release as a regression net |
| Synthetic users | LLM personas answer interview or survey questions in character; nobody touches the product | Fast hypotheses: likely objections, draft personas, holes in your interview guide | Real variance and magnitude. Answers spread consistently narrower than real humans’ (NN/g) | $2 to $60 per interview (vendor’s own pricing) | Minutes | Piloting a study design before spending real participants on it |
| Real users | Actual humans attempt real tasks, moderated or unmoderated, while you watch or record | Actual behavior, the why behind drop-off, and the surprises nobody predicted | Little in principle. In practice: whatever your 5 recruits and your chosen tasks never hit | The priciest: recruiting, incentives, moderation, synthesis | Days to weeks | Any decision that bets real money on how people behave |
| Heuristic evaluation | An expert walks the flows against Nielsen’s 10 heuristics and logs violations | Known violation patterns; ~35% of problems for a single evaluator, more with 3 to 5 | What fresh users would trip on. You grading your own homework has blind spots | Free if you run it yourself; hired out, real money | An afternoon | Day one, before you pay anyone for anything |
Now the receipts behind each cell, starting with the category that gets marketed hardest.
AI agents: the audit that clicks things
An agent-based audit works your product the way a stranger would: signup, forms, checkout, the lot, in a real browser. That is ClapBack’s category (paste a URL, get tickets, from $49), and we did not invent it alone. Loop11 sells AI browser agents that “navigate, interact with, and evaluate websites” alongside its human panels, and academia arrived at the same idea from two directions.
Amazon-affiliated researchers built UXAgent (CHI 2025), which set 60 LLM personas loose on a mock shopping site; the authors frame it as a way to pilot study designs before spending real participants, not as a replacement for them. Columbia’s WebProber agent probed 120 academic personal websites and surfaced 29 usability issues, “many of which were missed by traditional tools.” And the 2026 benchmark UXBench forces models to collect “interaction evidence before reporting” and finds they “differ meaningfully in report actionability.” Translation: agents work, unevenly by model, and the interaction is the entire point.
The blind spot this fixes has been measured: Baymard’s audit study found screenshot-only AI caught barely 1 in 7 of the issues present on a live page, and the full autopsy is in our ChatGPT prompts piece. Problems that exist only when something gets clicked need something that clicks.
What agents cannot do is feel. An agent can prove your form rejects valid emails; it cannot tell you whether anyone wants the product, trusts the pricing page, or winces at your tone. For the full anatomy of what an agent run covers, see what an AI UX audit actually is.
Synthetic users: interviews with software
A synthetic user is an LLM persona built to answer research questions in character. “Maria, 34, ops manager” will gladly tell you what she thinks of your onboarding, despite never having experienced it, or anything else. Synthetic Users sells these interviews at $2 to $60 each and claims its AI feedback “lined up with human feedback over 95% of the time”; Uxia pitches AI-powered user testing with “no recruiting needed.” To its credit, Synthetic Users’ own site calls the product “a discovery co-pilot, not a replacement for real research.”
NN/g examined the practice and published the bluntest line in the debate: “UX without real-user research isn’t UX.” Their guidance: treat every synthetic finding as an unvalidated hypothesis. Useful for drafting interview guides and proto-personas, dangerous as the basis for a design or business decision.
A synthetic user is a focus group that agrees with you, at scale, for $2 a head.
In NN/g’s review of three simulation studies, 605 synthetic survey respondents scored purchase likelihood at 1.58 versus 1.66 for 605 matched humans, with standard deviation consistently lower in the synthetic data. Raluca Budiu’s summary: synthetic users capture general trends but fail “at capturing the magnitude of the effects or the variability.” Lower variance means fewer surprises, and surprises are the entire product of user research.
Real users: the only source of truth
Nothing about real behavior can be established without real behavior, which is why moderated and unmoderated usability testing stays the gold standard. The famous efficiency claim comes from Nielsen’s 5-user article: with an average test user surfacing about 31% of problems, the formula N(1-(1-L)^n) says 5 users uncover roughly 85%, so run “as many small tests as you can afford.”
Quote that 85% honestly, though. It is a model, not a measurement of your product: the 31% is an average across old projects, and the math counts only problems your chosen tasks could surface at all. Nielsen’s own conclusion was to iterate, arguing 3 studies with 5 users each beat one monster study with 15.
The catch is the price of humans. Recruiting people who resemble your customers, paying incentives, moderating sessions, and synthesizing notes runs to weeks and real money, which is exactly why small teams skip it. We broke down the money side in what a UX audit costs.
Heuristic evaluation: the baseline everyone forgets
Before paying anyone for opinions, human or artificial, there is a method that costs an afternoon: walk your core flows against Nielsen’s 10 heuristics and log every violation. Nielsen’s data says a single evaluator finds about 35% of usability problems, which is why NN/g recommends 3 to 5 independent evaluators.
Two things make it the correct day-one move. It is free, and it stacks: Nielsen found heuristic evaluation and user testing “each finds usability problems overlooked by the other method,” so an early heuristic pass never wastes a later user test. We ran the whole method, all 10 heuristics with severities and fixes, in our worked heuristic evaluation example. Want an LLM riding shotgun on that pass? Our ChatGPT UX audit prompts are the $0 version.
Simulated opinions vs executed interactions
The label “AI UX research” hides the distinction that matters: some AI produces opinions, and some AI performs actions. A synthetic user’s output is plausible text. You cannot replay it, falsify it, or attach a screenshot to it; the only way to check it is to run the real research it was supposed to replace.
An agent’s output is a trace. It typed this email, the form silently rejected it, here is the screenshot and the rule it violates. Disagreeing with a trace takes a minute of replaying, not a research budget. UXBench turned that principle into a benchmark by requiring “interaction evidence before reporting,” because a critique without executed interactions cannot be checked.
The failure modes follow. Agents fail by omission: whatever flow the agent never walked, it cannot report on, and it will never tell you why users churn. Synthetic users fail by confident fiction: fluent answers to questions nobody asked a human. One is an evidence machine with gaps, the other an opinion machine with polish. Budget accordingly.
Which method should you use right now?
Match the method to your stage: heuristic and agent passes pre-launch, 5 real users on the money flow pre-PMF, continuous research at scale. Answer 3 questions and the picker below does the matching for you.
The reasoning, stage by stage, in ticket-sized pieces:
- Pre-launch, solo founder. Run your own heuristic afternoon, then an agent audit before the launch post goes live. Total spend: under $50 and a Saturday. There are no users to recruit yet, so “real user testing” is not on the menu; defect-hunting is.
- Pre-PMF, early revenue. Keep the agent as a regression net on every release, and put 5 real users on the signup-to-payment flow. Per Nielsen’s model that surfaces most findable problems in the one flow that pays rent, for the price of 5 incentives.
- Scaling. Continuous rounds of real research, agent checks before each release, and synthetic personas only for piloting study designs, which is the exact use UXAgent’s own authors endorse.
That ticket is self-serve: the agent half takes a URL and about the length of a coffee. ClapBack runs it, and your first 5 real users will thank you for the dead buttons they never met.