Skip to content
neovekContact us

AI Agent Clarifying Questions: When to Ask Before Acting

Share

AI agent clarifying questions are the questions an agent puts to a person before it acts, to settle what that person actually wants. They look like friction, and product teams are trained to remove friction. This article weighs when asking pays off and when it costs more than it returns, using an experiment Gamma described in October 2026, a 2026 study of coding agents given vague tasks, user studies from web search and a security benchmark. It ends with rules for when an agent should ask, how many questions it should ask, and whose answers it should accept.

The experiment comes from Gamma, the AI presentation company that announced in November 2025 that it had passed $100 million in annual recurring revenue with 70 million users (Gamma press release, 2025-11-10). Its co-founder and chief product officer, Jon Noronha, described it in an interview with Greg Isenberg published on 2026-10-28. Gamma’s agent normally opens a request with a short form of questions, then shows an outline for review before it builds anything. The team tested a fast mode that skipped both stops, on the theory that people would reach a finished deck sooner and like it more. Noronha admitted he hoped the theory would hold:

“I was really rooting for this test. I wanted to see it win, but it did not win for us actually.” - Jon Noronha

Gamma’s fast mode lost to the flow that asks first

According to Noronha, the slower path produced higher satisfaction ratings, users who did more with what they made, and a higher rate of conversion to a paid plan. He gave no sample size, effect size or test duration, so this is one company’s account rather than a published result. Two details still make it worth taking seriously. Conversion to paid is the signal he trusts most, because nobody pays to be polite. And Gamma did not throw the fast mode away: it shipped a version for brand-new users on mobile, where typing answers on a small screen is a real cost.

That second detail is the useful one. The test did not show that questions are good in general. It showed that, for people making a presentation at a desk, two particular stops were worth their time, and that the trade changes with the device. The rest of this article looks for the conditions behind that result.

Clarifying questions recover most of what a vague task loses

The cleanest measurement comes from software engineering, where a task either passes its tests or does not. Nicholas Edwards and Sebastian Schuster worked from the 500 tasks of SWE-bench Verified, using versions of each issue rewritten to drop file paths, stack traces and code snippets. Coding agents then tackled those vague tasks, some allowed to question a simulated user and some not (Edwards and Schuster, “Ask or Assume?”, arXiv 2603.26233, EMNLP 2026).

With Claude Sonnet 4.5, an agent that could not ask solved 54.8% of the vague tasks. Letting the same agent ask raised that to 61.2%. Splitting the job between one agent that judges whether the task is underspecified and another that writes the code raised it to 69.4%, against 70.8% when the agent received the full original issue. Asking recovered almost all of the ground that the missing details had cost.

Horizontal bar chart of Claude Sonnet 4.5 resolve rates on 500 SWE-bench Verified tasks: 54.8% on vague tasks without questions, 61.2% when the agent asks, 69.4% with a separate ambiguity check, and 70.8% with the full original issue Letting the agent ask closed most of the gap between a vague task and a complete one. Source: Edwards and Schuster, arXiv 2603.26233.

The authors flag the main weakness themselves: their simulated user, played by GPT-5.1, is more cooperative than real people are. Actual users answer late, answer half the question, or do not know the answer. The figures are an upper bound on what asking can recover, not a forecast for a live product.

More questions did not buy more accuracy

The same study holds a warning. With the open-weight Kimi K2.6 in place of Claude, the split design also reached 69.4%, but it asked 8.71 questions per task on average, against 3.06 for Claude. Same score, almost three times the interruptions. The authors attribute the gap to weaker calibration, meaning the model was worse at judging when it truly lacked information.

Web search learned the same lesson earlier, with real people. Jie Zou and colleagues showed users clarifying questions of graded quality during search tasks. Good questions improved both task performance and satisfaction. Poor and middling ones made both worse, to the point that the authors suggest letting users carry on without any question in those cases (Zou et al., ACM Transactions on Information Systems 41(1), 2023). A question is never neutral. A vague or irrelevant one tells the user that the system has not understood, and it spends their effort for nothing.

A good question has a recognizable shape. Gamma tests its agent on one: asked for a deck about tigers, it should find out whether the audience is children or adults. The question has two answers, each leads to a visibly different deck, and the user knows the answer instantly. Anthropic built a similar constraint into its Agent SDK. As of October 2026, its AskUserQuestion tool takes one to four questions per call, each with two to four options to pick from (Anthropic, Handle approvals and user input). Multiple choice makes the agent do the thinking and leaves the user a click.

Questions belong before the work, not in the middle

Timing matters as much as count. ChatGPT’s deep research runs a clarification step before any searching starts: an intermediate model works out the user’s intent and gathers missing context, and the request is then rewritten into a detailed prompt. OpenAI’s guide notes that the API version skips that step and leaves developers to build their own (OpenAI, Deep research guide). Anthropic describes plan mode the same way: Claude explores the code and asks its questions before proposing a plan, not while it edits files. Gamma’s form appears within the first seconds, before the outline is drafted.

The reason is cost. A question at the start costs the user a few seconds and the agent nothing. The same question twenty minutes into a run puts the finished work at risk, and it may reach a user who has moved on to something else. The coding study shows a version of this. Its single agent asked 43.9% of its questions in the late stage of its run, while the split design asked mostly in the early and middle stages and scored higher. The study does not prove that timing caused the gap, but the pattern matches what the products do.

Reviewing an outline gives people a stake in the result

Gamma’s result is not only about accuracy. Noronha’s point was that the people who answered the questions and reviewed the outline felt more invested in the deck and did more with it. Behavioral research has a name for something close to this. In four studies where participants assembled IKEA boxes, folded origami or built Lego sets, Michael Norton, Daniel Mochon and Dan Ariely found that people valued what they had made themselves more highly, and rated their amateur work close to that of experts (Norton, Mochon and Ariely, Journal of Consumer Psychology 22(3), 2012).

Applying that to an agent is my own reading, not something either source tested. A review step lets a person shape the result before it exists, which makes the result partly theirs. The paper adds a condition that matters for product design: the effect disappeared when participants failed to finish or when their creation was taken apart. A question flow that ends in a deck the user is proud of builds attachment. One that stalls, or produces something the user throws away, builds nothing.

Every answer to an agent is also an attack surface

Clarification has a cost that user studies miss. Researchers at Scale AI built ASPI, a benchmark of 728 task-and-attack scenarios, and ran ten frontier models on ambiguous tasks with and without the option to seek clarification. Seeking clarification made prompt injection far more likely to succeed. For o3, attack success rose from 1.8% to 34.0%; for Gemini 3 Flash, from 2.2% to 35.7% (Sehwag et al., Scale AI, 2026-05-17). The authors trace part of the effect to the clarification interface itself. My interpretation: an agent that has just asked for input is primed to act on whatever comes back as an answer.

Grouped bar chart of prompt injection success on the ASPI benchmark: o3 rises from 1.8% to 34.0% and Gemini 3 Flash from 2.2% to 35.7% when the agent seeks clarification On ambiguous tasks, seeking clarification multiplied the success of injection attacks for both models. Source: Sehwag et al., Scale AI, 2026-05-17.

The practical consequence is that an answer is only as trustworthy as its sender. In a private chat with one user, that is easy to guarantee. In a shared workspace, an email thread or any flow where the reply can come from a document, a web page or another person, the clarification step becomes a way to hand the agent new instructions. It is the same problem covered in AI agent permissions for one Slack agent shared by a team: the control has to sit outside the model, in who is allowed to answer.

AI agent clarifying questions: six rules for when to ask

The sources point to a short set of rules. They are my synthesis, not an industry standard.

  • Ask when readings diverge. A request with several valid interpretations that lead to visibly different results deserves a question. One with a sensible default does not.
  • Ask when a wrong guess is expensive. Long runs, payments, deletions and messages to customers justify a stop. A draft the user will edit anyway rarely does.
  • Ask early, once, with choices. Batch the questions at the start, offer options rather than blank fields, and keep the count low enough that answering takes seconds.
  • Measure the questions themselves. A question whose answer never changes the output is friction with no return. Gamma’s choice of signal applies here: watch payments, edits and return visits, not only survey scores.
  • Offer a fast path where answering is costly. Mobile users and repeat users with known preferences are the obvious cases, and Gamma’s mobile fast mode is the precedent.
  • Accept answers only from the requester. Anything else turns the clarification channel into an injection route.

Gamma’s test is a single data point, but it fits what the research shows. An agent that guesses saves the user a few seconds and risks the whole job. One that asks the right question, at the start, of the right person, is doing part of what an AI agent harness is for: getting the context right before the model does the work.

Share

Get new posts by email

At most one email a week, on Monday, with Neovek's new posts. Unsubscribe in one click. Privacy