AI search, SEO & online discovery
Practical coverage from Dubai

Saved
Measurement

How Many Questions Belong in an AI Visibility Test?

Summarize with AI
Analysing business data; illustrative photograph.
Photo by https://kaboompics.com/ / PexelsIllustrative photography
In this article 6 sections

Start with enough questions to cover the customer decisions you want to investigate, then reserve runs to see whether the answers repeat. A Dubai service business testing discovery, service fit, branch selection and booking can use two distinct questions for each task as a first planning exercise. That gives eight questions. Running each on two surfaces gives sixteen first observations, before repeats, retries or a second language. The eight-question design below illustrates how to make that allocation. The right size depends on the decision and the time needed to review the evidence.

Keep the question count separate from the run count. A new question broadens what you can learn about customer intent. A repeat tests what changes under a defined condition. If you spend the entire budget on either, you lose the other kind of evidence.

Define what the test needs to cover

Write down the decision you expect to make after reviewing the answers. A branch owner might want to know whether customers get the correct location and booking route. A content team might want to see which needs produce an answer that omits the service or cites a weak source. A monitoring team might want a stable panel to revisit every month. Those jobs require different question mixes.

Choose customer tasks before drafting the exact questions. A branded opening-hours question and an unbranded request for a nearby provider can both matter, but they measure different things. Separate discovery, eligibility, destination and action. Within each task, decide whether geography, language, service constraints or timing changes the expected answer enough to merit a distinct question.

Use evidence from the business where available. Common enquiry themes, search terms and branch questions can inform the starting panel. Keep any gaps visible if those sources are unavailable. If you choose questions by relevance rather than by random sampling from a known population, report them as a deliberate panel. Its results describe the selected questions and observed conditions. A bigger panel chosen the same way still needs that scope label.

The previously verified Google Maps announcement below illustrates how one product can address several user tasks, including place and route choices. That can prompt a team to separate location questions from action questions. A feature announcement supplies product context. It gives no reason to assume the surface will answer every planned question or to treat different products as interchangeable.

Count exact questions, conditions and attempts separately

Assign one ID to each exact question. If you test an Arabic version alongside English, keep a shared intent ID and distinct wording IDs. If a test changes one qualifier, such as whether a location is accessible on the same day, record it as a variant with its own wording. This lets a reader see whether a run repeated the exact question or answered a related one.

Define a planned slot as one exact wording under one combination of conditions. Conditions can include the surface, market, language settings, session setup and capture period. Two products are two conditions even when they receive the same wording. Keep the actual output language in the run record. A difference in prompt language, interface locale and account state can otherwise get buried in a label such as “Arabic results.”

An attempt is a submission against a slot. A preplanned repeat is a separate observation because it was scheduled to check variability. A retry after a capture failure belongs to the original slot under a documented retry rule. Keep both attempt records, and specify which result enters the comparison. The existing AI Visibility test-sheet guide covers the basic capture record. The sizing exercise here decides which slots to create before you fill it. AI Visibility test-sheet guide

Allocate some runs to breadth and some to repeats

Breadth helps you find distinct failure modes. A branch question may reveal confusion about addresses while a quote question reveals that the answer sends readers to the wrong contact route. Repeating only one question would leave those differences invisible. Repetition helps when you need to know whether a particular observation persists under the conditions that matter.

Start with the minimum coverage needed for the decision, then choose repeat candidates before seeing the results. One question from each task group gives you repeat evidence across the panel. If the team cares most about a single high-impact task, it can concentrate additional repeats there and describe the rest as exploratory. Keep planned repeats separate from follow-ups triggered by an unexpected answer.

Use comparable conditions for a repeat. If you want to know how the result changes in a fresh session, open a fresh session both times and document what “fresh” means in the workflow. If the question is about a second asking in the same conversation, keep the first answer as part of that context. The two repeat designs estimate different kinds of variation. NIST's guidance on blocking formal experiments illustrates a general reason to organize comparisons around important conditions. Holding those conditions steady in an audit helps interpretation without turning a practical panel into a controlled experiment. NIST guidance on randomized block designs

Published Lunasol reporting illustrates why some repetition deserves a place in the budget. On one date in Google AI Mode, across four Dubai sectors, the second asking retained 24 of the 31 first-run website placements and the third retained 23 of 31. The totals were counted within sectors and added together. Those observations show movement in that recorded source list. They cannot supply the expected repeat rate for another task or surface. AI Visibility monthly reporting guide

Build an eight-question panel on paper

Consider a fictitious Dubai leather repair business. The team wants to find out whether prospective customers can discover a relevant provider, understand service fit, find an appropriate branch and reach a quote route. It has enough review time to test two surfaces in English, with one designated repeat per task on each surface. The questions below are a constructed planning example and were not submitted to the products.

Task Question IDs and exact wording Planned repeat
Discovery Q1 “Where can I get a leather handbag repaired in Dubai?” Q2 “Which Dubai repair shops handle damaged leather bag straps?” Q1
Service fit Q3 “Does Harbor Leather Repair in Dubai replace handbag straps?” Q4 “Can Harbor Leather Repair assess peeling leather on a handbag?” Q3
Branch choice Q5 “Which Harbor Leather Repair location should I use for a handbag repair in Dubai Marina?” Q6 “Where can I drop off a leather handbag for Harbor Leather Repair in Deira?” Q5
Quote route Q7 “How do I request a handbag repair quote from Harbor Leather Repair?” Q8 “Does Harbor Leather Repair let me send photos before bringing in my bag?” Q7

The team would verify the real business details before applying this design to an actual brand. A branch question can deliberately test whether a surface invents a location, but the reviewer needs an authoritative list of locations to score it. The same is true for service boundaries and contact routes. Freeze the expected evidence and the scoring labels before collecting the main panel.

The first pass requires eight wordings times two surfaces, or sixteen runs. The four designated repeats on both surfaces add eight runs. The panel therefore contains eight distinct questions and twenty-four planned observations. If the team reserves four attempts for capture failures, the attempt budget rises to twenty-eight. The reserve does not automatically produce four extra scored observations.

Estimate time using the work your reviewers actually do. Suppose each attempt, including capture and first-pass coding, averages six minutes for this planning exercise. Twenty-eight attempts would use 168 minutes at that rate. A further forty-two minutes for reconciliation, disputed calls and reporting gives a budget of 210 minutes, or three and a half hours. A pilot can replace that assumption with measured handling time. If citation and branch verification regularly take longer, reduce the panel or increase the review budget before committing to a recurring schedule.

Make the cuts that preserve the intended decision

If the budget is smaller, reduce conditions before removing essential tasks. A local team could start with one product, keep the four customer tasks and retain a small repeat block. If the decision is specifically about two products, keep both products and reduce the number of near-duplicate variants. Write down which comparison was preserved and which customer tasks or conditions remain untested.

If Arabic and English coverage is central to the decision, treat the language comparison as matched pairs rather than appending whichever translations fit at the end. Translating the eight-question panel and applying the same two-surface repeat design doubles the planned observations from twenty-four to forty-eight. It also adds language review time and may require a larger retry reserve. With a fixed budget, a smaller paired panel can be more useful than a large English panel and a few unmatched Arabic questions.

Choose the extra questions according to what you learned in the pilot. If two draft questions lead to the same task and require the same evidence, one may be enough for the baseline. If a branch-specific prompt requires different verification from a citywide question, it may warrant its own slot. Keep an exploratory question list outside the recurring panel so newly discovered gaps can be tested without making the baseline drift.

If the team wants a formal precision target or a comparison that generalizes beyond the chosen panel, the design needs a defined population, a sampling or assignment plan, and assumptions about dependence and expected variability. Repeated runs are often related through wording and conditions. Doubling the run count does not necessarily double the information about distinct customer tasks. Use a fixed business panel for the operational decisions it can support and design broader inference separately.

Check the workload at the level of each condition. If one surface regularly produces long source panels, an average handling time can understate the review it needs. Record capture time and source-check time separately during the pilot. Use those observations to budget the next cycle, and reserve time for review of disputed cases. A panel that fits the collection window but cannot be verified by the reporting deadline still exceeds the practical budget.

Keep the capture order in the protocol when time could matter. Running all questions on one product before switching can make a product comparison harder to interpret if the collection window also changes. Alternate or group comparable slots where the workflow allows and record the actual timing.

Close the loop after the pilot

After the first cycle, reconcile planned, attempted, captured and scorable counts by task and surface. Ask where work took longer than expected and where a missed capture made a planned comparison unusable. Compare repeats with the original observations under the same scoring rule. If the largest uncertainty comes from destination verification, devote time there before adding more prompts.

Review every question against the decision. Keep questions that detect distinct customer problems or give stable coverage of a priority task. Retire ones that are redundant or no longer relevant, with a dated reason. Preserve an overlap set when revising a recurring panel. A new panel can improve coverage while making a direct comparison with the earlier headline harder.

Set a clear stopping rule for an exploratory run. A team might stop after each task has a captured example on both intended surfaces and one preselected repeat where available. A failed branch capture can trigger a documented retry within the attempt budget. An unexpected answer can go on a follow-up list rather than forcing the team to extend every slot until it gets a preferred result.

For the report, name the question and observation counts in the opening line. In this design the team would say that it planned eight exact questions, twenty-four observations and up to four retry attempts across two surfaces. When the work is done, replace the plan counts with the corresponding actual counts and retain any gaps. A reader can then see how much evidence supports the result and which part of the plan was never observed.

AI visibility question budget record

Save the decision, task inventory, exact wordings, condition matrix, repeat selection, retry limit and expected handling time. Keep the verification source and scoring rule with each task. At cutoff, report the planned and completed counts and document any change to the panel. That record makes the next cycle reproducible and helps the team decide whether to add breadth, repeat important questions or spend more time verifying what it already captured.

AI Visibility is published by Lunasol, which offers GEO and AEO services. The question panel and time budget are constructed. The published repeat figures are identified separately. About AI Visibility

You’ve reached the end of this article. Read as plain text

Search the blog

Summarize with AI

Choose your assistant. Copy the article and summary request, then paste it into a new chat.

The full article is included, so your assistant can read it without opening this website. Nothing is submitted automatically.

View or copy the summary requestOpen assistant

Summaries are generated by your chosen service and may miss details. Its account requirements and terms apply.