【Major Update】ChatGPT Query Fan-Out Analysis Tool Evolves | Visualization from Pre-Search Assumptions to Brand Diagnosis

When your company is not "recommended" by ChatGPT, what could be the cause? Was it not picked up in the search, or was the page retrieved but not adopted? Or, was it cited as a source, but the company name was not mentioned in the final answer?

We have made a major update to umoren.ai's "ChatGPT Query Fan-out Analysis Tool." In addition to the previous "extraction of search queries executed by ChatGPT," you can now analyze expected answers before the search, research plans, chunk evaluations of retrieved pages, brand funnels, and final answers all on one screen.

 

Figure 1｜New analysis screen. In addition to questions and search regions, you can specify user profiles and your brand as needed.

From "Knowing Search Queries" to "Analyzing Answer Construction"

The previous tool focused on confirming the search terms and sources used by ChatGPT, as well as the final answers. However, just looking at the final answer does not help identify the "reason for not being cited." Therefore, we have expanded the analysis target to multiple stages of answer generation.

 

If you want to check the basic concept of Query Fan-out (QFO) first, please also refer to the article explaining "What is Query Fan-out?".

AI Search Answers Can Be Interpreted in 7 Stages

The overall picture of the analysis is "User and Question → Pre-Answer Before Search → Research Plan → Evidence Target → Retrieval and Selection → Citation and Final Answer → Brand Diagnosis." The results will be displayed sequentially on the screen as the analysis progresses, allowing you to start checking without waiting for all processing to complete.

Figure 2｜The tool visualizes 7 analysis steps.

New Feature ① Specify "Who is Asking" Even for the Same Question

In the new "Advanced Settings," you can input the job title, industry, company size, and purpose of the user asking the question. For example, even the same question "What is the recommended CRM?" will have different important conditions between a sales manager of a company with 50 employees and an information systems officer of a large corporation. The tool passes this background information as a separate context rather than adding it to the question text.

Additionally, by entering your brand name and domain, it diagnoses whether that brand appeared at each stage of the search. The relevant parts will be highlighted on the screen, so there is no need to search for the appearance of the brand name.

New Feature ② Visualize "Pre-Answer Before Search" and Verification Targets

It reproduces the state equivalent to before ChatGPT's search through a simulation using another model. It lists company names, product names, and concepts that appear in the pre-answers as entities and extracts "claims that require verification," such as prices, comparisons, and achievements, with numbered labels like C1, C2.

What is important is whether your brand was included in the expected candidates at the pre-answer stage. If it is not included in the candidates, it needs to be discovered and evaluated in subsequent searches. However, this pre-answer is not directly obtained from ChatGPT's unpublished internal thoughts but is an estimate from a reproduction model.

Figure 3｜Pre-answer before search. You can summarize the candidate companies, entities, and claims for verification (C1~).

New Feature ③ Understand "Why Search" in Research Plans

The new feature in the research plan classifies search candidates into three types: "Broad Exploration," "Claim Verification," and "Supplementary Information." For each search query, it displays which claims are being verified, what is being searched for, and what kind of evidence is ideal.

This "ideal form of evidence" is the Evidence Target. It is a hypothetical text based on the concept of HyDE (Hypothetical Document Embeddings) and can also be used as a content brief when considering the structure of information that is easy for AI to reference. Since the prices and figures included in the hypothetical text are not factual, they should not be directly copied into articles, and verification with official information or primary sources is necessary.

Furthermore, it matches the estimated search plan with the actual search queries executed and displays whether searches with similar meanings were conducted. This allows you to understand the degree of consistency between the plan and the actual measurements, enabling evaluation without confusing simulations with actual results.

Figure 4｜Research plan. Divided into broad exploration, claim verification, and supplementary information, it also shows the matching status with executed searches.

New Feature ④ Retrieve Queries Actually Searched by ChatGPT

The extraction of queries used in actual ChatGPT web searches, which has been a core function, is still available. If the search was conducted multiple times, it organizes the initial investigation and deep dives/targeted searches by rounds.

Additionally, new "unplanned" labels have been added to actual queries that were not included in the prior research plan. Unplanned searches can provide clues to discover unexpected comparison axes or brand names. If the search operator "site:" was used, its form can also be confirmed as is.

Figure 5｜Search queries actually used by ChatGPT. Supports round-by-round display and "unplanned" labels.

New Feature ⑤ Separate Evaluation of "Retrieved" and "Cited"

One of the particularly important updates is the evaluation of retrieved pages. Previously, there was a tendency to focus on pages cited in the final answer, but the pages retrieved as search results are not necessarily used as the basis for the final answer. The new screen displays "Retrieved," "Evaluated," and "Cited" separately.

The text of the retrieved pages is divided into small units (chunks), and each is compared with the Evidence Target to calculate relevance. The displayed relevance is an indicator calculated by umoren.ai and is not the unpublished ranking score used internally by ChatGPT. There may also be pages that cannot be evaluated due to their retrieval status.

In actual screen examples, you can check the breakdown such as "19 retrieved, 10 evaluated, 5 cited." You can investigate which search intent was closest, including pages that were not cited.

Figure 6｜Retrieved pages and evaluations. Distinguishes between cited, retrieved only (not adopted), and unreadable.

By expanding the rows, you can check the chunks with high relevance and the places where the brand was mentioned within the page. The key point is to investigate whether "the necessary evidence for that question is explicitly stated in the text" rather than simply adding keywords.

Figure 7｜Page details. You can check which parts were closest to the search intent, at the chunk level of the text.

If you want to know more about the relationship between chunks and citations, the article on "The Mechanism of Chunks and Citations" is also helpful.

New Feature ⑥ Diagnose Where Your Brand Was Excluded from Consideration with "Brand Funnel"

By entering your brand name and domain, you can list the five stages: "Pre-Answer → Search Query → Retrieved Page → Citation → Final Answer." You can check whether the brand was found at each stage and diagnose where the information was cut off. The feature includes mentions on third-party sites such as comparison articles and reviews, not just your own site.

In the attached analysis example, while umoren.ai appears in the retrieved pages and sources, the brand name is not included in the final answer. The diagnosis result of "cited but not appearing in the answer" indicates that merely obtaining citations does not necessarily lead to recommendations or mentions.

Figure 8｜Brand funnel. Diagnoses cases where it appears in "Retrieved Pages" and "Cited" but not in the "Final Answer."

Five Patterns of Diagnosis. Connect to Next Measures

 

New Feature ⑦ Compare the "Replacement of Candidate Companies" Between Pre-Search and Final Answer

Finally, compare the brands and products mentioned in the pre-answer before the search with the actual final answer. By organizing them as "Retained," "Deleted," and "Added," you can understand whether companies that were candidates before the search disappeared from the final answer or if new candidates were added through the search.

This change represents a comparison between the simulation of the pre-answer and the observed final answer. It does not mean that the model's secret internal opinions were directly read.

Figure 9｜Differences between pre-search candidates and final answers. You can check "Deleted" and "Added" in a list (top of the screen).

Figure 10｜You can also check the content of the final answer and interpret the reasons for recommendations including sources (bottom of the screen).

Reasons Why You Can't Improve by Just Looking at the "Final Answer"

Prompt tracking is essential for understanding the brand mention rate and publication status. On the other hand, just the result of "the company name did not appear" makes it difficult to determine whether the issue is with discovery, the quality of information, or the association with the brand.

With this tool, you can differentiate between whether your brand was included in the candidates before the search, whether the page was retrieved in the search, which text was evaluated after retrieval, and whether the company name remained even if it was cited.

Figure 11｜Four points that cannot be understood from just the final answer. Broken down into pre-search, retrieval, plan matching, and content design.

Practical Use for Web Managers and LLMO Managers

1. Identify Reasons Why Competitors Are Recommended While Your Company Is Not

First, input questions that customers might naturally ask without including your company name, and also register your brand. Check where your company appears in the pre-answers, actual searches, retrieved pages, citations, and final answers. The priority of measures will differ between cases where there are few mentions on third-party sites and cases where your company page was retrieved but not adopted.

2. Reverse Engineer Necessary Content from Evidence Targets

Check the form of information required by the search plan, such as "price range," "target companies," "comparison conditions," and "implementation achievements." If existing content lacks answers, it is easier to verify improvement areas by creating independent paragraphs that clearly state conditions, targets, figures, and sources rather than increasing abstract expressions.

To check which aspect of a specific URL answers the question, you can also use the AI Search Content Diagnosis. This is a way to find search intent through QFO analysis and inspect page-level deficiencies through content diagnosis.

3. Compare Search Differences by Persona and Region

Run the same question with different user profiles, such as "small business owner" and "marketing manager of a large corporation," and compare search terms, sources, and final answers. Differences in search regions, such as Japan and the United States, can also be observed. However, since the results of AI searches can fluctuate based on timing, environment, and context, it is important not to make definitive judgments based on a single analysis but rather to assess them alongside multiple observations and actual user behavior.

If you want to compare search queries on the Gemini side, please refer to the Gemini Query Fan-out Analysis Tool and the Usage Guide.

How Much Can You Use for Free? CSV Output is Also Supported

The tool can be started for free. Even for first-time unregistered users, the top parts of research plans, actual queries, evaluations of retrieved pages, and an overview of the brand funnel will be displayed. Detailed results can be checked by creating a free account or by entering company information to view the current results. The latter requires inputting a company email address or similar.

When you open all results, you can check the search plans, matching results with executed queries, evaluations of retrieved pages, brand diagnoses, and final answers. The CSV includes the types of searches, claims to be verified, reasons for searches, Evidence Targets, match rates with actual searches, and related retrieved pages and evaluations, which can be used for prioritizing measures and sharing within the company.

The estimated analysis time is about 30 to 60 seconds (processing time may vary depending on the situation).

Points to Note for Correctly Reading Analysis Results

 

This does not directly obtain ChatGPT's unpublished intermediate reasoning or internal scores. It is designed to confirm the relationship between estimates and observations by matching the estimated search plans with actual search queries. Additionally, not all retrieved pages can necessarily be loaded and scored; evaluations are made based on pages that can be loaded. High relevance does not guarantee future search rankings or citations.

For basic site design, which is a prerequisite for content improvement, you can also check "How to Create a Site that is Cited and Recommended by ChatGPT".

Frequently Asked Questions (FAQ)

Q. What is Query Fan-out?

It is the action of AI executing multiple related searches to gather the necessary information to answer a single question. The search queries may not necessarily match the exact phrases of the initial question input.

Q. Are pre-answers and research plans the actual internal data of ChatGPT?

No. It does not directly view unpublished internal reasoning but is an estimated result reproduced by a different model. On the other hand, the actual executed search queries and final answers are obtained from the targeted search execution.

Q. Can I use Evidence Targets (HyDE) directly in articles?

While it can serve as a reference for considering the structure of sentences and necessary information, the specific figures and achievements included in the hypothetical evidence text require verification. Please replace them with official data, case studies, and primary information before publishing.

Q. What if ChatGPT does not perform a web search?

In cases where it answers without conducting a web search, the actual search queries will not be retrieved. However, by looking at the simulated pre-answers and plans, you can consider how that question relates to your company.

Q. Is "Retrieved but Not Cited" a failure?

Not necessarily. Even if it was found in the search, it may be that evidence from another page was prioritized for the purpose of the answer. Check the specificity of the page text, the recency of information, and comparison conditions, and improve from the necessary points.

Figure 12｜You can also check FAQs regarding QFO, Evidence Targets, personas, and unexecuted searches within the tool page.

Conclusion: Transforming Information on "Reasons for Not Appearing" into Improvements

The ChatGPT Query Fan-out Analysis Tool has evolved from a tool for investigating search queries to an analysis tool for understanding the formation of AI search answers in stages. By viewing the pre-candidates, actual searches, evaluations of retrieved pages, occurrences of the brand, and differences in final answers together, the specific measures to take next become clearer.

First, input one question that customers are likely to ask, and try the ChatGPT Query Fan-out Analysis.

If you want to systematically advance AI search measures, you can also check umoren.ai's AI Search Strategy Consulting and implementation and verification examples. For inquiries about your company's search, citation, and recommendation status, you can also consult through free consultations.