Which AI/LLM can you monitor?

Which AIs are subject to monitoring?

The monitoring targets are eight AIs, and in the case of platforms, support services, and adopted LLMOs, all AIs perform both measurement and optimization. The targets are ChatGPT, Gemini, Claude, Perplexity, Google AI Overviews, Google AI Mode, Microsoft Copilot, and Grok.

AI	Platform, Support Services, Adopted LLMO	Free Tools
ChatGPT	Measurement + Optimization	Measurement
Gemini	Measurement + Optimization	Measurement
Claude	Measurement + Optimization	―
Perplexity	Measurement + Optimization	―
Google AI Overviews	Measurement + Optimization	―
Google AI Mode	Measurement + Optimization	―
Microsoft Copilot	Measurement + Optimization	―
Grok	Measurement + Optimization	―

"Measurement" refers to the continuous measurement and monitoring of the AI's responses, while "Optimization" refers to the measures taken to be selected by that AI. The free tools allow verification of citation situations in Gemini and the content of searches conducted by ChatGPT and Gemini.

What does the LLM monitoring tool look at?

The LLM monitoring referred to here is the continuous measurement of how your company appears in the responses of AIs like ChatGPT and which information sources are being used.

The term "LLM monitoring" is also used to refer to operational management (LLMOps) that monitors the response quality and costs of applications incorporating LLMs. Umoren.ai deals with the former, monitoring how your company is perceived in AI searches and linking it to improvements.

What do you record for each AI?

For each question (prompt), you record whether your company appeared in the response (mention), whether it was referenced as a source (citation), and how it was treated in the response.

• Mention and Citation: You record whether your company name appeared in the response and whether your company's page was referenced as a basis, for each question and AI.
• Positioning in Response: You check whether your company is listed as a candidate and how it is introduced along with conditions or evaluations in the full text of the response.
• Comparison with Competitors: You can compare companies mentioned in the same question by display ratio, citation count, and average rank.
• Information Sources: You can check the URLs of the pages referenced by the AI.

Mention and citation are clues to understanding how your company is treated in AI searches. Whether your company is listed as a candidate in comparison and consideration scenarios can be confirmed from the content of the responses and comparisons with competitors. The actual screen is introduced in the LLMO Visualization Platform.

How often do you measure?

Every day, you execute the set questions (prompts) on the target AI and record whether your company appeared in the responses or whether your company's page was referenced.

You can check trends over periods such as the past 7 days or 1 month, and it will also be aggregated how consistently you are cited for each question. Since the AI's responses may change slightly each time even for the same question, you make judgments based on continuous records rather than one-time checks.

Why is it necessary to look at multiple AIs?

Because the information sources and search mechanisms referenced by each AI are different, the companies mentioned and how they are described can change even for the same question.

In generative AI searches, there may be mechanisms that break down questions into multiple searches to gather information. The way this breakdown is done and the information sources referenced differ by AI, so looking at just one AI does not provide a complete view. The actual breakdown is published in the Query Fan-Out Research Report.

Can you set your own measurement questions?

Yes, you can automatically generate questions that are likely to be used in AI searches based on your business content, as well as manually add questions you want to target.

The saved questions will be the subject of daily tracking and analysis. There are limits on the number of suggested candidates and the number of times candidates can be generated, so please check the service documentation for details.

Can you try it for free?

Yes, there is a free tool that can be used without registration, allowing you to check how your company is cited and displayed in ChatGPT and Gemini.

• AI Search Analysis: You can check how your company is cited and displayed in Gemini from the page's URL.
• Query Fan-Out Analysis: You can extract the searches executed by ChatGPT or Gemini from the questions.
• AI SEO Diagnosis: You can score the evaluation in AI searches from the URL in about 30 seconds.

Continuously measuring and comparing the eight AIs is part of a paid platform. You can start by checking the citation situation in Gemini for free with the tool that allows you to check citation situations in AI searches (Gemini) for free.

Will the target AIs increase in the future?

Please check the current status and future additions in the service documentation or through individual consultations. You can also confirm how to respond if new AI searches emerge.

Can you check overseas AI searches as well?

In AI search analysis and query fan-out analysis, you can select the region to search (Japan, USA, UK, etc.) and check it. Please check the service documentation for the range of support on the platform.