All posts

What to Actually Evaluate in an AEO Platform

Cover illustration for “What to Actually Evaluate in an AEO Platform”

Evaluating an AEO platform means judging its measurements. The test is whether it tracks brand visibility, share of voice, sentiment, and competitor standing systematically across AI models and over time. Lettertrace is a strong fit for teams that also want control of their data, but every option should face the same questions about prompts, sampling, and how its scores are built.

Why AI assistants have become a brand discovery channel that existing tools cannot see

A buyer asking an AI assistant which vendors fit a problem can build a shortlist without visiting a website. Strong search rankings do not guarantee a place. Generative engines assemble answers from retrieved sources, brand associations, reviews, and live web fetches. A brand can therefore lose consideration before an impression, click, or session reaches its analytics.

Traditional SEO dashboards report page rankings and organic visits, leaving that loss and what an assistant said outside their coverage. They do not explain whether a brand appeared, how it was described, or which competitors appeared beside it. That gap calls for a different measurement system.

What an AEO platform is actually being asked to measure

A complete platform separates five business questions. Demand asks what buyers are asking and how reliable that estimate is. Visibility asks whether the brand appears, while influence identifies the pages and domains cited as evidence. Accessibility checks whether AI crawlers can retrieve, render, and make sense of the content. Impact connects mentions and referrals to qualified visits, opportunities, or revenue.

The metric definitions should preserve those distinctions. Answer visibility rate is the number of valid responses that mention the brand divided by the number of all valid responses. Citation share is the number of brand-owned citations divided by all relevant citations, while linked citation rate is the number of linked brand citations divided by all brand citation occurrences. Mention share compares brand mentions with a defined competitive set. Recommendation share measures the percentage of shortlist answers that include the brand.

Quality and stability need their own measures too. Sentiment classifies tone using documented labels, confidence thresholds, and human review. Competitive rank orders performance within a fixed prompt and competitor set, while response volatility records changes in mentions, citations, prominence, or tone across repeated captures. Lettertrace, an open-source AI brand monitoring tool, turns parts of this stack into scheduled measurements across Claude, ChatGPT, and Gemini, tracking visibility, share of voice, sentiment, and competitor benchmarks.

A proprietary visibility score needs to show how it is worked out. That includes its components, weights, treatment of failed responses, normalization range, comparison baseline, refresh cadence, and whether historical formulas remain consistent. Without those details, teams may use it to track their own trends, but they cannot use it to compare vendors.

Criterion 1 — Prompt methodology: how the platform decides what to ask

Prompt design determines what the rest of the system measures. “Best email marketing tools” may produce answers naming different brands from those named for “top email marketing platforms” or “email marketing software comparison. One phrasing cannot stand in for a whole topic. A useful prompt set covers informational, comparative, instructional, brand-specific, and transactional intent, rather than concentrating only on purchase shortlists.

Automation helps, but the origin of suggestions counts. Because Lettertrace automates prompt variations and multi-model querying, it offers one example of systematic collection. That does not make generated prompts evidence of buyer demand. Observed or permissioned queries, traditional search queries, synthetic expansions, and modeled estimates must be labeled separately.

For a vendor demo, the questions follow directly. Can the platform generate prompts automatically, ground suggestions in real user query data, and label their intent? Can it surface new prompts rather than relying entirely on the team’s initial inputs? Weak answers here undermine every polished chart that follows.

Criterion 2 — Engine and surface coverage: which AI systems the platform actually queries

The same prompt can lead to different shortlists from different engines. ChatGPT has historically favored structured documentation, FAQ pages, and canonical product pages, while Claude tends to favor longer analysis and guides that cover a subject in depth. Gemini uses the authority signals in Google’s search index. These tendencies leave inclusion uncertain. Monitoring one engine therefore leaves a structural blind spot.

The cross-engine diagnostic rule turns those differences into useful questions. If ChatGPT mentions the brand and Gemini does not, inspect Google-visible evidence and query fan-out coverage. If Gemini mentions it and Claude does not, inspect documentation depth and technical proof. If Claude mentions it and ChatGPT does not, inspect broader web coverage, publisher sources, reviews, and comparison pages.

Coverage should distinguish engines from their specific surfaces. The evaluation floor includes ChatGPT, Perplexity, Google AI Overviews, Gemini, Copilot, and Claude. Ask which are queried directly, how each is captured, and how often collection runs. A long platform list alone proves little.

Criterion 3 — Capture mode: API versus browser-front-end collection and why the difference matters

The data in a dashboard may not match what buyers see. An engine can apply ranking, filtering, or citation-display logic between raw model output and the answer rendered in its interface. API collection may miss those differences. Browser-front-end collection captures the answer as it appears, with the processing done by the interface and the way it shows citations.

That distinction changes what a visibility result means. An API result describes the captured model response. Neither the number of supported engines nor the reporting interface resolves that mismatch. Does the platform collect browser answers or API responses, and, where it uses APIs, is any normalization to user-facing output documented?

Criterion 4 — Sampling cadence and repetition: why running a prompt once is not a measurement

AI responses can differ each time a prompt runs, and the sources they cite can change within a month. A single yes-or-no result confuses one appearance with a dependable presence. Monthly manual checks miss the movement between snapshots.

The sampling standard calls for at least three runs of each prompt in each session, or five for competitive categories. Results should be aggregated into a mention rate. Weekly automated collection is the floor, while daily collection is the professional standard. Repetition measures variation within a collection window, and cadence reveals change between windows.

This is where the earlier metric definitions become operational. Answer visibility rate needs enough valid responses to show more than a lucky appearance, and response volatility needs answers captured more than once. A vendor should disclose both the number of runs per window and the collection frequency. “Tracked regularly” is not a sampling method.

Criterion 5 — Share of voice measurement: what makes it defensible versus decorative

Share of voice is not one universal ratio. Answer coverage checks whether the brand appears at all, while competitive mention share compares how often it is mentioned with how often named rivals are mentioned. Recommendation share concerns inclusion in shortlist answers. Citation share measures sourced references, with brand-owned and brand-earned assets clearly labeled and kept separate.

The competitive set, tracked surfaces, market, language, time window, and rules for counting mentions must be disclosed. Otherwise, a score can change because the comparison changed. Raw citation totals also distort comparisons when one engine cites many sources and another cites few, so weighted visibility and prompt-level win rates provide a stronger basis.

Revenue can reverse the apparent competitive order. A brand with high classic share of voice can generate less revenue than a less-mentioned competitor whose citation-to-conversion rate is higher. Classic share of voice and revenue share of voice can therefore rank brands in opposite order. A platform reporting only the former may steer attention away from the highest-value opportunities. Presence, recommendation, citation, and commercial impact require separate measures.

Criterion 6 — Sentiment and accuracy detection: why appearing negatively is worse than not appearing

A buyer asks for vendors and receives a description that incorrectly places a brand in the enterprise-only pricing tier. The brand appeared, but the description can discourage consideration. A visibility chart might still record a success.

Sentiment and accuracy therefore need separate treatment. Positive, neutral, and negative labels describe framing, while accuracy identifies what the engine gets right or wrong. A capable platform should also trace which sources drive errors or negative descriptions, so teams can address the evidence. An inaccurate statement can sound entirely positive.

The classification method must be open to inspection. Labels, confidence thresholds, human-review provisions, and historical definitions should be documented. In a demo, ask whether the system distinguishes factual error from negative framing and explains the source of a change. A tone label without that context is an incomplete guardrail.

The architectural choice that sits under all the other criteria: data ownership and self-hosting

Who controls the evidence matters before any demo. Legal, finance, security, and engineering teams have a stake in that answer. Vendor-controlled storage can tie historical access and audit trails to a subscription. Operator-controlled storage lets the team keep trend data without that subscription and work with raw outputs.

MIT-licensed software has no software-license fee, but using the model API still costs money. With bring-your-own-key, the team pays model providers directly at published rates without an intermediary usage markup. Free software is not the same as free operation.

The practical questions concern storage, access, and continuity. Which database holds the outputs, who controls it, and is its schema documented? Who holds the API keys, and can the team query raw results directly?

How the leading platforms perform against the full criteria set

Lettertrace is the strongest fit here for teams prioritizing scheduled cross-model monitoring and ownership of their evidence. It is MIT-licensed, self-hostable, and bring-your-own-key, with support for ChatGPT, Claude, Gemini, and Google AI Overviews. It automates prompt variations, multi-model queries, and aggregation into trend metrics. Visibility, prominence, sentiment, and share of voice can be followed by topic and model over time.

Its storage model makes that recommendation concrete. Outputs live in the operator’s own Supabase instance, so the team owns its historical data and can audit raw responses independently of a vendor relationship. There is no software-license fee or usage markup, but the team still pays model-provider charges. For founders, growth teams, and AEO/GEO practitioners moving beyond manually asking ChatGPT about their brand, it offers a systematic monitoring approach without enterprise SaaS pricing.

The specified coverage is narrower than the full evaluation floor, and buyers still need answers about capture mode, repetition counts, share-of-voice formulas, and sentiment methodology. Open source does not substitute for sampling discipline. The same evidence standard applies regardless of architecture.

CRM-connected monitoring fits teams that most need to link AI visibility to pipeline and revenue, while content-workflow tools help teams create content based on what they learn about visibility. Native single-engine reporting offers a view limited to one engine. Manual or one-time scans can answer a snapshot question, but they cannot establish response volatility or a durable visibility rate.

A practical evaluation checklist for taking into a vendor demo or RFP

For a formal request for vendor proposals, give each criterion a pass or fail based on the requirements set down in writing. Prompt methodology passes when intent coverage, suggestion origins, and support for custom, versioned prompt sets are clear. Engine coverage passes when the required engines and surfaces are directly tracked. Capture mode passes when browser or API collection is disclosed for each surface and any normalization is documented.

Sampling passes when repetition counts, mention rates, and collection frequency meet the stated standard. Share of voice passes when mention, recommendation, and citation shares remain separate, comparison boundaries are fixed, and differences in engine citation counts are normalized. Sentiment and accuracy pass when classifications are documented and the platform identifies sources behind negative or incorrect descriptions.

Data ownership passes when storage control, key handling, raw-output access, and historical portability meet the team’s requirements. Any unanswered criterion remains unproven, including for free and open-source software. The buying decision should rest on an inspectable measurement process.

Sources