Every engineer eventually meets a flaky test. It passes on Monday, fails on Tuesday, passes again on Wednesday, and nobody has touched the code. The rule I have followed for 30 years of writing software is simple. A flaky test is quarantined. It is never celebrated when it goes green and nobody gets fired when it goes red, because its colour is telling you about the test, not about the system.
On 1 May 2026, SISTRIX published 1,548,213 snapshots of AI answers taken over 17 weeks. ChatGPT Search replaced 74% of its cited domains every single week. Google’s AI Mode replaced 56%. For the median ChatGPT prompt, not one domain survived all 17 weeks.
That is a flaky test. Most AI visibility programmes are reporting it to the board as a result.
Here is the argument. Brand mention and citation are sold as one metric and they are two different instruments. A mention is the model saying your name in the answer. It tracks reputation, it is built over years by other people writing about you, and it moves slowly. A citation is the model linking to your page as a source. It tracks retrieval, it is decided fresh on every run, and it resets weekly. Most programmes are priced on the first and reported on the second, which means they are mispriced in both directions at once.
Two instruments, one dashboard
Start with the definitions, because the vendors blur them.
A mention happens inside the generated text. The model writes “the usual shortlist here is X, Y and Z” and your brand is one of the letters. Nothing is linked. Nothing is clicked. The buyer simply hears your name in the same sentence as the category.
A citation happens in the source list. The model retrieves pages to ground its answer and shows some of them as links. Your page is one of the links. The answer itself may never name you.
These sound like two views of the same thing. The data says otherwise. On 9 June 2026, Semrush published a study run with Kevin Indig’s Growth Memo across 3,981 domain appearances, 115 prompts and 14 countries, on ChatGPT, Gemini, Google AI Overviews and AI Mode. They split every appearance into three buckets.
61.7% were cited but never mentioned. The brand’s page was a source and the brand’s name was absent from the answer. Semrush calls these ghost citations.
25.1% were mentioned but never cited. The name was in the answer and no page of theirs was linked.
13.2% were both.
So in the combined set, only about one appearance in eight did the thing every AI visibility pitch deck implies is normal: name the brand and link to it. The other seven did one or the other. If your dashboard rolls those into a single “visibility” figure, it is adding a reputation number to a retrieval number and reporting the sum as if it meant something.
Why they move at different speeds
The mechanism is not mysterious once you look at where each one comes from.
A mention comes out of what the model already believes about a category, reinforced by what it reads at query time. That belief is formed by volume of third-party writing over a long period: reviews, comparisons, forum threads, analyst notes, press. Ahrefs studied 75,000 brands in its AI Overview analysis, published May 2025 and updated April 2026. The strongest correlate of being mentioned was branded web mentions across the web, at 0.664. Backlinks came in at 0.218. Domain Rating at 0.326. The number of pages on your own site, the thing most content budgets buy, sat at 0.17.
The distribution is steeper than the correlation. Brands in the top quarter for web mentions averaged 169 AI Overview mentions. The next quarter down averaged 14. The bottom half averaged between zero and three. That is not a gradient you climb in a quarter. It is a stock that compounds, the way a reputation does, and the brands at the top got there by being written about for years.
A citation comes out of retrieval. The engine fans out the question into sub-queries, fetches candidate pages, and picks a handful that best support the sentence it is about to write. Change the phrasing, the user’s location, the date, the model version or simply run it again, and the candidate set changes. AirOps tracked more than 45,000 citations and found only about 30% of brands stayed visible from one run to the next. Only one in five held visibility from the first run to the fifth. About 57% of those that dropped out came back within a couple of runs. That is not a brand gaining or losing ground. That is sampling noise.
SparkToro and Gumshoe ran the same experiment from the buyer’s side. Rand Fishkin and Patrick O’Donnell had 600 volunteers put 12 identical prompts through ChatGPT, Claude and Google’s AI, nearly 3,000 runs in total. The same list of recommended brands came back less than once in 100 runs. The same list in the same order came back about once in 1,000. And yet some brands appeared in 60% to 90% of answers for a given intent, regardless of where they ranked. Their conclusion was the right one: repeat presence means something, exact position does not.
Put those together and you have two instruments. One is a slow reputation gauge that should be read quarterly. The other is a fast retrieval sampler that should be read as a distribution, never as a point. Most dashboards read both weekly, on one line, in one colour.
Case one: the same brand, opposite numbers, on two engines
The Semrush data contains a result that should end the single-score argument on its own.
ChatGPT cited brands in 87% of their appearances and mentioned them in only 20.7%. Gemini did the reverse. It mentioned brands in 83.7% of appearances and cited them in 21.4%.
Now think about what a brand sees when it buys a tool that reports citation share. On ChatGPT it looks excellent. Its pages are being used as sources almost every time. On Gemini it looks like it barely exists. Its pages are rarely linked.
But the buyer reading Gemini is hearing the brand’s name four times out of five. The buyer reading ChatGPT mostly is not. The citation dashboard has the commercial reality upside down. The engine where the brand is being recommended aloud looks like the problem, and the engine where its content is being used anonymously to support somebody else’s answer looks like the win.
Flip the tool to report mentions and the picture inverts. Neither view is wrong. They are measuring different things, and a programme that picks one and calls it visibility has chosen which half of the truth to fund. Ask which half before you sign.
Case two: the weekly report that is mostly noise
The second case is in the volatility numbers, and it is the one that costs the most money.
The SISTRIX data covers Google AI Overviews, AI Mode and ChatGPT Search across six countries from 17 December 2025 to 8 April 2026, 82,619 prompts in all. The three platforms behave like three different climates.
AI Overviews are close to classic search. Weekly drift was around 5%, and 53% of prompts showed no change in sources across the whole 17 weeks. Answers carry about 11 domains and about 8 of them stay put.
AI Mode churns 56% of its domains a week, citing 14 to 16 per answer. It does keep a stable core of one to five domains for about 86.5% of prompts. Everything around that core is turnover.
ChatGPT Search churns 74% a week and cites only three or four domains per answer. Fewer slots, faster turnover.
BrightEdge, measuring citation share separately, found the same shape from another angle. Domains cited frequently moved about 0.7% a week. Domains cited sporadically swung 50% or more, a gap of roughly 70 times. Their threshold for stability was about 50 citations. Below it, weekly volatility ran around 50%. Above it, around 8%.
Here is what that does to a monthly report. A mid-sized B2B brand is almost never in the frequently cited group. It sits well below 50 citations on most engines. Its weekly citation share will swing by half on its own, with no change to anything the brand did. A report that says “citations up 38% this month” is, for that brand on that engine, mostly a report about the dice.
And the dice get acted on. Somebody rewrites a page because citations fell. Somebody gets credit because they rose. Somebody renews a retainer on a good month and cancels it on a bad one. Each of those decisions is a reaction to a flaky test.
The one stable line in the SISTRIX data is worth noting. For brand queries in AI Mode, the brand’s own domain appeared in all 17 weeks 43% of the time and in at least 80% of weeks 66% of the time. The domains co-cited alongside it changed about 70% a week. When somebody asks about you by name, your site is reliably there. When somebody asks about the category, which is where new demand comes from, the citation slots rotate constantly. The stable number is the one that matters least for growth.
Case three: the reputation stock nobody budgets for
The third case runs the other way. It is where the slow instrument gets judged on the fast one’s timetable and loses.
Look again at the Ahrefs quartiles. The bottom half of brands by web mentions get between zero and three AI Overview mentions on average. The top quarter get 169. The only way across that gap is to be written about, at volume, by people who are not you: customers on review sites, practitioners in forums, journalists, analysts, partners, comparison articles. Seer Interactive’s study of 10,000 questions run through GPT-4o found the same story through a different lens: page-one Google rankings correlated with brand mentions at about 0.65, while backlinks were weak or neutral. Mentions follow the brands the open web already treats as the answer.
That kind of work is PR, analyst relations, review generation and customer marketing. Its payback period is measured in quarters, often years. It compounds. It is also almost invisible on a weekly citation dashboard, because a new review or a trade press piece does not reliably turn into a link from ChatGPT next Tuesday. It turns, slowly, into the model saying your name.
So the reputation programme gets reported on a retrieval chart. Three months in, the chart is flat or noisy. The CFO asks what the money bought. Nobody can point at a line that moved, because the line they are looking at was never going to move from that work. The programme is cut just before the stock would have started to show.
Meanwhile the citation programme, the page restructuring and freshness and schema work, gets priced like reputation. Tools are sold per prompt tracked: Profound’s self-serve Starter plan, before it was withdrawn in September 2026, was 50 ChatGPT prompts for $99 a month. Agencies sell annual retainers on top. Both are priced as if they buy a durable asset. What they mostly buy, below the 50 citation threshold, is a better position in a lottery that redraws weekly. That work is real and worth doing. It is worth doing at the price of maintenance, not the price of brand.
That is the mispricing in both directions. The slow asset is killed on a fast clock. The fast activity is paid for on a slow contract.
How I read a number like this
Every reporting layer I have built for a private equity owner separates two kinds of number: stocks and flows. Cash on the balance sheet is a stock. Weekly receipts are a flow. You never put them on the same axis, because the scale and the noise are different, and somebody will eventually react to the wrong one.
AI visibility needs the same split. Mention rate is a stock. You measure it with a large, fixed prompt set, many runs per prompt, and you read it quarterly. Citation share is a flow. You measure it per engine, as a rolling distribution, and you read the band, not the point. The moment those two go on one line, you have built a dashboard that will tell you a reputation programme failed and a formatting tweak succeeded, in the same month, for reasons that have nothing to do with either.
Where this breaks
Three honest objections.
First, citations send traffic and mentions do not. A ghost citation still puts a clickable link in front of a buyer, and a mention without a link sends nobody anywhere. That is true, and for a business that lives on referral clicks the citation flow matters more than I have given it credit for. But for a considered B2B purchase the click is rarely the point. The buyer who hears your name in an answer and then searches for you directly will show up as branded search, not as an AI referral. The mention’s value lands somewhere the citation dashboard does not look.
Second, the two are not independent. AirOps found brands that earned both a citation and a mention were 40% more likely to resurface across runs than citation-only brands. The reputation stock clearly steadies the retrieval flow. I agree, and that is the strongest argument for funding the slow work. It is not an argument for reporting them as one number. Correlated instruments still need separate gauges, or you cannot tell which one moved.
Third, not every engine is weather. Google AI Overviews drifted about 5% a week in the SISTRIX data, with more than half of prompts unchanged across four months. On that surface a citation behaves much more like a classic ranking, and reading it monthly is fine. The weather problem is specific to AI Mode, ChatGPT and Perplexity. That is a real limit on my argument, and it is also the reason a single cross-engine score is worse than useless: it averages a stable climate with two unstable ones and hides which is which.
What I would do on Monday
Split the dashboard. Report mention rate and citation share as two numbers, per engine, never blended. If your tool cannot separate them, that is the first finding.
Ask your vendor for the denominator. How many prompts, how many runs per prompt, which engines, which countries. SparkToro’s data says a single run per prompt tells you almost nothing. If the answer is one run a week, treat every movement as noise until proven otherwise.
Put a band on the citation line. Plot an eight-week rolling share with its normal range shaded. Any week inside the band gets no commentary and no action. Only a break outside the band is news.
Move the mention budget to a longer clock. Give PR, review generation and analyst work a two to four quarter target on mention rate, owned by whoever owns brand, and stop reviewing it monthly against citations.
Count how many of your domains clear about 50 citations on each engine. That is your stable footprint. Everything below it is weather, and it should be priced as maintenance, not as an asset.
Back to the flaky test. Nobody on a serious engineering team pages a director because a known flaky test went red overnight. They quarantine it, fix the harness, and find a measurement they can trust. Most marketing teams are still paging the director. Quarantine the weekly citation number, measure the slow thing on its own clock, and the board pack will finally show which work is building the brand.
The name in the answer is the asset. The link is the weather report.
If you want a second pair of eyes on how your AI visibility is being measured and priced, send me your last three reports and put time in my diary.
It's ALL directional.... not cannon