Comprehensive analysis of 4.3 million crawler requests shows major frontier bots never request Markdown, leaving Cloudflare Markdown for Agents with zero impact on AI search citations.
PHOENIX, AZ
AnswerShare, the pioneer of the AI Translation Layer released findings from a new study demonstrating that serving Markdown content via tools like Cloudflare Markdown for Agent has no impact on frontier AI crawler behavior, search engine citations, or recommendations.
Despite claims from industry experts that Markdown solves HTML bloat and makes pages easier for AI to ingest, the study shows that major frontier bots simply do not request Markdown format.
Methodology and Findings
Leveraging an active weekly crawl volume of approximately 1.5 million requests across multiple edge-controlled domains, the study conducted a rigorous, data-driven evaluation.
The first phase of the experiment tested whether wildcard accept headers (such as the complex header sent by OpenAI’s GPTBot) would trigger Markdown delivery.
Testing with 20,000 crawler simulations confirmed that Cloudflare only serves Markdown when it is explicitly requested in the headers via ‘Accept: text/markdown’.
The second phase analyzed a sample of 4,300,723 real-world requests across five domains to determine how often frontier crawlers request Markdown. Out of those requests, only 108,736 (about 2.5%) requested Markdown. Crucially, the frontier bots responsible for the vast majority of AI search traffic never requested Markdown:
- PerplexityBot: 605,811 observations – 0 Markdown requests
- Meta-ExternalAgent: 359,222 observations – 0 Markdown requests
- GPTBot: 260,472 observations – 0 Markdown requests
- OAI-SearchBot: 33,631 observations – 0 Markdown requests
- ClaudeBot: 18,831 observations – 0 Markdown requests
In contrast, only specific specialized or unclassified crawlers requested Markdown:
- ExaSearchBot: 67,578 requests (99.02%)
- ShapBot: 3,588 requests (100%)
- Unclassified bots: 4,615 requests
- Claude-User Impersonators: 248 requests
- GoogleBot impersonators: 32,747 requests
Implications for GEO/AEO
“While we didn’t look at all markdown services, the result for Cloudflare is clear.” said Robert Maynard, Founder of AnswerShare. “They built the pipe, but major AI crawlers aren’t using the faucet. Until AI labs update their scrapers to request text/markdown, enabling the feature yields no real-world impact on frontier AI indexing.
While automated Markdown translation fails to influence key crawlers, organizations looking to meaningfully impact AI engine behavior must look to more direct solutions, such as those provided by AnswerShare.
Manifests, logs and methodology are available at AnswerShare Research.
About AnswerShare
AnswerShare is the pioneer of the AI Translation Layer, helping clients actively shape and influence AI crawler behavior and citation outcomes.
Media Contact Details
Kevin Myers
Email: Send Email
Website: answershare.com