Ask an AI assistant for a CRM for a 20-person sales team and you get a short list. It names four or five vendors, adds a sentence on each, and maybe notes which one suits a tight budget. The buyer now has a shortlist. Every vendor outside it lost the deal before anyone booked a demo.
Business software is a useful test case for how this works. The market has thousands of vendors, and products change every few weeks. The details that decide a purchase are pricing, limits and integrations. When assistants get a category like this wrong, the cost shows up directly in the sales pipeline, and the same causes apply to almost any industry.
An assistant knows a brand in two ways
The first way is memory. During training, a model reads a large share of the public web. What it learns about a brand depends on how often and how consistently that brand was written about. That knowledge stops at the training cutoff, so a feature shipped last quarter or a plan launched last month may not exist for the model at all.
The second way is live lookup. When an assistant searches the web to answer a question, it sends a crawler to fetch pages and summarizes what it finds. Most major AI providers publish the names of their crawlers, and several run separate ones for training and for search. A site’s robots.txt file, the web standard for telling crawlers what they may fetch (RFC 9309), can allow or block them by name, though fetches a user triggers directly don’t always follow it.
That split matters. A brand can block training crawlers and still appear in search-based answers. It can also block everything by accident and disappear from both. It is worth knowing which of these your site does today.
Why brands go missing
The door is closed. Many sites block AI crawlers without anyone deciding to. An old robots.txt rule, a security plugin, or bot protection at the hosting provider can turn away crawlers the marketing team would gladly let in. To the assistant, a blocked brand looks the same as one that doesn’t exist.
The page is empty until a script runs. Many AI crawlers read the HTML a server sends and do not run the scripts that fill in the page. Software marketing sites are often built as single-page apps, with pricing tables, feature comparisons and integration directories loaded through scripts. A person sees a complete pricing page, while the crawler sees a header, a footer and very little in between.
The facts are in the wrong format. Crawlers struggle with pricing behind a “contact sales” form, feature details in a gated PDF, or a comparison chart saved as an image. Pricing adds its own trap: “$29” means little unless the page says whether that is per user or per account, per month or per year, and which limits apply. Models tend to fill those gaps with confidence.
Someone else tells the story. When a brand’s own site can’t be read, the model fills the gap with whatever it can read: review platforms, comparison posts from three years ago, forum threads. That is how assistants end up quoting retired plans, listing integrations a product dropped, or describing a product under its name from before a rebrand.
The names collide. Software names collide easily. Many products carry common words as names, get renamed after an acquisition, or share a name between a suite and one of its modules. The assistant may blend them into one product that never existed.
What makes a brand easy to recommend
Assistants favor information they can confirm. When a brand’s site, its documentation, its partners’ pages and independent reviews all state the same facts, the model has one consistent picture and repeats it. When the sources disagree, the brand looks uncertain, and uncertain brands get left off short lists.
Category language helps more than most brands expect. Buyers ask for “a password manager for a small team with single sign-on” or “a CRM for real estate agents.” A model can only connect a brand to that request if the brand’s pages use those words. A site that describes its product only with a tagline and a brand story gives the model nothing to match.
Structured data hands machines the facts directly. The schema.org SoftwareApplication vocabulary lets a page state name, category, platform and price in a form software reads without interpretation. Search engines have supported it for years, and it serves AI crawlers just as well.
Evidence on the page also counts. Researchers from Princeton, Georgia Tech, the Allen Institute for AI and IIT Delhi tested how content changes affect visibility in AI-generated answers (Aggarwal et al., KDD 2024). Adding citations, quotations from relevant sources and statistics raised visibility by over 40% in their benchmark, and by up to 37% on a commercial AI search engine. The effect varied by subject area, so no single tactic works everywhere.
Some brands are also adding an llms.txt file, a proposed format that gives AI systems a short, curated map of a site. It is still a proposal, and AI providers have not all confirmed that they read it. It costs little to publish, though it won’t help a site whose product pages can’t be read in the first place.
What a brand can check about itself
A brand can start without special tools. These five checks fit into an afternoon.
1 Ask the assistants the way buyers do. Ask about the brand by name, then ask the category question without it, for example “best CRM for a 20-person sales team under $50 per user.” Ask several assistants, several times each, because answers vary between runs. Note who gets named, and what is said about you when you appear.
2 Read your robots.txt against the published crawler names. Check which AI crawlers are allowed. Then ask whoever runs your hosting or security whether bot protection turns them away before robots.txt is ever read.
3 Look at your pricing page the way a crawler does. Turn off JavaScript in your browser or view the page source, and see whether prices, plans and limits are still there.
4 Check that your prices and limits stand on their own. Every price should say what it covers and how it is billed, and every limit should state its number. Say it the same way on your site, in your docs, on partner listings and on your review profiles.
5 Follow the citations. Assistants that show their sources reveal which pages speak for you. If they cite a two-year-old comparison post instead of your pricing page, you know where the model gets its picture of you.
Your website now has two audiences
For two decades, brand websites were built for people and tuned for search engines. They now have a second audience that reads differently. It skips images, ignores scripts, and trusts what it can confirm across several sources.
The brands that get recommended are the ones whose facts are easiest to read and easiest to confirm. That makes this a website problem, and website problems can be fixed.
