Why it matters to a buyer
An assistant cannot cite a page its crawler was not allowed to read. Many sites block AI crawlers without meaning to, through a firewall preset or a content delivery network setting labelled as bot protection. For a buyer, checking crawler access is one of the cheapest steps in any AI visibility plan, and it is also where a real business decision sits: search and user-triggered bots affect citations, while training bots affect only what future models learn.
How it is measured or applied
Access is controlled in robots.txt, which names each crawler's user agent and allows or disallows paths, and at the firewall or content delivery network, which can block requests before robots.txt is read. Each company documents its own agents and what each one is used for. Applying it means listing the agents you want, allowing them explicitly, and confirming that the firewall does not block them anyway. Verification uses server logs or the network's bot analytics to see which AI crawlers are actually fetching pages and receiving successful responses.
Sources
- Overview of OpenAI Crawlers, OpenAI
