Why it matters to a buyer
robots.txt is a common source of two opposite mistakes: sites that accidentally disallow their whole site, or a key section of it, after a platform change, and sites that assume disallowing a page there will remove it from search, when in fact a disallowed page can still appear in results, without a snippet, if enough other sites link to it. A buyer should know the file controls crawling access, not indexing removal, and that noindex is the correct tool for the second job.
How it is measured or applied
The file lists rules per user agent, naming which crawlers, Googlebot, Bingbot, GPTBot and others, are allowed or disallowed from which paths, and it also typically points to the site's XML sitemap. It is applied by disallowing genuinely low-value or sensitive paths, like admin areas or internal search results, while leaving content meant to be found openly crawlable. It is checked with a robots.txt testing tool and by reviewing server logs to confirm the crawlers a site expects, and only the ones it intends, are actually able to reach the paths they need.
Sources
- Robots.txt introduction and guide, Google Search Central
