Why it matters to a buyer
A page a buyer cares about can be well written and still invisible to search if nothing points a crawler to it or if a setting blocks the crawler outright. Crawling is worth understanding on its own because it is the step before indexing and ranking, so a report that jumps straight to rankings can miss a simpler, earlier problem: the page was never fetched at all, which shows up differently in diagnostics than a page that was fetched but not chosen to rank.
How it is measured or applied
Crawling is guided by internal links, XML sitemaps and, to a lesser extent, external links, all of which give a crawler a path to a URL; pages with no path in, called orphan pages, are rarely crawled. Sites also influence crawling directly through robots.txt rules and response codes. It is monitored using Search Console's crawl stats report, which shows how often Googlebot visits and what it finds, and cross-checked against server logs, which record every crawler request a site actually received, including ones that never reach Search Console's own reporting.
Sources
- What is Googlebot, Google Search Central
