Skip to content
GEO and AI search

GPTBot

GPTBot is the crawler OpenAI uses to collect publicly available web content to help train and improve its AI models. It is separate from OAI-SearchBot, which OpenAI uses for ChatGPT's live web search, so blocking one does not automatically block the other.

Also called: OpenAI GPTBot

Why it matters to a buyer

Because GPTBot feeds model training rather than live ChatGPT search citations, blocking it is a decision about whether a site's content should help improve future OpenAI models, not a decision about whether the site can be cited by ChatGPT today, and confusing the two leads to sites blocking the wrong bot for the outcome they actually want. A buyer should decide separately, and deliberately, on GPTBot and on OAI-SearchBot rather than treating "OpenAI" as one single crawler to allow or block.

How it is measured or applied

GPTBot is named explicitly in robots.txt with its own user-agent line, allowed or disallowed independently of OAI-SearchBot and ChatGPT-User, OpenAI's separate crawlers for search indexing and live user-triggered fetches. OpenAI documents the bot's published IP ranges for firewalls or CDNs that need to verify a request genuinely came from GPTBot rather than a spoofed user agent. Access is confirmed by checking robots.txt directly and by reviewing server logs for GPTBot requests and the response codes they received, since a firewall can silently block the bot even when robots.txt allows it.

Sources

  1. Overview of OpenAI crawlers: GPTBot, OpenAI

Thirty minutes to see where your site is invisible.

Book a free consultation with Ibrahem and Jeremy. We look at your site on search, AI answers and conversion while we talk, and you leave with a prioritised plan.

30 minutes. No pitch deck. You leave with a prioritised plan.