If a page isn’t crawled, it can’t be indexed. If it isn’t indexed, it can never rank — no matter how good the content is, how fast the page loads, or how many backlinks point to it. The gatekeeper for all of it is a single piece of software: Googlebot.
Most explanations of Googlebot stop at “it’s Google’s web crawler,” which is technically true and practically useless. It doesn’t tell you why Googlebot might be skipping half your product pages, why your new blog post took three weeks to show up in search, or why your site’s crawl stats in Search Console look nothing like your traffic.
This guide breaks down what Googlebot actually is, the different versions of it you’ll encounter in your server logs, and the full three-stage pipeline — crawling, indexing, ranking — that turns a URL into a search result. We’ll also cover the specific things that block Googlebot from doing its job, and how to verify, in your own data, that it’s doing what you think it’s doing.
If you want the wider picture of how crawlability fits into technical SEO as a whole, our covers the other pillars — indexability, site architecture, speed, and security — that this article’s topics feed directly into.
Table of Contents
What Is Googlebot?
Googlebot is the web crawler Google uses to discover, fetch, and process pages so they can be added to Google’s index. It’s not one program running on one machine — it’s a distributed system of automated crawlers that continuously visit URLs, follow links, and hand off what they find to Google’s indexing systems.
Googlebot doesn’t decide where your page ranks. It has one job: find content and retrieve it. Ranking happens later, in a completely separate stage, using hundreds of different signals. Confusing crawling with ranking is the single most common misunderstanding in SEO — a page can be crawled perfectly and still rank on page 9, and a page can be poorly optimized and never get crawled at all. They’re independent problems with independent fixes.
How Googlebot decides what to crawl
Googlebot doesn’t wander the internet at random. It works from a constantly updated list of URLs built from three main sources:
- Previously crawled pages — URLs Google already knows about from past crawls.
- XML sitemaps — URLs that site owners explicitly submit via Search Console or
robots.txt. - Links — both internal links (from your own site’s navigation and content) and external links from other sites Google has already crawled.
This is why internal linking matters so much: a page with zero incoming links, internal or external, is effectively invisible to Googlebot until it’s added to a sitemap. It’s also why backlinks do double duty — they’re not just a ranking signal, they’re a discovery mechanism. If you want to see which external sites are currently passing that discovery signal into your domain (and which competitor sites are picking up links you aren’t), our walks through the tools that surface this data.
The Different Types of Googlebot
Server logs rarely show a single user-agent called “Googlebot.” In practice, you’ll see several variants, each simulating a different device or content type:
Since 2018, Google has crawled the overwhelming majority of the web with mobile-first indexing, meaning the Googlebot Smartphone version is what generates your actual index entry for most sites. If your mobile page hides content, images, or links that exist on desktop (a common leftover from old “mobile-friendly” redesigns), Googlebot Smartphone simply never sees that content — and it won’t be indexed, full stop.
How Search Engines Actually Work: The Three-Stage Pipeline
“How do search engines work?” is a bigger question than “what is Googlebot,” but Googlebot is stage one of a three-stage pipeline that every page passes through. Understanding all three stages — and where they fail independently — is the fastest way to diagnose why a page isn’t showing up.
Stage 1: Crawling
Crawling is discovery and retrieval. Googlebot requests a URL, downloads the HTML (and, increasingly, renders the JavaScript on the page using a headless Chromium browser), and follows the links it finds to queue up more URLs. Every site gets an implicit crawl budget — a rough ceiling on how many pages Googlebot will fetch from that domain in a given period, based on the site’s size, server response speed, and perceived value. Waste that budget on low-value URLs (infinite filter combinations, duplicate parameter URLs, expired pages returning 200 instead of 404) and your genuinely new or updated pages wait longer to be seen.
Stage 2: Indexing
Once a page is fetched, Google’s systems analyze the content, extract signals (title tags, headings, structured data, canonical tags, main content vs. boilerplate), and decide whether to add it to the index — the actual searchable database. Crawling doesn’t guarantee indexing. Google skips indexing pages it judges to be duplicate, thin, low-value, or blocked by directives like noindex or a canonical tag pointing elsewhere. This is why “Crawled – currently not indexed” is one of the most common statuses site owners see in Search Console: Googlebot did its job, but the indexing stage rejected the page.
Stage 3: Ranking
Only indexed pages are eligible to rank, and ranking is a separate system using hundreds of signals — relevance, content quality, page experience, links, and many factors specific to the query itself. This is the stage most SEO content focuses on, but it’s downstream of the first two. Optimizing ranking factors on a page Googlebot can’t reach, or that Google won’t index, changes nothing.
Diagram 2 (to be designed): “The Crawl → Index → Rank Pipeline” — three connected boxes with a funnel shape (wide at crawl, narrowing at index, narrowing further at rank), each box annotated with the specific Search Console status/report that corresponds to it: Crawl Stats report → Coverage/Page Indexing report → Search Analytics performance report. This gives readers a direct map from “what I’m reading in Search Console” to “which pipeline stage that describes.”
We live, breathe & Dream Traffic
What Blocks Googlebot From Crawling Your Site
- robots.txt disallow rules left over from staging, or overly broad rules (Disallow: /*?) that accidentally block legitimate URL parameters.
- noindex meta tags or headers applied at the template level, so an entire section (paginated archives, a whole subfolder) is invisible without anyone intending it.
- Orphan pages — pages with no internal links pointing to them and no sitemap entry, so Googlebot has no path to find them.
- Broken or redirect-chained internal links that waste crawl budget and dilute the link equity that helps discovery.
- Slow server response times. Googlebot backs off its crawl rate when a server responds slowly or returns errors, directly reducing how much of the site gets crawled per visit.
- JavaScript-only rendering without a server-rendered fallback, which can delay indexing since rendering is a more resource-intensive second pass for Google’s systems.
- Soft 404s — pages that return a 200 status but show “not found” or empty content, confusing what should and shouldn’t be indexed.
How to Check What Googlebot Is Actually Doing on Your Site
Don’t guess — Google gives you direct visibility into all three pipeline stages:
- Crawl Stats report (Search Console → Settings → Crawl stats): Shows total crawl requests, average response time, and a breakdown by response code and file type. A sudden drop in crawl requests is an early warning sign, not a good sign of “less waste.”
- URL Inspection tool: Enter any URL to see the last crawl date, the indexing status, and whether the live version matches what’s indexed. This is the fastest way to confirm Googlebot has actually seen a specific page.
- Page Indexing report: Aggregates every indexed and non-indexed URL Google knows about for your property, with a reason code for every exclusion.
- Server log file analysis: The most granular option — filtering raw server logs for verified Googlebot user-agents shows exactly which URLs were requested, when, how often, and with what response code. This is the only method that shows crawl behavior Search Console doesn’t sample or summarize.
Our full walkthrough on goes deeper into reading these reports and turning them into a prioritized fix list.
Googlebot, Crawl Budget, and Competitive Sites
Crawl budget matters most on large or frequently updated sites — e-commerce catalogs, marketplaces, publishers — where the number of URLs competes for a finite amount of Googlebot attention. But crawl efficiency is also a competitive variable: if a competitor’s site architecture makes it easier for Googlebot to find and re-crawl their fresh content, they get indexed faster and can capture demand on a trending query before you do.
This is worth checking directly rather than assuming. Once you understand your own crawl and index footprint, the natural next step is sizing it up against competitors — how many of their pages are indexed, how their internal linking distributes authority, and where the content gaps are that a stronger site architecture could close. Our guide to covers exactly that process.
Frequently Asked Questions
Is Googlebot the same as Google’s ranking algorithm?
No. Googlebot only crawls and fetches pages. Ranking is handled by entirely separate systems that run after a page has already been indexed. A page can be crawled flawlessly and still rank poorly, or fail to rank at all, for reasons unrelated to crawling.
How often does Googlebot crawl a website?
It varies by site and has no fixed schedule. Frequently updated, high-authority sites may be crawled multiple times a day; smaller or rarely updated sites might be crawled every few days or weeks. Crawl frequency is influenced by how often content changes, server response speed, and the site’s overall crawl budget.
What’s the difference between Googlebot Smartphone and Googlebot Desktop?
Googlebot Smartphone simulates a mobile device and, under mobile-first indexing, is the version that determines what gets indexed for the vast majority of sites. Googlebot Desktop simulates a desktop browser and plays a secondary role today, mainly for sites without a working mobile version.
Why does Search Console say a page is “Crawled – currently not indexed”?
This means Googlebot successfully fetched the page, but Google’s indexing systems chose not to add it to the index — usually because the content was judged duplicate, thin, or lower-value relative to similar pages already indexed. It’s an indexing-stage decision, not a crawling failure.
Does submitting a sitemap guarantee my pages will be indexed?
No. A sitemap only helps with discovery — it tells Googlebot a URL exists and should be considered for crawling. It doesn’t override quality or duplication signals at the indexing stage, and it doesn’t guarantee a crawl will happen on any particular timeline.
Check out more blog posts:
