Google’s search attorney, John Mueller, shared insights on diagnosing widespread crawling issues.
This guide was shared in response to a disturbance reported by Adrian Schmidt on LinkedIn. Google’s crawler stopped accessing several of his domains at the same time.
Despite the outage, Schmidt noted that live tests via Search Console continued to work without error messages.
Studies showed no increase in 5xx errors or problems with robots.txt requests.
What could be the problem?
Mueller’s response
In relation to the situation, Mueller pointed to shared infrastructure as the likely cause:
“If it’s shared across a bunch of domains and focused on something like crawling, it’s probably a problem with a shared piece of infrastructure. If it’s already recovering, at least it’s no longer urgent and you’ve suffered time to poke at the latest changes / infrastructure logs.”
Infrastructure survey
All affected sites used Cloudflare as their CDN, which raised some eyebrows.
When asked about troubleshooting, Mueller recommended checking Search Console data to determine if DNS or failed requests were causing the problem.
Mueller stated:
“The crawl stats in Search Console will also show a bit more, maybe help decide between eg DNS vs requests that fail.”
He also pointed out that timing was a key clue:
“If it’s all at the exact same time, it wouldn’t be robots.txt and probably not DNS.”
Impact on search results
Regarding search visibility, Mueller assured that this type of disruption would not cause any problems:
“If this is from today and it just lasted a few hours, I wouldn’t expect any visible problems in the search.”
Why this matters
When Googlebot suddenly stops crawling across several websites simultaneously, it can be challenging to identify the cause.
While temporary crawl pauses may not immediately affect search rankings, they can interfere with Google’s ability to discover and index new content.
The incident highlights a vulnerability organizations may face without realizing it, especially those that rely on shared infrastructure.
How this can help you
If time Googlebot stops crawling your sites:
- Check if the problem affects multiple sites at once
- First, look at your shared infrastructure
- Use Search Console data to narrow down the cause
- Don’t exclude DNS just because regular traffic looks fine
- Keep an eye on your logs
For anyone running multiple sites behind a CDN, make sure to:
- Have good logging set up
- Keep an eye on your crawl speeds
- Know who to call when things go sideways
- Keep an eye on your infrastructure provider
Featured image: PeopleImages.com – Yuri A/Shutterstock