State of Search: Crawling Frequency

Google’s status as a global tech superpower is unquestioned – in Q3, it drove 92% of all US organic searches. Its influence is everywhere. However, even Google is limited by resources and bandwidth. The SEO community received a useful reminder of this last month in the form of a Q&A with Google’s John Mueller, who confirmed again what we all observe on a daily basis: the search engine’s web crawlers visit and refresh URLs on a very irregular basis. According to Mueller, “Some URLs we will crawl daily. Some URLs maybe weekly. Other URLs every couple of months, maybe even once every half year or so.”

This quote is Exhibit A in why it’s critical to have a proactive team working on your behalf to make sure content updates are caught and properly reflected by Google’s crawlers. Mueller recommends things like uploading fresh sitemaps and using Google’s URL inspection tool to trigger reviews of fresh content when needed; site owners cannot just sit back and assume updates will be noticed by the algorithm.

What does this mean for you?

In order to avoid falling into the category of content that is crawled “every half year or so”, our team is regularly triggering fresh assessments of everything we manage on your behalf. This helps speed up the process, but, as Mueller’s comments make clear, even Google operates under certain constraints – sometimes a necessary amount of patience is part of the game.