Googlebot: The Basics You Need to Know

The traditional way of thinking about web indexing is through the idea of ‘crawlers’, ‘ants’ or ‘spiders’ – billions of workman-like insect things scurrying about the internet, collecting up data for search engines’ algorithms to analyse. But Google have kind of rebranded their data-gathering internet software recently. They’re now calling it Googlebot. Which, instead of loads of little bugs doing the work, it makes you think about some enormous web-obsessed android sat hunched over a laptop, avidly reading every web page that gets published.

At least, that’s what it makes us think of anyway…maybe we’re just a little odd.

Google – metaphorically – employ the Googlebot to prowl the web, reading and retrieving useful information (words, resources, links, codes) about new and existing web content to help keep the web giant’s search engine as the biggest and most definitive search engine on the internet. It’s constantly working, endlessly reviewing new pages, updates to existing ones and feeding back that information to Google Headquarters so they can ultimately decide what pages are worthy of indexing (and in what order) and which are not.

Think of the Googlebot as a collector. The bot does his rounds and, when it’s your turn, he or she goes through what you have on your site and hungrily gathers up information. In a split second. When Googlebot’s report is written up, it gets sent over to Google HQ and their indexing software dictates your Google placing. So, obviously, the Googlebot and its Index supervisor are people you want to impress. SEO is, fundamentally, about auditing and improving your web content to get a good score after the Googlebot has been and done his rounds.

So, it’s important that you ensure that your website is Googlebot-accessible. You should also make sure that it’s optimised in such a way that makes crawling a fast and easy process for Googlebot. The smoother the process, the better. Check your crawlability (there are lots of online tools to help you out with this – we particularly like Moz’s crawl test and the Yoast SEO plugin for WordPress users). Then fix any and all errors you stumble upon, perhaps with the help of a web developer you know and trust.

Keep adding fresh and relevant new content too (blogs are a great way of doing this). That’s important. You’ll then get crawled more often and the improvements you make will be picked up on – and rewarded – faster.

Another tactic you can employ is making the most of social media. Create shareable content that gains traction on SM channels and Googlebot will be impressed by your popularity and ability to engage your audience. Make sure you feature prominent social sharing buttons to make liking and sharing what you produce as easy as possible.

There’s not just one Googlebot, though. They’d be far too busy. There are quite a few and you need to optimise your site dependant on what’s important to you. The main types of bots that Google use are:

  • Google Mobile
  • Googlebot (regular desktop web search)
  • Googlebot Images
  • Googlebot Video
  • Googlebot News
  • Google Adsense
  • Google Mobile Adsense
  • Google Adsbot (landing page quality check)

If you want more details on each of these and how to optimise for each individual bot, it’s probably worth visiting the search engine’s rundown of all their Google crawlers.

That’s just the basics. If you need to really drill down into the detail on Googlebot and what’s really involved, there are plenty of resources online. Good luck and, as ever, if you think we can help you out with your online presence, please let us know!

Free Resource


Inbound Marketing Review

Heard about Inbound Marketing but not sure how it can help your business?

Sign up for an Inbound Review and we’ll show you how you can boost your lead generation.