Googlebots are essential to the smooth running of the search engine, ceaselessly probing websites to ensure that Google stays up to date with the latest developments and site changes.
But for every 25 genuine Googlebots that visit your site you'll also be visited by a fake Googlebot -- more than 23 percent of which are used for DDoS attacks, hacking, scraping and spamming. This is among the findings of security company Incapsula's 2014 Search Engine Study.
The study is based on over 400 million search engine visits to 10,000 sites over 30 days and shows Google is the most active search engine. Googlebots are responsible for more than 60 percent of all page crawls, the second in line the MSN/Bing bot notches up only 24.5 percent. What's also interesting is that there's no correlation between the number of human visits to a site and how often it’s crawled by Googlebots. Google pays just as much attention to the backwaters of the web as it does to popular sites.
The study does note though that content-heavy and frequently updated websites were more thoroughly crawled. This behavior was most notable in the cases of big forums, news sites and large-scale shop sites with a wide array of frequently updated products.
Most interesting and concerning though is the number of fake Googlebots. The study shows that over 4 percent of bots using Google's HTTPS user agent aren't what they claim to be. The benefit of this to hackers is that site owners generally allow unhindered access to Google's crawlers in order to protect their search results.
Incapsula's logs reveal that fake Googlebots are used mainly for DDoS attacks but also in spamming and hacking activity. Security solutions that don't use case-by-case traffic inspection are unable to spot the real bots from the fakes.
To remain secure, sites need to use a security solution that combines heuristics and IP and ASN verification to identify bots based on their origin.
You can read more about Googlebot and fake bot activity on Incapsula's blog and there's a summary of the report's findings in infographic form below.
Photo Credit: Gunnar Assmy/Shutterstock