php — how to detect spider/crawler with php

I am going to write a PHP script today that detects whether a search engine spider is crawling a page.With PHP, we can check the value of $_SERVER['HTTP_USER_AGENT'],a predefined variables which store the contents of the User-Agent: header from the current request, if there is one.

$pattern = ‘/(google|msnbot|MSNPTC|yahoo|Slurp|||DoCoMo|DoCoMo)/i’;
if ( preg_match($pattern, $_SERVER[‘HTTP_USER_AGENT’] === FALSE) { // not a spider

} else { // spider


You can find a list of search engine spiders/crawlers fromĀ

Leave a Reply

Your email address will not be published. Required fields are marked *