What is robot.txt? Syntax of robot.txt?

What is robot.txt?

This is a file for in the website some of the folder we dont want to crawl by the search engine optimization, suppose in the images folder we dont want to crawl .

The syntax for the robot.txt





User- agent  : This is for Search engine name

Disallow : This is for the folder name


User-agent : Googlebot


then google search engine does not crawling the images folder of site.

For all search engines does not want to crawling then use the following code:

User-agent : *

Disallow: /images/

Disaallow : /admin/

Hope this will be useful …

Anil Kumar Panigrahi

With more than 10 years experience in PHP and Founder of Anil Labs, a blog for PHP and related posts. contributed posts regarding CodeIgniter, CakePHP and Learn PHP online ... Contact : [email protected]

You may also like...

1 Response

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.