What is robot.txt? Syntax of robot.txt?

What is robot.txt?

This is a file for in the website some of the folder we dont want to crawl by the search engine optimization, suppose in the images folder we dont want to crawl .

The syntax for the robot.txt

****************

User-agent:

Disallow:

****************

User- agentĀ  : This is for Search engine name

Disallow : This is for the folder name

eg:

User-agent : Googlebot

Disallow:/images/

then google search engine does not crawling the images folder of site.

For all search engines does not want to crawling then use the following code:

User-agent : *

Disallow: /images/

Disaallow : /admin/

Hope this will be useful …


Anil Kumar Panigrahi

With more than 10 years experience in PHP and Founder of Anil Labs, a blog for PHP and related posts. contributed posts regarding CodeIgniter, CakePHP and Learn PHP online ... Contact : [email protected]

You may also like...

1 Response

Leave a Reply

Your email address will not be published. Required fields are marked *