What is robot.txt?
This is a file for in the website some of the folder we dont want to crawl by the search engine optimization, suppose in the images folder we dont want to crawl .
The syntax for the robot.txt
****************
User-agent:
Disallow:
****************
User- agent : This is for Search engine name
Disallow : This is for the folder name
eg:
User-agent : Googlebot
Disallow:/images/
then google search engine does not crawling the images folder of site.
For all search engines does not want to crawling then use the following code:
User-agent : *
Disallow: /images/
Disaallow : /admin/
Hope this will be useful …
1 Comment
Srinivas Tamada · April 24, 2009 at 9:19 am
Nice Information.