What is robot.txt? Syntax of robot.txt?

What is robot.txt?

This is a file for in the website some of the folder we dont want to crawl by the search engine optimization, suppose in the images folder we dont want to crawl .

The syntax for the robot.txt





User- agent  : This is for Search engine name

Disallow : This is for the folder name


User-agent : Googlebot


then google search engine does not crawling the images folder of site.

For all search engines does not want to crawling then use the following code:

User-agent : *

Disallow: /images/

Disaallow : /admin/

Hope this will be useful …

1 thought on “What is robot.txt? Syntax of robot.txt?”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.