How to Create Robots.txt file?

Posted: 03 Jan 2011 12:34 PM PST


Simply create a text document and save the new document as robots.txt
> Open Text file
> Insert instructions to robots
> Click on save as
> Save document as robots.txt

Allow all robots to crawl and index all files.
This allows all robots to crawl all files.

User-agent: *
Disallow:

Crawl all files except the images file.

User-agent: *
Disallow: /images/

Crawl all files except the images file and the stats file.

User-agent: *
Disallow: /images/
Disallow: /stats/

Denies access to Googlebot-image to any files in your domain

User-agent: Googlebot-Image
Disallow: /

Denies Googlebot-image to your images file

User-agent: Googlebot-Image
Disallow: /images/