Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.



About Robots.txt Generator

Sometimes you don’t want search engine crawlers to index some of the pages on your website. This is where a robots.txt generator comes in. It gives you the opportunity to make a list of pages on your website that you don’t want indexed in search engines.
Our robots.txt generator is very easy to use. All you need is to include all the relevant information and click on the robots.txt generator. Our tool also allows you to select crawl-delay, default, and sitemap if you already have one. You also have the option of selecting the search engine robots that you want crawled by selecting ‘same as default’ ‘allowed’ or ‘refused’, as the case may be. Also, you can include any directories that you don’t want crawled, like private photo galleries, video selection and so on. Our robots.txt generator is very effective and simple to use for everyone, even if you’re new to internet stuff.