# # Robots.txt for www/tbssoft.com # note: different bots support differing extensions to the robots std # so don't try using on any bots! # # Allow any bot User-agent: * Disallow: # to prevent any bots, uncomment next line #Disallow: / # # 01Dec2004 # crawl-delay extension...number of seconds between web page access by the bot # Not all bots support this extension, so they are specified only for them # User-agent: msnbot Crawl-Delay: 1200 User-agent: Slurp Crawl-Delay: 300 User-agent: BecomeBot Crawl-Delay: 1200 # # 01Dec2004 # syntax extension..* (generic) and $( end of name) # Only these 1 bots currently support this extension # User-agent: googlebot Disallow: /*.gif$