id	author	title	date	pages	extension	mime	words	sentences	flesch	summary	cache	txt
www-robotstxt-org-3124		The Web Robots Pages			.html	application/xhtml+xml	199	30	79	The /robots.txt Frequently Asked Questions Mailing list /robots.txt checker Robots Database IP lookup The Web Robots Pages The Web Robots Pages Web Robots (also known as Web Wanderers, Crawlers, or Spiders), as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses. On this site you can learn more about web robots. About /robots.txt explains what /robots.txt is, and how to use it. The FAQ answers many frequently asked questions, such as How do I stop robots visiting my site? and How can I get the best listing The Other Sites page links to external resources for robot writers and webmasters. The Robots Database has a list of robots. The /robots.txt checker can check your site's /robots.txt file and meta tags. The IP Lookup can help find out more about what robots are visiting you. About this site About this site	./cache/www-robotstxt-org-3124.html	./txt/www-robotstxt-org-3124.txt