Robots Spider reads your site's robots.txt file and the site to check for client-side references to the robots.txt entries. This allows you to prevent disclosure of directories that you don't want indexed by identifying the ones that aren't referenced by local pages. Search engines read the site and build their indexes accordingly. Directories and files that aren't referenced on the site shouldn't be indexed by spiders, and their presence in the robots.txt file may be security risk.
Robots Spider reads your site's robots.txt file and the site to check for client-side references to the robots.txt entries. This allows you to prevent disclosure of directories that you don't want indexed by identifying the ones that aren't referenced by local pages. Search engines read the site and build their indexes accordingly. Directories and files that aren't referenced on the site shouldn't be indexed by spiders, and their presence in the robots.txt file may be security risk.
» JavaScrips
» CSS Compress
Analyzer
» Robots Spider
» Password Checker
Blog Tools
» Trackback Test
» RSS Button
» Favicon Generator
» Blog Worth
Keyword Tools
» Keyword Suggest
» Keyword Density
Online Robots.txt Generator Tool
► Robots.txt Generator
► Create your own robots.txt