Robots.txt Tester

Read a site's robots.txt, see the rules that apply to each crawler, and test whether a specific URL would be allowed or blocked — the file where an accidental line can remove a whole site from search.