Robots.txt for SEO: examples, mistakes and validation

Where robots.txt belongs

It must live at the domain root, such as https://example.com/robots.txt.

Basic example

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Dangerous mistakes

• Disallow: / blocks the entire site for the current group.
• Missing sitemap directives can reduce URL discovery.
• Blocking CSS or JavaScript can hurt rendering.
• Robots.txt is not a privacy or security mechanism.