Robots.txt Examples
User-agent: * Disallow: /admin/ Disallow: /login/ Allow: / Sitemap: https://example.com/sitemap.xml
What is robots.txt?
A robots.txt file tells search engines which pages they can or cannot crawl. It helps improve SEO by preventing indexing of unnecessary pages.
Best practices
- Block admin and private pages
- Always include your sitemap
- Do not block important content
FAQ
Where to upload robots.txt?
Upload it to your root domain: yourdomain.com/robots.txt
Does robots.txt improve SEO?
Yes, it helps search engines crawl your site efficiently.