Robots.txt Checker

Fetch and analyze any website's robots.txt file. View rules per user-agent, find sitemap URLs, and test if a URL is blocked. Free, no account needed.

Enter a domain without protocol (e.g., example.com). Fetched via CORS proxy.

About Robots.txt Checker

Analyze any website's robots.txt file to understand how search engines are instructed to crawl the site. Extract rules per user-agent, find sitemap references, and test URL paths against blocking rules. All in your browser.

Parse robots.txt

Extract all rules from any robots.txt: User-agent directives, Allow/Disallow rules, Crawl-delay, and Sitemap URLs. Structured tables per agent.

Test URL Against Rules

Check if a specific URL path would be blocked by the robots.txt rules. See which rule matches and whether the path is allowed or disallowed.

Sitemap Discovery

Automatically extract all Sitemap directives from the robots.txt file. Never miss a sitemap URL when auditing a site's SEO setup.

Raw View & Summary

See the full raw robots.txt alongside structured stats: total rules, user-agents found, and sitemap references. Copy raw content for further analysis.

📋 Common Use Cases

Page-Level SEO Setup

Generate complete meta tag sets — title, description, Open Graph, and Twitter Cards — for new pages or existing ones missing optimization.

Rich Result Preparation

Create validated JSON-LD schema markup for articles, products, FAQs, and more. Qualify for rich results that increase SERP click-through rates.

Site-Wide Technical Audit

Check robots.txt configuration, sitemap structure, and URL format across your site. Fix technical issues before they impact crawl budget.

Platform Migration

Generate proper redirect maps, canonical tags, and structured data when moving between platforms or restructuring your URL hierarchy.

🪜 How to Use This Tool

  1. Enter page or site details
    Provide the URL, page content, or site configuration depending on the specific SEO tool.
  2. Configure output requirements
    Select schema types, meta tag formats, or audit scope based on your needs.
  3. Generate or analyze
    Get generated meta tags, validated schema markup, or technical audit findings with prioritized recommendations.
  4. Implement on your site
    Copy generated code to your HTML head section, robots.txt, or sitemap. Re-audit after implementation to confirm fixes.

📚 Related Guides

Frequently Asked Questions

Are these SEO tools free?▼
Yes. All SEO tools are completely free. No signup needed.
Do the meta tags work for Google?▼
Yes. The meta tag generator creates standard Open Graph, Twitter Card, and HTML meta tags recognized by Google, Facebook, Twitter, and other platforms.
Is the schema markup valid?▼
Yes. The schema generator produces valid JSON-LD following Schema.org specifications for Article, FAQ, Product, LocalBusiness, and other types.