Robots.txt Parser
Professional robots.txt parser and analyzer that extracts crawl directives, user-agent rules, disallow/allow patterns, sitemap URLs, and crawl delays. Provides comprehensive SEO insights including accessibility analysis, rule conflicts, syntax validation, and security recommendations. Perfect for SEO audits, bot management, website compliance, and automated crawling workflows.
6 subscribers
2 endpoints
The in-depth APIMemo review for this API hasn't been published yet —
the data below comes straight from the public marketplace listing.
Robots.txt Parser endpoints
| Method | Endpoint | Description |
|---|---|---|
| api | ||
| GET |
robots-txt-parser-doc /api/web/robots-txt-parser/v1/info |
Parse and analyze robots.txt files for SEO insights, crawl directives, and sitemap discovery |
| POST |
robots-txt-parser /api/web/robots-txt-parser/v1 |
Parse and analyze robots.txt files for SEO insights, crawl directives, and sitemap discovery |
Robots.txt Parser pricing
| Plan | Price | Rate limit | Quotas |
|---|---|---|---|
| BASIC | Free | — |
|
| PRO | $8 / month | 1000 / hour |
|
| ULTRA Recommended | $12 / month | 1000 / hour |
|
| MEGA | $18 / month | — |
|
