Popularity
0.9
Stable
Activity
3.6
Growing
10
1
2
Description
Use this library within crawler / scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.
Programming language: PHP
License: MIT License
crwlr / robots-txt alternatives and similar libraries
Based on the "Markup" category.
Alternatively, view robots-txt alternatives based on common mentions on social networks and blogs.
-
PHP Markdown
Parser for Markdown and Markdown Extra derived from the original Markdown.pl by John Gruber. -
CommonMark PHP
Highly-extensible PHP Markdown parser which fully supports the CommonMark and GFM specs.
CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
Promo
coderabbit.ai

* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of crwlr / robots-txt or a related project?