Popularity
1.2
Growing
Activity
4.5
-
10
1
2
Description
Use this library within crawler / scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.
Programming language: PHP
License: MIT License
Latest version: v1.1.2
crwlr / robots-txt alternatives and similar libraries
Based on the "Markup" category.
Alternatively, view robots-txt alternatives based on common mentions on social networks and blogs.
-
PHP Markdown
Parser for Markdown and Markdown Extra derived from the original Markdown.pl by John Gruber. -
CommonMark PHP
Highly-extensible PHP Markdown parser which fully supports the CommonMark and GFM specs.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
Promo
www.saashub.com
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of crwlr / robots-txt or a related project?