Popularity
0.8
Growing
Activity
3.2
-
8
1
2

Description

Use this library within crawler / scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

Programming language: PHP
License: MIT License

crwlr / robots-txt alternatives and similar libraries

Based on the "Markup" category.
Alternatively, view robots-txt alternatives based on common mentions on social networks and blogs.

Do you think we are missing an alternative of crwlr / robots-txt or a related project?

Add another 'Markup' Library