Popularity
1.2
Growing
Activity
4.5
-
10
1
2

Description

Use this library within crawler / scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

Programming language: PHP
License: MIT License
Latest version: v1.1.2

crwlr / robots-txt alternatives and similar libraries

Based on the "Markup" category.
Alternatively, view robots-txt alternatives based on common mentions on social networks and blogs.

Do you think we are missing an alternative of crwlr / robots-txt or a related project?

Add another 'Markup' Library