Popularity
0.8
Growing
Activity
3.2
-
8
1
2
Description
Use this library within crawler / scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.
Programming language: PHP
License: MIT License
crwlr / robots-txt alternatives and similar libraries
Based on the "Markup" category.
Alternatively, view robots-txt alternatives based on common mentions on social networks and blogs.
-
PHP Markdown
Parser for Markdown and Markdown Extra derived from the original Markdown.pl by John Gruber. -
CommonMark PHP
Highly-extensible PHP Markdown parser which fully supports the CommonMark and GFM specs.
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Promo
www.influxdata.com
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of crwlr / robots-txt or a related project?