Popularity
0.6
Stable
Activity
0.0
-
5
1
1
Description
Use this library within crawler / scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.
Programming language: PHP
License: MIT License
crwlr / robots-txt alternatives and similar libraries
Based on the "Markup" category.
Alternatively, view robots-txt alternatives based on common mentions on social networks and blogs.
-
PHP Markdown
Parser for Markdown and Markdown Extra derived from the original Markdown.pl by John Gruber. -
CommonMark PHP
Highly-extensible PHP Markdown parser which fully supports the CommonMark and GFM specs. -
HTML to Markdown
Convert HTML to Markdown with PHP -
Cebe Markdown
A super fast, highly extensible markdown parser for PHP -
A HTML DOM parser written in PHP
๐ Modern Simple HTML DOM Parser for PHP -
Shortcode
Advanced shortcode (BBCode) parser and engine for PHP -
Decoda
A lightweight lexical string parser for BBCode styled markup.
Access the most powerful time series database as a service
Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
Promo
www.influxdata.com
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of crwlr / robots-txt or a related project?