diff options
| author | LukenShiro <lukenshiro@ngi.it> | 2011-12-14 13:05:27 -0600 |
|---|---|---|
| committer | Niels Horn <niels.horn@slackbuilds.org> | 2011-12-18 00:07:49 -0200 |
| commit | d2894d596903d61f8cf80b155968a23e45581a23 (patch) | |
| tree | 3bb44beae444a83d1d12afda5e642e864f2d6c9d /perl/perl-www-robotrules/README | |
| parent | ca407505737e5d4712f1675a0a44a404de6debc8 (diff) | |
| download | slackbuilds-d2894d596903d61f8cf80b155968a23e45581a23.tar.gz | |
perl/perl-www-robotrules: Added (DB of robots.txt-derived perms)
Signed-off-by: Robby Workman <rworkman@slackbuilds.org>
Diffstat (limited to 'perl/perl-www-robotrules/README')
| -rw-r--r-- | perl/perl-www-robotrules/README | 10 |
1 files changed, 10 insertions, 0 deletions
diff --git a/perl/perl-www-robotrules/README b/perl/perl-www-robotrules/README new file mode 100644 index 0000000000..54915b3f0b --- /dev/null +++ b/perl/perl-www-robotrules/README @@ -0,0 +1,10 @@ +This module parses /robots.txt files as specified in "A Standard for +Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> +Webmasters can use the /robots.txt file to forbid conforming robots +from accessing parts of their web site. +The parsed files are kept in a WWW::RobotRules object, and this +object provides methods to check if access to a given URL is +prohibited. The same WWW::RobotRules object can be used for one +or more parsed /robots.txt files on any number of hosts. + +This requires perl-uri-escape. |