4 commits

Author	SHA1	Message	Date
Skip Montanaro	663f6c2ad2	rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.	2001-01-20 15:59:25 +00:00
Guido van Rossum	dc8b7980e0	Skip Montanaro: ... The robotparser.py module currently lives in Tools/webchecker. In preparation for its migration to Lib, I made the following changes: * renamed the test() function _test * corrected the URLs in _test() so they refer to actual documents * added an "if __name__ == '__main__'" catcher to invoke _test() when run as a main program * added doc strings for the two main methods, parse and can_fetch * replaced usage of regsub and regex with corresponding re code	2000-03-27 19:29:31 +00:00
Guido van Rossum	986abac1ba	Give in to tabnanny	1998-04-06 14:29:28 +00:00
Guido van Rossum	bbf8c2fafd	Skip Montanaro's robots.txt parser.	1997-01-30 03:18:23 +00:00