Guido van Rossum 3edbb35023 Added robots.txt support, using Skip Montanaro's parser. ... Fixed occasional inclusion of unpicklable objects (Message in errors). Changed indent of a few messages.		1997-01-30 03:19:41 +00:00
..
mimetypes.py	mime types guesser	1997-01-30 02:44:20 +00:00
robotparser.py	Skip Montanaro's robots.txt parser.	1997-01-30 03:18:23 +00:00
webchecker.py	Added robots.txt support, using Skip Montanaro's parser.	1997-01-30 03:19:41 +00:00