User-agent: * Disallow: /cgi-bin/ Disallow: /carshare/ Disallow: /counter/ Disallow: /forms/ Disallow: /images Disallow: /includes/ #idiot-bot; generates tons of 404 and malformed urls User-agent: wget Disallow: / User-agent: ExtractorPro Disallow: /