# Whitelisted user-agents are allowed User-agent: Mediapartners-Google Disallow: /cgi-bin User-agent: googlebot Disallow: /cgi-bin User-agent: msnbot Disallow: /cgi-bin User-agent: teoma Disallow: /cgi-bin User-agent: slurp Disallow: /cgi-bin User-agent: ScoutJet Disallow: /cgi-bin # Disallow all others User-agent: * Disallow: /