User-agent: * Disallow: /html/ User-agent: Slurp Disallow: /*.?$ User-agent: psbot Disallow: /