# All robots will spider the domain User-agent: * Disallow: # Disallow directory /webalizer User-agent: * Disallow: /_private Disallow: /_vti_bin Disallow: /_vti_cnf Disallow: /_vti_log Disallow: /_vti_pvt Disallow: /_vti_txt Disallow: /awmdata Disallow: /error Disallow: /Newsletters Disallow: /stats Disallow: /u6rlyjl0ohow Disallow: /Xtra