# Robots.txt file for http://www.worldtravelguide.net # Disallow the HBX content level that some search engines think are URLs User-agent: * Disallow: /EN/ Disallow: /DE/ Disallow: /ES/ Disallow: /FR/ Disallow: /travel_centre/redirect.ehtml User-agent: * Disallow: /newsletter/ User-agent: VoilaBot Crawl-Delay: 30 User-agent: Speedy Crawl-Delay: 30 User-agent: TurnitinBot Crawl-Delay: 30 User-agent: ccbot Crawl-Delay: 30 User-agent: Yanga Crawl-Delay: 30