Get a video review of your website by our experts & improve your enquiries, leads and sales. Yes please!

Cookies

To make this site work properly, we sometimes place small data files called cookies on your device. Find out more

OK
Share

How do you stop crawlers?

Using your robots.txt file you can stop search engines like Google from crawling certain pages or the entire site. To stop Google crawling your website altogether set your root domain to be disallowed. In many websites, especially sites like WordPress you can use SEO tools like Yoast to block search engines if your site is temporarily under construction.