To make this site work properly, we sometimes place small data files called cookies on your device. Find out more


How should my robots.txt file look?

A robots.txt file is a list of simple commands to Google search, so as such it looks fairly plain. It’s a txt document with lines of text with code and instructions. You’ll see commands like ‘Disallow: /pagenameexample’ or ‘Allow: /site.example’. These commands tell Googlebot what to crawl and what to ignore.

Sign up to our newsletter