Get a video review of your website by our experts & improve your enquiries, leads and sales. Yes please!

Cookies

To make this site work properly, we sometimes place small data files called cookies on your device. Find out more

OK
Share

How should my robots.txt file look?

A robots.txt file is a list of simple commands to Google search, so as such it looks fairly plain. It’s a txt document with lines of text with code and instructions. You’ll see commands like ‘Disallow: /pagenameexample’ or ‘Allow: /site.example’. These commands tell Googlebot what to crawl and what to ignore.